Abnormal root cause positioning method and device for target system

Abstract

The embodiment of the invention provides an abnormal root cause positioning method and device for a target system, and the method comprises the steps: obtaining a service call graph of the target system; for any first micro-service call, a corresponding first call pattern graph is determined, the first micro-service call corresponding to a call pointing to a first node through a first edge in the service call graph, and the first call pattern graph comprises a preorder node of the first node and a first-order child node. And determining a target calling mode pattern matched with the first calling mode pattern from the known calling mode patterns. And obtaining the estimated execution time of the first micro-service call based on the historical execution time distribution of the target call mode pattern. And determining a first abnormal score of the first micro-service call according to the difference between the estimated execution time of the first micro-service call and the actual observed execution time. And determining an abnormal root cause of the target system according to each abnormal score of each micro-service call in the service call graph.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and more particularly, to a method and device for locating a root cause of an abnormality in a target system. Background Art

[0002] In a distributed framework, the target system is usually split into multiple microservices, and there are complex call relationships between the microservices. When an exception occurs in a microservice or a call in the call chain, it will affect the entire call chain, and then cause the target system to be abnormal. Only by quickly locating the root cause of the exception can timely measures be taken to avoid damage to user interests.

[0003] Therefore, it is necessary to provide an effective solution for locating the root cause of abnormalities in the target system. Summary of the invention

[0004] One or more embodiments of this specification describe a method for locating the root cause of anomalies in a target system, which can quickly and accurately identify abnormal microservice calls.

[0005] In a first aspect, a method for locating a root cause of an abnormality in a target system is provided, comprising:

[0006] Get the service call graph of the target system, where a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call;

[0007] For any first microservice call, determine a corresponding first call pattern graph, the first microservice call corresponds to a call pointing to a first node through a first edge in the service call graph, the first call pattern graph includes a preceding node of the first node and a first-order child node;

[0008] Determine a target call pattern graph that matches the first call pattern graph from each known call pattern graph;

[0009] Obtaining an estimated execution time of the first microservice call based on the historical execution time distribution of the target call pattern graph;

[0010] Determining a first anomaly score for the first microservice call according to a difference between an estimated execution time of the first microservice call and an actual observed execution time;

[0011] The abnormal root cause of the target system is determined according to each abnormal score of each microservice call in the service call graph.

[0012] In a second aspect, a device for locating an abnormality root cause of a target system is provided, comprising:

[0013] An acquisition unit is used to acquire a service call graph of a target system, in which a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call;

[0014] A determining unit, configured to determine a corresponding first call pattern graph for any first microservice call, wherein the first microservice call corresponds to a call pointing to a first node through a first edge in the service call graph, and the first call pattern graph includes a preceding node of the first node and a first-order child node;

[0015] The determining unit is further used to determine a target calling pattern graph matching the first calling pattern graph from each known calling pattern graph;

[0016] An acquisition unit, configured to obtain an estimated execution time of the first microservice call based on a historical execution time distribution of the target call pattern graph;

[0017] The determining unit is further configured to determine a first anomaly score of the first microservice call according to a difference between the estimated execution time of the first microservice call and the actual observed execution time;

[0018] The determining unit is further configured to determine the abnormal root cause of the target system according to each abnormal score of each microservice call in the service call graph.

[0019] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0020] In a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0021] One or more embodiments of the present specification provide a method for locating the root cause of anomalies in a target system. For each microservice call in a service call graph, the estimated execution time of each microservice call is calculated based on the historical execution time distribution of the corresponding known call pattern graph. Then, by comparing the estimated execution time of each microservice call with the actual observed execution time, it is determined whether each microservice call is abnormal, thereby quickly and accurately identifying abnormal microservice calls. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of a service call graph in one example is shown;

[0024] Figure 2 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0025] Figure 3 A schematic diagram showing a call pattern diagram in an example;

[0026] Figure 4 A schematic diagram showing a method for calculating the first and second Gaussian distributions in one example;

[0027] Figure 5 A flow chart of a method for locating an abnormality root cause of a target system according to an embodiment is shown;

[0028] Figure 6 A schematic diagram of a device for locating an abnormality root cause for a target system according to an embodiment is shown. DETAILED DESCRIPTION

[0029] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0030] There are many reasons for the target system to be abnormal in the production environment, such as abnormal service response, machine abnormality, improper code writing, etc. In order to monitor the health of the target system and locate the root cause of the abnormality, the call requests (hereinafter referred to as calls) between the microservices in the target system can be tracked to obtain a service call graph, which is also called a distributed trace diagram or tracing visualization. The service call graph includes multiple nodes and multiple directed edges, where a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call.

[0031] Figure 1 A schematic diagram of a service call graph in one example is shown. Figure 1In the figure, nodes A to F represent 6 microservices respectively, among which A calls B once, represented by directed edge S1; A calls C once, represented by directed edge S2; B calls D once, represented by directed edge S3; D calls E twice, represented by directed edges S4 and S5 respectively; D calls F twice, represented by directed edges S6 and S7 respectively.

[0032] Generally, the microservice that initiates the call is called the upstream microservice or parent microservice, and the microservice being called is called the downstream microservice or child microservice. For example, when A calls B, A can be called the upstream microservice or parent microservice, and B can be called the downstream microservice or child microservice.

[0033] In addition, in the service call graph of this solution, when a parent microservice calls the same child microservice multiple times, the multiple calls are represented as multiple directed edges, and each directed edge points to a separate node representing the child microservice. In other words, there is only one edge connecting any two nodes in the service call graph.

[0034] Furthermore, the microservice represented by the root node in the service call graph can be called the entry microservice, which will be called by the external system or platform (referred to as external call), and the external call can also be represented as a directed edge. For example, Figure 1 The directed edge S0 pointing to node A represents the external call to microservice A.

[0035] It should also be noted that a single directed edge in the service call graph also corresponds to the observed response time, which can also be understood as the observed response time of calling the target microservice pointed to by the directed edge, that is, the observed response time of the microservice call. The observed response time can be divided into two parts: the observed execution time of the microservice call and the observed response time of the sub-microservice call. The sub-microservice call can be understood as the call of the target microservice to the sub-microservice under the above microservice call. For example, Figure 1 In the figure, S1 and S2 are sub-microservice calls of S0, and S3 is a sub-microservice call of S1.

[0036] This solution will locate the root cause of the exception in the target system based on the above service call graph, which is described in detail below.

[0037] Figure 2 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification. Figure 2 In the service call graph, each microservice call S i , and determine the corresponding call pattern diagram. For example, for S0, a call pattern diagram can be generated: Pattern1; for another example, for S7, a call pattern diagram can be generated: Pattern n .

[0038] After that, call S for each microservice i , we can get the known call pattern diagrams: Q1, Q2, ..., Q m The target call pattern graph Q that matches its call pattern graph is determined j , and the historical execution time distribution based on the target call pattern graph: θ(Q j ), and get the estimated execution time of the microservice call: ET(S i ). Then, you can call S according to each microservice i Estimated execution time: ET(S i ) and the actual observed execution time: ET′(S i ) to determine the corresponding abnormal score: AS(S i ). Finally, the root cause of the abnormality of the target system is determined based on the abnormality score of each microservice call.

[0039] The following is an explanation of the process of determining the above-mentioned known call pattern diagrams and their historical execution time distribution.

[0040] Each known call pattern graph described in this solution may be generated by traversing each node in a plurality of historical service call graphs, wherein a single known call pattern graph may include a preceding node and a first-order child node of a currently traversed node.

[0041] Figure 3 A schematic diagram of a call pattern diagram in one example is shown. Figure 3 Based on historical service calls Figure 1 (or graph N) and its context, we can get the call pattern graph Q1, which includes node A and its first-order child nodes B and C; based on historical service calls Figure 1 (or N) and its context, we can get the call pattern graph Q2, which includes the previous node A and the first-order child node D of node B; and the call pattern graph Q2 based on the historical service call Figure 1 (or graph N) node D and its context, we can get the call pattern graph Q m , which includes the predecessor nodes A and B of node D and the first-order child nodes e and F, etc.

[0042] In practice, a known call pattern graph can be represented as a list of pre-order nodes and a set of subsets, for example, Q1 = ([A], {B, C}), Q2 = ([A, B], {D}), ..., Q m =([A,B,D],{E,F}).

[0043] According to the above, each known call pattern graph is generated for a certain node, or each known call pattern graph corresponds to a node. In the following, the corresponding node contained in each known call pattern graph is called the target node. For example, node A in the call pattern graph Q1 is called the target node, and node A in the call pattern graph Q2 is called the target node. m The node D in is called the target node.

[0044] For any of the above known call pattern graphs, the process of determining the corresponding historical execution time distribution can be as follows:

[0045] First, in this solution, the historical execution time of any known call pattern graph is divided into two parts: a first sub-time that is not related to the call sub-microservice, and a second sub-time that is related to the call sub-microservice. The sub-microservice here is relative to the target microservice (the microservice represented by the target node) in the known call pattern graph. It should be understood that in the case of multiple sub-microservices, the number of the above second sub-times is also multiple.

[0046] It should be understood that, in the case where the above historical execution time is divided into a first sub-time and a second sub-time, the above historical execution time distribution includes: a first distribution of the first sub-time and a second distribution of the second sub-time. In the case where it is assumed that the first distribution and the second distribution are both Gaussian distributions, the first and second distributions are also referred to as first and second Gaussian distributions.

[0047] In one embodiment, a maximization EM algorithm or a greedy EM algorithm can be used to solve the mixed Gaussian distribution satisfied by the first sub-time and the second sub-time based on the observed execution time of each historical microservice call statistics for the known call pattern graph, and obtain the distribution parameters of the first and second Gaussian distributions. The distribution parameters here include the mean (also called expectation) and the variance.

[0048] Of course, in practice, the nonlinear least square method may also be used to calculate the means of the first and second Gaussian distributions, which is not limited in this specification.

[0049] Figure 4 A schematic diagram showing a method for calculating the first and second Gaussian distributions in one example. Figure 4 The left side shows the map that can be mapped to the calling pattern diagram: m= ([G, A], {B, C}), and the target microservice of the three historical microservice calls is A. It should be understood that for the first historical microservice call, the corresponding observed execution time can be calculated as: 25ms = 88ms-21ms-12ms-11ms-19ms; for the second historical microservice call, the corresponding observed execution time can be calculated as: 19ms = 80ms-24ms-19ms-18ms; for the third historical microservice call, the corresponding observed execution time can be calculated as: 22ms = 100ms-23ms-25ms-14ms-16ms. Of course, in practice, other observed execution times can also be calculated.

[0050] also, Figure 4 In the figure, the rectangle represents the first sub-time, which can be specifically expressed as R A ,The two squares represent two second sub-times respectively, where the square filled with horizontal lines represents the second sub-time related to calling sub-microservice B, which is specifically represented as C (A,B) The square filled with vertical lines represents the second sub-time associated with calling the sub-microservice C, which is specifically denoted as C (A,C) .

[0051] Specifically, it can be determined that R A , C (A,B) and C (A,C) The initial distribution parameters of the Gaussian distributions are obtained by calculating the initial distribution parameters of the three Gaussian distributions as initial coefficients to construct a mixed Gaussian distribution model. The EM algorithm or the greedy EM algorithm is used to iteratively calculate the coefficients of the mixed Gaussian distribution model based on the observed execution time: 19ms, 18ms, and 16ms, etc., to obtain the final coefficients of the model. According to the final coefficients, R is determined. A , C (A,B) and C (A,C) The final distribution parameters of the Gaussian distribution they obey. A The two distribution parameters that obey the Gaussian distribution can be expressed as: (t r (A),σ r (A) 2 );C (A,B) The two distribution parameters that obey the Gaussian distribution can be expressed as: (t c (A,B),σ c (A,B) 2 );C (A,C) The two distribution parameters that obey the Gaussian distribution can be expressed as: (t c (A,C),σ c (A,C) 2 ).

[0052] Similarly, the historical execution time distribution of each known call pattern graph can be determined.

[0053] Figure 5 A flow chart of a method for locating the root cause of anomalies in a target system according to an embodiment is shown. The method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 5 As shown, the method may include the following steps:

[0054] Step S502: Obtain a service call graph of the target system, wherein a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call.

[0055] Among them, the service call graph here can be as follows Figure 1 As shown, there is only one edge connecting any two nodes, and the root node also has a directed edge pointing to it. In addition, a single directed edge in the service call graph also corresponds to the observed response time, which can also be understood as the observed response time of calling the target microservice pointed to by the directed edge, that is, the observed response time of the microservice call. The observed response time can be divided into two parts: the observed execution time of the microservice call and the observed response time of the sub-microservice call. The sub-microservice call can be understood as the call of the target microservice to the sub-microservice under the above-mentioned microservice call. For example, Figure 1 In the figure, S1 and S2 are sub-microservice calls of S0, and S3 is a sub-microservice call of S1.

[0056] Step S504: for any first microservice call, determine a corresponding first call pattern graph, where the first microservice call corresponds to a call pointing to a first node through a first edge in the service call graph, and the first call pattern graph includes a preceding node of the first node and a first-order child node.

[0057] The first edge mentioned above can be any directed edge in the service call graph, and the first node is the node pointed to by the first edge. For example, the first microservice call here can be Figure 1 The call to D is pointed to by S3, or the call to E is pointed to by S4, etc. In this solution, the microservice called in the first microservice call is called the target microservice, that is, the microservice represented by the first node is the target microservice.

[0058] Specifically, the corresponding first call pattern graph can be determined based on the preceding node of the first node in the service call graph and its first-order child node. Taking the first microservice call as a call directed to D through S3 as an example, the corresponding first call pattern graph can be P i =([A,B,D],{E,F}).

[0059] Step S506: determine a target calling pattern graph that matches the first calling pattern graph from each known calling pattern graph.

[0060] Specifically, the first calling mode diagram P i Compared with the known call pattern diagrams Q1, Q2, ..., Q m If there is a known call pattern graph that is consistent with it, the known call pattern graph is used as the target call pattern graph. If there is no known call pattern graph that is consistent with it, a similar call pattern graph of the first call pattern graph is selected from each known call pattern graph as the target call pattern graph.

[0061] It should be noted that, as mentioned above, the call pattern graph can be represented as a pre-order node list and a subset set, so the consistency described in this solution means that the pre-order node list and the subset set are consistent.

[0062] In addition, the above-mentioned similar call pattern graph can meet the following conditions: the target microservice of the similar call pattern graph is the same as the first call pattern graph, or the similar call pattern graph contains a sub-microservice of the first call pattern graph; and the similarity between the similar call pattern graph and the first call pattern graph is greater than a preset threshold. In one example, the similarity here is the inverse of the edit distance, which refers to the minimum number of edit operations required to convert from one to another between two coding groups. The larger the distance, the more different they are. The editing operation may include replacing one code with another, inserting a code, and deleting a code.

[0063] In the first call mode diagram P i =([D],{F,K}) for example, where the target microservice is D, and the two sub-microservices of the target microservice are F and K, then the corresponding similar call pattern diagrams may include: Q1 = ([D],{E,F}), Q2 = ([D],{E,F,G}), Q3 = ([M],{E,K}) and Q4 = ([J],{F,K}), etc. Among them, Q1 and Q2 are related to P i Contains the same target microservice D. In addition, Q1, Q2, and Q4 contain P i Sub-microservices F, Q3 and Q4 in P i Sub-microservice K in.

[0064] Step S508: Obtain an estimated execution time of the first microservice call based on the historical execution time distribution of the target call pattern graph.

[0065] As mentioned above, the historical execution time distribution here includes: a first Gaussian distribution of a first sub-time and a second Gaussian distribution of a second sub-time, and the number of the second Gaussian distributions may be multiple.

[0066] Specifically, the estimated execution time of the first microservice call can be obtained according to the means of the first and second Gaussian distributions and the number of times the target microservice corresponding to the first call pattern diagram (i.e., the microservice represented by the first node) calls the sub-microservice in the above service call graph.

[0067] Among them, when the target call pattern graph is the same as the first call pattern graph, the mean of the first Gaussian distribution can be superimposed with the product of the mean of the second Gaussian distribution and the number of times the target microservice calls the sub-microservice to obtain the estimated execution time of the first microservice call.

[0068] Also in the first call mode diagram P i =([D],{F,K}) for example, assuming that in the service call graph, D calls F twice and calls K three times, then the target call pattern graph can be selected: Q j =([D],{F,K}), ET(S i ) = t r (D)+2t c (D,F)+3t c (D,K). Among them, t r (D) is Q j The mean of the first Gaussian distribution, t c (D,F) is Q j The mean of the second Gaussian distribution associated with D calling F, t c (D,K) is Q j The mean of the second Gaussian distribution associated with D calling K, ET(S i ) is the estimated execution time of the first microservice call.

[0069] In addition, when the target call pattern graph includes multiple similar call pattern graphs of the first call pattern graph, first, each first similar pattern graph containing the same target microservice as the first call pattern graph is selected from the multiple similar call pattern graphs, and the means of the corresponding first Gaussian distributions are weighted and integrated to obtain a first comprehensive value. And, each second similar pattern graph containing a sub-microservice of the first call pattern graph is selected from the multiple similar call pattern graphs, and the means of the corresponding second Gaussian distributions are multiplied by the number of times the target microservice calls the sub-microservice, and each product is weighted and integrated to obtain a second comprehensive value. Finally, the first comprehensive value and the second comprehensive value are superimposed as the estimated execution time of the first microservice call.

[0070] Also in the first call mode diagram P i=([D],{F,K}) for example, and assuming that in the service call graph, D calls F twice and calls K three times, then in the case where multiple similar call pattern graphs include: Q1 = ([D],{E,F}), Q2 = ([D],{E,F,G}), Q3 = ([M],{E,K}) and Q4 = ([J],{F,K}), the first similar pattern graph selected above may include: Q1 and Q2, so that the first comprehensive value Q1 and Q2 are respectively i The similarity of are the means of the first Gaussian distributions of Q1 and Q2 respectively.

[0071] In addition, since the first call pattern graph contains two child nodes F and K, two groups of second similar pattern graphs can be selected for the two child nodes: (v1, Q2, v4) and (Q3, Q4). It should be noted that for each group of second similar pattern graphs selected, further screening can be performed on them, for example, the second similar pattern graphs whose target microservices are the same as the first call pattern graph can be screened out. For example, the two groups of second similar pattern graphs finally used are: (Q1, Q2) and (Q3, Q4).

[0072] After selecting the above two groups of second similarity pattern graphs, based on (Q1, Q2), a second comprehensive value can be calculated: in, Q1 and Q2 are respectively i The similarity of are the means of the second Gaussian distributions of Q1 and Q2, respectively, related to D calling F. And based on (Q3, Q4), another second comprehensive value can be calculated in, Q3, Q4 and P i The similarity of is the mean of the second Gaussian distribution of Q3 associated with M calling K, is the mean of the second Gaussian distribution of Q4 associated with J calling K.

[0073] Finally, ET(S i )=Com1+Com21+Com22.

[0074] Similarly, the estimated execution time of each microservice call can be calculated.

[0075] Step S510: determining a first anomaly score of the first microservice call according to a difference between the estimated execution time of the first microservice call and the actual observed execution time.

[0076] As mentioned above, any microservice call has a corresponding observed response time, and it can be divided into the observed execution time of the service call and the observed response time of the sub-microservice call. Therefore, the actual observed execution time of the above-mentioned first microservice call is the difference between the observed response time of the first microservice call and the observed response of the sub-microservice call.

[0077] by Figure 1 For example, the actual observed execution time of the microservice calling S3 in in, Observe response time for microservice calls to S3, are the observed response times of each of its sub-microservice calls. Similarly, the observed execution time of each microservice call can be calculated.

[0078] In one embodiment, the difference between the estimated execution time and the observed execution time of each microservice call may be used as the anomaly score of each microservice call.

[0079] In another embodiment, for any microservice call, the difference between its estimated execution time and observed execution time can be calculated to obtain a difference value, and then the ratio of the difference value to the target standard deviation is used as the corresponding anomaly score. The target standard deviation is determined according to the variance of the first and second Gaussian distributions and the number of times the target microservice in the corresponding call pattern diagram calls the sub-microservice.

[0080] In one example, any microservice call S can be calculated according to the following formula i Anomaly score:

[0081]

[0082] Among them, ET(S i ) is the microservice calling S i The estimated execution time, ET′(S i ) is the microservice calling S i The observed execution time, ε is the smoothing factor, AS(S i ) is the microservice calling S i The anomaly score of σ ET is a target standard deviation, which can be determined based on the variance of the first and second Gaussian distributions of the same or similar call pattern graph of the first call pattern graph.

[0083] In one example, σ ET It can be calculated according to the following formula:

[0084]

[0085] Among them, P iCall S for microservices i The corresponding call pattern diagram, assuming P i =([D],{F,K}), if the target call pattern can be selected: Q j =([D],{F,K}), then in, Q j The variance of the first Gaussian distribution, Q j The variance of the second Gaussian distribution associated with D calling F, Q j The variance of the second Gaussian distribution associated with D calls K.

[0086] If we only select multiple similar call pattern graphs: Q1 = ([D], {E, F}), Q2 = ([D], {E, F, G}), Q3 = ([M], {E, K}) and Q4 = ([J], {F, K}), then Among them, Com1', Com21', Com22' can be specifically obtained by replacing the means in Com1, Com21, Com22 with corresponding variances, which will not be elaborated in this specification.

[0087] Step S512: Determine the root cause of the abnormality of the target system according to each abnormality score of each microservice call in the service call graph.

[0088] In one embodiment, the abnormal root cause of the target system can be determined based on the microservice call with the highest abnormal score among the microservice calls. For example, the target microservice of the microservice call with the highest abnormal score is determined as the abnormal microservice.

[0089] In another embodiment, the abnormality scores of each microservice call may be corrected first, and then the abnormal root cause of the target system may be determined based on the corrected scores of each microservice call.

[0090] Taking any microservice call (hereinafter referred to as the first microservice call) as an example, the corresponding correction process can be as follows:

[0091] According to the historical abnormality proportions of the first microservice call and its sub-microservice call, and the historical average number of times the target microservice calls the sub-microservice corresponding to the first call pattern diagram, a corresponding target transfer parameter is determined, which indicates the probability that the abnormality of the first microservice call is caused by itself and is unrelated to the calling sub-microservice. At least based on the target transfer parameter, the abnormality score of the first microservice call is corrected to obtain a corrected score.

[0092] Specifically, the observed response time of each historical microservice call mapped to the first call pattern diagram (i.e., the call pattern diagram determined based on the first microservice call) can be counted, and then based on the 3σ verification method, etc., the abnormal observed response time can be determined, and the abnormal observed response time ratio is obtained by counting, which is used as the historical abnormal ratio of the first microservice call. Similarly, the historical abnormal ratio of the sub-microservice call can be obtained by counting.

[0093] In one example, any microservice call S can be calculated according to the following formula i Number of target transfers:

[0094]

[0095] Among them, p e (S i ) is the microservice calling S i The historical anomaly ratio, S j For S i Sub-microservice call, p e (S j ) calls S for the child microservice j The historical anomaly ratio, P i For S i The corresponding calling pattern diagram, Corresponding to P i Microservices i ) calls the child microservice ms(S j )’s historical average number of times.

[0096] After calculating the corresponding target transfer times for the first microservice call, the abnormal score of the first microservice call can be used as the initial score, and multiple score transfers can be iterated; wherein a single score transfer includes determining a retention ratio according to a target transfer parameter, multiplying the retention ratio by the current score of the first microservice call as the retention score of the first microservice call, and transferring the remaining score obtained by subtracting the retention score from the current score to the sub-microservice call.

[0097] As mentioned above, there can be multiple sub-microservice calls for any microservice call, and the number of sub-microservice calls is related to the out-degree of the target node of the microservice call. Therefore, when there are multiple sub-microservice calls, the transfer ratio corresponding to each sub-microservice call can be determined according to the out-degree. For example, when the out-degree is 5, the corresponding transfer ratio can be 20%. Afterwards, according to the transfer ratio, the remaining score is transferred to each sub-microservice call. For example, each microservice call can get 20% of the remaining score.

[0098] It should be understood that after the above round of iterations, the current score of the first microservice call is updated to its reserved score, and the current score of each sub-microservice call is updated to its current score plus the transfer score for the next round of iterations.

[0099] After multiple score transfers, the correction score of each microservice call can be obtained, and then the root cause of the abnormality of the target system can be determined based on the microservice call with the highest correction score.

[0100] In one example, determining the root cause of the abnormality of the target system includes determining the target microservice called by the microservice corresponding to the highest correction score as the abnormal microservice.

[0101] Of course, in practice, the root cause of the abnormality of the target system can also be determined based on each microservice call whose corresponding correction score is greater than a preset threshold, and this specification does not limit this.

[0102] In summary, the method for locating the root cause of anomalies for the target system provided by the embodiment of this specification analyzes the historical service call graph, mines the call pattern graph, and calculates the corresponding historical execution time distribution. Afterwards, when locating the root cause of anomalies for a new service call graph, the same or similar call pattern graph can be referred to to estimate the estimated execution time of each microservice call to improve the performance of the algorithm. At the same time, it can also be used to compare with the actual observed execution time to determine the abnormal score of the microservice call, so that this solution has the advantages of low performance overhead and high efficiency. In addition, this solution is based on the link call information and microservice call response time obtained in advance at the data level, without the need to extract data from the log data, which improves the performance of the algorithm. Finally, this solution can realize the root cause location of the target system based only on a single service call graph, thereby making the processing flow of the root cause analysis of the target system more efficient and reducing the dependence on abnormal data.

[0103] Corresponding to the above-mentioned method for locating the root cause of anomaly in a target system, an embodiment of the present specification also provides a device for locating the root cause of anomaly in a target system, such as Figure 6 As shown, the device may include:

[0104] The acquisition unit 602 is used to acquire a service call graph of the target system, in which a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call.

[0105] The determination unit 604 is used to determine a corresponding first call pattern graph for any first microservice call, where the first microservice call corresponds to a call pointing to the first node through a first edge in the service call graph, and the first call pattern graph includes a preceding node of the first node and a first-order child node.

[0106] The determination unit 604 is further configured to determine a target calling pattern graph that matches the first calling pattern graph from each known calling pattern graph.

[0107] The acquisition unit 602 is further configured to obtain an estimated execution time of the first microservice call based on the historical execution time distribution of the target call pattern graph.

[0108] The determining unit 604 is further configured to determine a first anomaly score of the first microservice call according to a difference between the estimated execution time of the first microservice call and the actual observed execution time.

[0109] The determination unit 604 is further configured to determine the abnormal root cause of the target system according to each abnormal score of each microservice call in the service call graph.

[0110] In one embodiment, the historical execution time distribution includes: a first Gaussian distribution of a first sub-time that is not related to calling the sub-microservice, and a second Gaussian distribution of a second sub-time that is related to calling the sub-microservice;

[0111] The acquisition unit 602 is specifically used for:

[0112] According to the means of the first and second Gaussian distributions and the number of times the target microservice corresponding to the first call pattern graph calls the sub-microservice, an estimated execution time of the first microservice call is obtained.

[0113] In one embodiment, the target call pattern graph is the same as the first call pattern graph;

[0114] The acquisition unit 602 is further specifically configured to:

[0115] The estimated execution time is obtained by superimposing the mean of the first Gaussian distribution and the product of the mean of the second Gaussian distribution and the above number of times.

[0116] In another embodiment, the target call pattern graph includes a plurality of similar call pattern graphs of the first call pattern graph;

[0117] The acquisition unit 602 includes:

[0118] A selection submodule 6022 is used to select first similar pattern graphs whose target microservices are the same as the first call pattern graph from multiple similar call pattern graphs, and perform weighted synthesis on the means of the corresponding first Gaussian distributions to obtain a first comprehensive value;

[0119] The selection submodule 6022 is further used to select each second similar pattern graph of the sub-microservice including the first call pattern graph from the multiple similar call pattern graphs, multiply the mean of the corresponding second Gaussian distribution by the above number of times, and perform weighted synthesis on each product to obtain a second comprehensive value;

[0120] The superposition submodule 6024 is used to superimpose the first comprehensive value and the second comprehensive value as the above-mentioned estimated execution time.

[0121] In one embodiment, the above similar call pattern diagram satisfies the following conditions:

[0122] The target microservice of the similar call pattern graph is the same as the first call pattern graph, or the similar call pattern graph includes a sub-microservice of the first call pattern graph;

[0123] The similarity between the similar call pattern graph and the first call pattern graph is greater than a preset threshold.

[0124] In one embodiment,

[0125] The selection submodule 6022 is specifically used for:

[0126] Using the similarity between the first call pattern graph and each first similar pattern graph as a weight coefficient, weighted synthesis is performed on the means of the corresponding first Gaussian distributions;

[0127] The selection submodule 6022 is also specifically used for:

[0128] The similarity between the first call pattern graph and each second similar pattern graph is used as a weight coefficient to perform weighted synthesis on the corresponding products.

[0129] In one embodiment, the determining unit 604 includes:

[0130] The difference calculation submodule 6042 is used to calculate the difference between the estimated execution time and the observed execution time to obtain a first difference value;

[0131] An acquisition submodule 6044 is used to use the ratio of the first difference value to the target standard deviation as a first anomaly score;

[0132] The target standard deviation is determined according to the variance of the first and second Gaussian distributions, and the above-mentioned order.

[0133] In one embodiment, the apparatus further comprises:

[0134] The solving unit 606 is used to adopt a maximizing EM algorithm or a greedy EM algorithm to solve the mixed Gaussian distribution satisfied by the first sub-time and the second sub-time based on each observed execution time of each historical microservice call statistics mapped to the target call pattern graph, and obtain the distribution parameters of the first and second Gaussian distributions.

[0135] In one embodiment, the determining unit 604 further includes:

[0136] The determination submodule 6046 is used to determine, for the first microservice call, a corresponding target transfer parameter according to the historical exception ratios of the first microservice call and its sub-microservice calls, and the historical average number of times the target microservice calls the sub-microservices corresponding to the first call pattern diagram, indicating the probability that the exception of the first microservice call is caused by the first microservice itself and is unrelated to the calling sub-microservice;

[0137] A correction submodule 6048, configured to correct the abnormality score of the first microservice call based at least on the target transfer parameter to obtain a correction score;

[0138] The determination submodule 6046 is further used to determine the abnormal root cause of the target system based on the correction score of each microservice call.

[0139] In one embodiment, the correction submodule 6048 is specifically used to:

[0140] The abnormal score of the first microservice call is used as the initial score, and multiple score transfers are iterated; wherein a single score transfer includes determining a retention ratio according to a target transfer parameter, multiplying the retention ratio by the current score of the first microservice call as the retention score of the first microservice call, and transferring the remaining score obtained by subtracting the retention score from the current score to the sub-microservice call.

[0141] In one embodiment, the number of sub-microservice calls is multiple, and the correction sub-module 6048 is further specifically used to:

[0142] Determine the transfer ratio corresponding to each sub-microservice call according to the out-degree of the first node;

[0143] According to the transfer ratio, the remaining scores are transferred to each sub-microservice call.

[0144] In one embodiment, the determining unit 604 is specifically configured to:

[0145] The target microservice of the microservice call corresponding to the highest abnormal score among the microservice calls is determined as the abnormal microservice.

[0146] The functions of the functional units of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0147] An abnormality root cause locating device for a target system provided by an embodiment of the present specification can quickly and accurately identify abnormal microservice calls.

[0148] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 5 The method described.

[0149] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 5 The method described.

[0150] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the medium or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0151] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this specification. It should be understood that the above description is only the specific implementation method of this specification and is not intended to limit the protection scope of this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of this specification should be included in the protection scope of this specification.

Claims

1. A method for locating the root cause of an abnormality in a target system, comprising: Get the service call graph of the target system, where a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call; For any first microservice call, determine a corresponding first call pattern graph, the first microservice call corresponds to a call pointing to a first node through a first edge in the service call graph, the first call pattern graph includes a preceding node of the first node and a first-order child node; Determine a target call pattern graph that matches the first call pattern graph from each known call pattern graph; Obtaining an estimated execution time of the first microservice call based on the historical execution time distribution of the target call pattern graph; Determining a first anomaly score for the first microservice call according to a difference between an estimated execution time of the first microservice call and an actual observed execution time; The abnormal root cause of the target system is determined according to each abnormal score of each microservice call in the service call graph.

2. The method according to claim 1, wherein: The historical execution time distribution includes: a first Gaussian distribution of a first sub-time that is not related to the calling sub-microservice, and a second Gaussian distribution of a second sub-time that is related to the calling sub-microservice; The obtaining the estimated execution time of the first microservice call includes: The estimated execution time of the first microservice call is obtained according to the means of the first and second Gaussian distributions and the number of times the target microservice corresponding to the first call pattern graph calls the sub-microservice.

3. The method according to claim 2, wherein: The target call pattern diagram is the same as the first call pattern diagram; The step of obtaining the estimated execution time of the first microservice call according to the means of the first and second Gaussian distributions and the number of times the target microservice corresponding to the first call pattern graph calls the sub-microservice includes: The estimated execution time is obtained by superimposing the mean of the first Gaussian distribution and the product of the mean of the second Gaussian distribution and the number of times.

4. The method according to claim 2, wherein: The target call pattern graph includes a plurality of similar call pattern graphs of the first call pattern graph; The step of obtaining the estimated execution time of the first microservice call according to the means of the first and second Gaussian distributions and the number of times the target microservice corresponding to the first call pattern graph calls the sub-microservice includes: Selecting first similar pattern graphs containing the same target microservice as the first call pattern graph from the multiple similar call pattern graphs, and performing weighted synthesis on the means of the corresponding first Gaussian distributions to obtain a first comprehensive value; Selecting each second similar pattern graph of the sub-microservice including the first call pattern graph from the multiple similar call pattern graphs, multiplying the mean of the corresponding second Gaussian distribution by the number of times, and performing weighted synthesis on each product to obtain a second comprehensive value; The first comprehensive value and the second comprehensive value are superimposed to obtain the estimated execution time.

5. The method according to claim 4, wherein: The similar call pattern diagram meets the following conditions: The target microservice of the similar call pattern graph is the same as the first call pattern graph, or the similar call pattern graph includes a sub-microservice of the first call pattern graph; The similarity between the similar call pattern graph and the first call pattern graph is greater than a preset threshold.

6. The method according to claim 4, wherein: The weighted synthesis of the means of the corresponding first Gaussian distributions includes: Using the similarity between the first calling pattern graph and each first similar pattern graph as a weight coefficient, performing weighted synthesis on the means of the corresponding first Gaussian distributions; The weighted integration of the products comprises: The similarity between the first calling pattern graph and each second similar pattern graph is used as a weight coefficient to perform weighted synthesis on the corresponding products.

7. The method according to claim 2, wherein: The determining a first abnormality score of the first microservice call includes: Subtract the estimated execution time from the observed execution time to obtain a first difference value; Using the ratio of the first difference to the target standard deviation as the first anomaly score; The target standard deviation is determined according to the variance of the first and second Gaussian distributions, and the order.

8. The method according to claim 2, wherein: The distribution parameters including the means of the first and second Gaussian distributions are obtained by the following steps: A maximizing EM algorithm or a greedy EM algorithm is adopted to solve the mixed Gaussian distribution satisfied by the first sub-time and the second sub-time based on the observed execution time of each historical microservice call statistics mapped to the target call pattern graph, and obtain the distribution parameters of the first and second Gaussian distributions.

9. The method according to claim 1, wherein: Determining the abnormal root cause of the target system includes: For the first microservice call, according to the historical exception ratios of the first microservice call and its sub-microservice call, and the historical average number of sub-microservice calls by the target microservice corresponding to the first call pattern diagram, a corresponding target transfer parameter is determined, which indicates the probability that the exception of the first microservice call is caused by the first microservice itself and is unrelated to the calling sub-microservice; Based at least on the target transfer parameter, correct the abnormal score of the first microservice call to obtain a corrected score; Based on the correction scores of the respective microservice calls, a root cause of the abnormality of the target system is determined.

10. The method according to claim 9, wherein: The correcting the abnormal score of the first microservice call includes: The abnormal score of the first microservice call is used as the initial score, and multiple score transfers are iterated; wherein a single score transfer includes determining a retention ratio according to the target transfer parameter, multiplying the retention ratio by the current score of the first microservice call as the retention score of the first microservice call, and transferring the remaining score obtained by subtracting the retention score from the current score to the sub-microservice call.

11. The method according to claim 10, wherein: The number of sub-microservice calls is multiple; the transfer to the sub-microservice call includes: Determine, according to the out-degree of the first node, a transfer ratio corresponding to each sub-microservice call; According to the transfer ratio, the remaining score is transferred to each sub-microservice call.

12. The method according to claim 1, wherein: The determining the abnormal root cause of the target abnormality includes: The target microservice of the microservice call corresponding to the microservice call with the highest abnormal score among the microservice calls is determined as the abnormal microservice.

13. A device for locating the root cause of an abnormality in a target system, comprising: An acquisition unit is used to acquire a service call graph of a target system, in which a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call; A determining unit, configured to determine a corresponding first call pattern graph for any first microservice call, wherein the first microservice call corresponds to a call pointing to a first node through a first edge in the service call graph, and the first call pattern graph includes a preceding node of the first node and a first-order child node; The determining unit is further used to determine a target calling pattern graph matching the first calling pattern graph from each known calling pattern graph; The acquisition unit is further configured to obtain an estimated execution time of the first microservice call based on the historical execution time distribution of the target call pattern graph; The determining unit is further configured to determine a first anomaly score of the first microservice call according to a difference between the estimated execution time of the first microservice call and the actual observed execution time; The determining unit is further configured to determine the abnormal root cause of the target system according to each abnormal score of each microservice call in the service call graph.

14. The device according to claim 13, wherein: The historical execution time distribution includes: a first Gaussian distribution of a first sub-time that is not related to the calling sub-microservice, and a second Gaussian distribution of a second sub-time that is related to the calling sub-microservice; The acquisition unit is specifically used for: The estimated execution time of the first microservice call is obtained according to the means of the first and second Gaussian distributions and the number of times the target microservice corresponding to the first call pattern graph calls the sub-microservice.

15. The device according to claim 14, wherein: The target call pattern diagram is the same as the first call pattern diagram; The acquisition unit is further specifically used for: The estimated execution time is obtained by superimposing the mean of the first Gaussian distribution and the product of the mean of the second Gaussian distribution and the number of times.

16. The device according to claim 14, wherein: The target call pattern graph includes a plurality of similar call pattern graphs of the first call pattern graph; The acquisition unit comprises: A selection submodule is used to select first similar pattern graphs whose target microservices are the same as the first call pattern graph from the multiple similar call pattern graphs, and perform weighted synthesis on the means of the corresponding first Gaussian distributions to obtain a first comprehensive value; The selection submodule is further used to select each second similar pattern graph of the sub-microservice containing the first call pattern graph from the multiple similar call pattern graphs, multiply the mean of the corresponding second Gaussian distribution by the number of times, and perform weighted synthesis on each product to obtain a second comprehensive value; The superposition submodule is used to superimpose the first comprehensive value and the second comprehensive value as the estimated execution time.

17. The device according to claim 14, wherein: The determining unit comprises: A difference calculation submodule, used for calculating the difference between the estimated execution time and the observed execution time to obtain a first difference value; an acquisition submodule, configured to use a ratio of the first difference to a target standard deviation as the first anomaly score; The target standard deviation is determined according to the variance of the first and second Gaussian distributions, and the order.

18. The apparatus according to claim 14, further comprising: A solving unit is used to adopt a maximizing EM algorithm or a greedy EM algorithm to solve the mixed Gaussian distribution satisfied by the first sub-time and the second sub-time based on each observed execution time of each historical microservice call statistics mapped to the target call pattern graph, and obtain the distribution parameters of the first and second Gaussian distributions.

19. The device according to claim 13, wherein: The determining unit comprises: a determination submodule, for determining, for the first microservice call, a corresponding target transfer parameter according to the historical exception proportions of the first microservice call and its sub-microservice call, and the historical average number of times the target microservice corresponding to the first call pattern diagram calls the sub-microservice, indicating the probability that the exception of the first microservice call is caused by the first microservice itself and is unrelated to the calling sub-microservice; a correction submodule, configured to correct the abnormality score of the first microservice call based at least on the target transfer parameter to obtain a corrected score; The determination submodule is further used to determine the abnormal root cause of the target system based on the correction score of each microservice call.

20. The device according to claim 19, wherein The correction submodule is specifically used for: The abnormal score of the first microservice call is used as the initial score, and multiple score transfers are iterated; wherein a single score transfer includes determining a retention ratio according to the target transfer parameter, multiplying the retention ratio by the current score of the first microservice call as the retention score of the first microservice call, and transferring the remaining score obtained by subtracting the retention score from the current score to the sub-microservice call.

21. The device according to claim 20, wherein: The number of sub-microservice calls is multiple; the correction submodule is also specifically used for: Determine, according to the out-degree of the first node, a transfer ratio corresponding to each sub-microservice call; According to the transfer ratio, the remaining score is transferred to each sub-microservice call.

22. The device according to claim 13, wherein: The determining unit is specifically used for: The target microservice of the microservice call corresponding to the microservice call with the highest abnormal score among the microservice calls is determined as the abnormal microservice.

23. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 12.

24. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 12 is implemented.

Images