Abnormal root cause positioning method and device for target system

Abstract

Embodiments of the present specification provide a method and device for locating an abnormal root cause of a target system. In the locating method, a service call graph of the target system is obtained. For any first microservice call, a corresponding first call pattern graph is determined, the first microservice call corresponding to a call directed to a first node through a first edge in the service call graph, the first call pattern graph including a predecessor node and a first-order child node of the first node. A target call pattern graph matching the first call pattern graph is determined from each known call pattern graph. Based on a historical execution time distribution of the target call pattern graph, an estimated execution time of the first microservice call is obtained. According to a difference between the estimated execution time and an actual observed execution time of the first microservice call, a first abnormal score of the first microservice call is determined. According to each abnormal score of each microservice call in the service call graph, an abnormal root cause of the target system is determined.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of computer technology, and particularly relate to an abnormal root cause positioning method and device for a target system. BACKGROUND

[0002] In a distributed framework, the target system is usually split into multiple microservices, and there is a complex calling relationship between the microservices. When an exception occurs in a microservice or a call in the calling chain, it will affect the entire calling chain and thus cause an exception of the target system. Only when the abnormal root cause is quickly located, can measures be taken in time to avoid damage to user interests.

[0003] Therefore, it is necessary to provide an effective abnormal root cause positioning scheme for a target system. SUMMARY

[0004] One or more embodiments of the present specification describe an abnormal root cause positioning method for a target system, which can quickly and accurately identify an abnormal microservice call.

[0005] In a first aspect, an abnormal root cause positioning method for a target system is provided, comprising:

[0006] Obtaining a service call graph of the target system, wherein a single directed edge in the service call graph represents a single call between microservices, and a single node represents a microservice in a single call;

[0007] For any first microservice call, determining a corresponding first call pattern graph, the first microservice call corresponds to a call directed to a first node by a first edge in the service call graph, and the first call pattern graph includes the predecessor nodes of the first node and the first-order sub-nodes;

[0008] Determining a target call pattern graph matching the first call pattern graph from the known call pattern graphs;

[0009] Based on the historical execution time distribution of the target call pattern graph, obtaining an estimated execution time of the first microservice call;

[0010] According to the difference between the estimated execution time and the actual observed execution time of the first microservice call, determining a first abnormal score of the first microservice call;

[0011] According to the abnormal scores of each microservice call in the service call graph, determining the abnormal root cause of the target system.

[0012] In a second aspect, an abnormal root cause positioning device for a target system is provided, comprising:

[0013] The acquisition unit is configured to acquire a service call graph of a target system, wherein a single directed edge in the service call graph represents a single call between microservices, and a single node represents a microservice in the single call.

[0014] The determination unit is configured to determine, for any first microservice call, a corresponding first call pattern graph, the first microservice call corresponding to a call directed to a first node by a first edge in the service call graph, the first call pattern graph including a predecessor node of the first node and a first-order child node.

[0015] The determination unit is further configured to determine, from the known call pattern graphs, a target call pattern graph that matches the first call pattern graph.

[0016] The acquisition unit is configured to obtain an estimated execution time of the first microservice call based on a historical execution time distribution of the target call pattern graph.

[0017] The determination unit is further configured to determine a first anomaly score of the first microservice call according to a difference between the estimated execution time and an actual observed execution time of the first microservice call.

[0018] The determination unit is further configured to determine an anomaly root cause of the target system according to the anomaly scores of the microservice calls in the service call graph.

[0019] In a third aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed in a computer, the computer program causes the computer to execute the method of the first aspect.

[0020] In a fourth aspect, a computing device is provided, which includes a memory and a processor. The memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0021] The method for locating an anomaly root cause of a target system provided by one or more embodiments of the present specification calculates an estimated execution time of each microservice call in a service call graph based on a historical execution time distribution of a corresponding known call pattern graph. Then, by comparing the estimated execution time of each microservice call with an actual observed execution time, it is determined whether each microservice call is abnormal. Thus, the abnormal microservice call can be quickly and accurately identified. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present specification, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0023] Figure 1 Service call diagram in one example is shown.

[0024] Figure 2 Service call diagram in one example is shown.

[0025] Figure 3 Service call diagram in one example is shown.

[0026] Figure 4 Service call diagram in one example is shown.

[0027] Figure 5 Abnormal root cause positioning method flow chart for target system according to one embodiment is shown.

[0028] Figure 6 Abnormal root cause positioning method flow chart for target system according to one embodiment is shown. DETAILED DESCRIPTION

[0029] The solutions provided by the present specification will be described below in combination with the drawings.

[0030] There are various reasons for the abnormality of the target system in the production environment, such as service response abnormality, machine abnormality, improper code writing, etc. In order to monitor the health status of the target system and locate the abnormal root cause, the call requests (hereinafter referred to as calls) between each microservice in the target system can be tracked, and then a service call diagram (also known as a distributed trace diagram or tracing visualization) can be obtained. The service call diagram includes multiple nodes and multiple directed edges. A single directed edge in it represents a single call between microservices, and a single node represents a microservice in a single call.

[0031] Figure 1 Service call diagram in one example is shown. Figure 1In the figure, nodes A-F represent six microservices, in which A calls B once, represented by directed edge S1; A calls C once, represented by directed edge S2; B calls D once, represented by directed edge S3; D calls E twice, represented by directed edges S4 and S5 respectively; D calls F twice, represented by directed edges S6 and S7 respectively.

[0032] Generally, the microservice that initiates a call is called an upstream microservice or a parent microservice, and the microservice that is called is called a downstream microservice or a child microservice. For example, when A calls B, A can be called an upstream microservice or a parent microservice, and B can be called a downstream microservice or a child microservice.

[0033] In addition, in the service call graph of the present solution, when a parent microservice calls the same child microservice multiple times, the multiple calls are represented as multiple directed edges, and each directed edge points to a separate node representing the child microservice. In other words, only one edge connects any two nodes in the service call graph.

[0034] Further, the microservice represented by the root node in the service call graph can be called an entry microservice, which is called by an external system or platform (referred to as external call). The external call can also be represented as a directed edge. For example, Figure 1 In the figure, directed edge S0 pointing to node A represents an external call to microservice A.

[0035] It should be noted that a single directed edge in the service call graph also corresponds to an observed response time, which can also be understood as the observed response time of calling the target microservice through the directed edge, i.e., the observed response time of microservice call. The observed response time can be divided into two parts: the observed execution time of microservice call and the observed response time of child microservice call. The child microservice call can be understood as a call to a child microservice by the target microservice in the above microservice call. For example, Figure 1 In the figure, S1 and S2 are child microservice calls of S0, and S3 is a child microservice call of S1.

[0036] The present solution will locate the root cause of the exception based on the above service call graph for the target system, which will be described in detail below.

[0037] Figure 2 An implementation scenario diagram for an embodiment disclosed in the present solution. Figure 2 In the figure, each microservice call S i in the service call graph can be traversed, and the corresponding call pattern graph can be determined. For example, for S0, the call pattern graph Pattern1 can be generated; for example, for S7, the call pattern graph Pattern n .

[0038] Afterwards, for each microservice call S i , a target call pattern graph Q m that matches its call pattern graph can be determined from a set of known call pattern graphs: Q1, Q2, …, Q j N, and based on the historical execution time distribution of the target call pattern graph: θ(Q j ), the estimated execution time of the microservice call: ET(S i ) can be obtained. Then, according to the difference between the estimated execution time: ET(S i ) and the actual observed execution time: ET'(S i ) of each microservice call S i , the corresponding anomaly score: AS(S i ) can be determined. Finally, according to the anomaly scores of each microservice call, the anomaly root cause of the target system can be determined.

[0039] The determination process of each known call pattern graph and its historical execution time distribution is described as follows.

[0040] Each known call pattern graph described in the present solution can be generated by traversing each node in a plurality of historical service call graphs, wherein a single known call pattern graph can include the predecessor nodes and first-order child nodes of the current traversed node.

[0041] Figure 3 The call pattern graph diagram in one example is shown. Figure 3 , based on the node A and its context in the historical service call Figure 1 (or graph N), the call pattern graph Q1 can be obtained, which contains the node A and its first-order child nodes B and C; based on the node B and its context in the historical service call Figure 1 (or graph N), the call pattern graph Q2 can be obtained, which contains the predecessor node A and first-order child node D of the node B; and based on the node D and its context in the historical service call Figure 1 (or graph N), the call pattern graph Q m can be obtained, which contains the predecessor nodes A and B and first-order child nodes e and F of the node D.

[0042] In practice, the known call pattern graph can be represented as a predecessor node list and a child set collection, for example, Q1 = ([A], {B, C}), Q2 = ([A, B], {D}), …, Q m = ([A, B, D], {E, F}).

[0043] According to the above, each known invocation pattern graph is generated for a node, or each known invocation pattern graph corresponds to a node, and hereinafter, the corresponding node contained in each known invocation pattern graph is referred to as a target node. For example, the node A in the invocation pattern graph Q1 is referred to as a target node, and the node D in the invocation pattern graph Q m

[0044] For any known invocation pattern graph described above, the determination process of the historical execution time distribution corresponding thereto can be as follows:

[0045] First, in this scheme, the historical execution time of any known invocation pattern graph is divided into two parts: a first sub-time irrelevant to the invocation of the sub-microservice, and a second sub-time related to the invocation of the sub-microservice. Among them, the sub-microservice here is relative to the target microservice (the microservice represented by the target node) in the known invocation pattern graph. It should be understood that in the case of multiple sub-microservices, the number of the above-mentioned second sub-time is also multiple.

[0046] It should be understood that in the case of the above-mentioned historical execution time being divided into a first sub-time and a second sub-time, the above-mentioned historical execution time distribution includes a first distribution of the first sub-time and a second distribution of the second sub-time. In the case of assuming that the first distribution and the second distribution are Gaussian distributions, the first and second distributions are also referred to as first and second Gaussian distributions.

[0047] In one embodiment, the maximum EM algorithm or the greedy EM algorithm can be used to solve the mixed Gaussian distribution satisfied by the first sub-time and the second sub-time based on the observation execution time of each historical microservice invocation statistics for the known invocation pattern graph, to obtain the distribution parameters of the first and second Gaussian distributions. Among them, the distribution parameters include mean (also referred to as expectation) and variance.

[0048] Of course, in practice, a nonlinear least squares method can also be used to calculate the mean of the first and second Gaussian distributions, which is not limited in this specification.

[0049] Figure 4 A schematic diagram showing the calculation method of the first and second Gaussian distributions in one example is shown. Figure 4 In the above, the left side shows the invocation pattern graph: P m ​The third historical microservice call is ([G, A], {B, C}), and the target microservice of the third historical microservice call is A. It should be understood that for the first historical microservice call, the corresponding observed execution time can be counted as: 25 ms = 88 ms - 21 ms - 12 ms - 11 ms - 19 ms; for the second historical microservice call, the corresponding observed execution time can be counted as: 19 ms = 80 ms - 24 ms - 19 ms - 18 ms; for the third historical microservice call, the corresponding observed execution time can be counted as: 22 ms = 100 ms - 23 ms - 25 ms - 14 ms - 16 ms. Of course, other observed execution times can also be calculated in practice.

[0050] In addition, Figure 4 In the formula, the rectangle represents the first sub-time, which can be specifically represented as R A The two squares respectively represent two second sub-times, wherein the square filled with a horizontal line represents the second sub-time related to the calling of the sub-microservice B, which is specifically represented as C (A,B) The square filled with a vertical line represents the second sub-time related to the calling of the sub-microservice C, which is specifically represented as C (A,C) .

[0051] Specifically, the initial distribution parameters of the Gaussian distributions to which R A , C (A,B) and C (A,C) respectively conform can be determined, and the initial distribution parameters of the three Gaussian distributions are used as initial coefficients to construct a mixed Gaussian distribution model. Based on the observed execution times: 19 ms, 18 ms and 16 ms, etc., the EM algorithm or the greedy EM algorithm is used to iteratively calculate the coefficients of the mixed Gaussian distribution model to obtain the final coefficients of the model. According to the final coefficients, the final distribution parameters of the Gaussian distributions to which R A , C (A,B) and C (A,C) respectively conform are determined. The two distribution parameters of the Gaussian distribution to which R A conforms can be represented as: (t r (A), σ r (A) 2 ); the two distribution parameters of the Gaussian distribution to which C (A,B) conforms can be represented as: (t c (A, B), σ c (A, B) 2 ); and the two distribution parameters of the Gaussian distribution to which C (A,C) conforms can be represented as: (t c (A, C), σ c (A, C) 2 .

[0052] Similarly, the historical execution time distribution of each known call pattern graph can be determined.

[0053] Figure 5 This diagram illustrates a method for locating the root cause of anomalies in a target system according to one embodiment. This method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. Figure 5 As shown, the method may include the following steps:

[0054] Step S502: Obtain the service call graph of the target system, where a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call.

[0055] The service call graph here can be as follows: Figure 1 As shown, any two nodes are connected by only one edge, and the root node also has a directed edge pointing to it. Furthermore, each directed edge in the service call graph corresponds to an observed response time, which can be understood as the observed response time of the target microservice called through that directed edge, i.e., the observed response time of the microservice call. This observed response time can be divided into two parts: the observed execution time of the microservice call and the observed response time of the sub-microservice call. The sub-microservice call can be understood as the call from the target microservice to the sub-microservice within the aforementioned microservice call. For example, Figure 1 In the diagram, S1 and S2 are sub-microservice calls of S0, and S3 is a sub-microservice call of S1.

[0056] Step S504: For any first microservice call, determine the corresponding first call pattern diagram. The first microservice call corresponds to the call in the service call diagram that points to the first node through the first edge. The first call pattern diagram includes the predecessor node of the first node and the first-order child node.

[0057] Here, the first edge can be any directed edge in the service call graph, and the first node is the node pointed to by the first edge. For example, the first microservice call here could be... Figure 1 In this scheme, the microservice called in the first microservice call is called the target microservice, that is, the microservice represented by the first node is the target microservice.

[0058] Specifically, the corresponding first call pattern diagram can be determined based on the first node's predecessor node and its first-order child nodes in the service call graph. Taking the first microservice call as an example where the call is made through S3 pointing to D, the corresponding first call pattern diagram could be P. i =([A,B,D],{E,F}).

[0059] Step S506: Determine the target call pattern diagram that matches the first call pattern diagram from each known call pattern diagram.

[0060] Specifically, the first call pattern diagram P can be... i With each known call pattern diagram Q1, Q2, ..., Q m A comparison is performed. If a known call pattern diagram that matches the target call pattern diagram is found, that known call pattern diagram is used as the target call pattern diagram. If no known call pattern diagram that matches the target call pattern diagram is found, a similar call pattern diagram to the first call pattern diagram is selected from the known call pattern diagrams as the target call pattern diagram.

[0061] It should be noted that, as mentioned earlier, the calling pattern graph can be represented as a list of preceding nodes and a set of subsets. Therefore, the consistency mentioned in this scheme means that both the list of preceding nodes and the set of subsets are consistent.

[0062] Furthermore, the aforementioned similar call pattern diagram can satisfy the following conditions: the target microservice of the similar call pattern diagram is the same as that of the first call pattern diagram, or the similar call pattern diagram contains a sub-microservice of the first call pattern diagram; and the similarity between the similar call pattern diagram and the first call pattern diagram is greater than a preset threshold. In one example, the similarity here is the reciprocal of the edit distance, which refers to the minimum number of edit operations required to transform one encoding group into another; the greater the distance, the more different they are. The edit operations can include replacing one encoding with another, inserting an encoding, and deleting an encoding.

[0063] Using the first call pattern diagram P i Taking the example of Q = ([D],{F,K}), where the target microservice is D, and its two sub-microservices are F and K, the corresponding similar call pattern diagrams could include: Q1 = ([D],{E,F}), Q2 = ([D],{E,F,G}), Q3 = ([M],{E,K}), and Q4 = ([J],{F,K}), etc. Among these, Q1 and Q2 are related to P... i They contain the same target microservice D. Additionally, Q1, Q2, and Q4 contain P. i The sub-microservices F, Q3, and Q4 contain P i The sub-microservice K in the middle.

[0064] Step S508: Based on the historical execution time distribution of the target call pattern graph, obtain the estimated execution time of the first microservice call.

[0065] As mentioned earlier, the historical execution time distribution here includes: the first Gaussian distribution of the first sub-time and the second Gaussian distribution of the second sub-time, and there can be multiple second Gaussian distributions.

[0066] Specifically, the estimated execution time of the first microservice call can be obtained according to the mean values of the first and second Gaussian distributions, and the number of times that the target microservice (i.e., the microservice represented by the first node) calls the sub-microservice corresponding to the first call pattern graph in the above service call graph.

[0067] In the case where the target call pattern graph is the same as the first call pattern graph, the mean value of the first Gaussian distribution can be superimposed with the product of the mean value of the second Gaussian distribution and the number of times that the target microservice calls the sub-microservice, to obtain the estimated execution time of the first microservice call.

[0068] Also take the first call pattern graph P i = ([D], {F, K}) as an example. Assuming that D calls F twice and calls K three times in the service call graph, in the case where the target call pattern graph Q j = ([D], {F, K}) can be selected, ET(S i ) = t r (D) + 2t c (D, F) + 3t c (D, K). Wherein, t r (D) is the mean value of the first Gaussian distribution of Q j , t c (D, F) is the mean value of the second Gaussian distribution of Q j related to D calling F, t c (D, K) is the mean value of the second Gaussian distribution of Q j related to D calling K, and ET(S i ) is the estimated execution time of the first microservice call.

[0069] In addition, in the case where the target call pattern graph includes multiple similar call pattern graphs of the first call pattern graph, first, select each first similar pattern graph containing the same target microservice as the first call pattern graph from the multiple similar call pattern graphs, and weight and combine the mean values of the corresponding first Gaussian distributions to obtain a first combined value. And select each second similar pattern graph containing the sub-microservice of the first call pattern graph from the multiple similar call pattern graphs, respectively multiply the mean values of the corresponding second Gaussian distributions by the number of times that the target microservice calls the sub-microservice, and weight and combine each product to obtain a second combined value. Finally, superimpose the first combined value and the second combined value as the estimated execution time of the first microservice call.

[0070] Also take the first call pattern graph P iFor example, and assuming that in the service call graph, D calls F twice and K three times, then the multiple similar call pattern graphs include: Q1 = ([D], {E, F}) and Q2 = ([D], {E, F, G}) and Q3 = ([M], {E, K}) and Q4 = ([J], {F, K}). The selected first similar pattern graphs can include: Q1 and Q2, and thus the first comprehensive value The similarity of Q1 and Q2 with P i , respectively, The mean of the first Gaussian distribution of Q1 and Q2, respectively.

[0071] In addition, since the first call pattern graph includes two child nodes F and K, two groups of second similar pattern graphs can be selected for the two child nodes: (v1, Q2, v4) and (Q3, Q4). It should be noted that for each group of selected second similar pattern graphs, further screening can be performed, for example, the second similar pattern graphs whose target microservice is also the same as the first call pattern graph are screened out. For example, the two groups of second similar pattern graphs finally used are: (Q1, Q2) and (Q3, Q4).

[0072] After the two groups of second similar pattern graphs are selected, based on (Q1, Q2), a second comprehensive value can be calculated, wherein The similarity of Q1 and Q2 with P i , respectively, The mean of the second Gaussian distribution of Q1 and Q2, respectively, related to D calling F. And based on (Q3, Q4), another second comprehensive value can be calculated, wherein The similarity of Q3 and Q4 with P i , respectively, The mean of the second Gaussian distribution of Q3, related to M calling K, The mean of the second Gaussian distribution of Q4, related to J calling K.

[0073] Finally, ET(S i ) = Com1 + Com21 + Com22.

[0074] Similarly, the estimated execution time of each microservice call can be calculated.

[0075] In step S510, a first anomaly score of the first microservice call is determined according to the difference between the estimated execution time of the first microservice call and the actual observed execution time.

[0076] As mentioned above, any microservice call has a corresponding observed response time, which can be decomposed into the observed execution time of the service call and the observed response times of the sub-microservice calls, so that the actual observed execution time of the first microservice call is the difference between the observed response time of the first microservice call and the observed response time of the sub-microservice calls.

[0077] For example, the actual observed execution time of the microservice call S3 in the first calling pattern graph Figure 1 wherein, is the observed response time of the microservice call S3, is the observed response time of each sub-microservice call thereof, respectively. Similarly, the observed execution time of each microservice call can be calculated.

[0078] In one embodiment, the difference between the estimated execution time and the observed execution time of each microservice call can be used as the anomaly score of each microservice call.

[0079] In another embodiment, for any microservice call, the difference between the estimated execution time and the observed execution time can be calculated, and the ratio of the difference to a target standard deviation can be used as the corresponding anomaly score. The target standard deviation can be determined according to the variances of the first and second Gaussian distributions and the number of target microservice calls in the corresponding calling pattern graph.

[0080] In one example, the anomaly score of any microservice call S i may be calculated according to the following formula:

[0081]

[0082] wherein, ET(S i ) is the estimated execution time of the microservice call S i , ET'(S i ) is the observed execution time of the microservice call S i , ε is a smoothing factor, AS(S i ) is the anomaly score of the microservice call S i , and σ ET is the target standard deviation, which can be determined according to the variances of the first and second Gaussian distributions of the same or similar calling pattern graphs of the first calling pattern graph.

[0083] In one example, σ ET may be calculated according to the following formula:

[0084]

[0085] wherein, P i ​S i The corresponding call pattern diagram, assuming P i = ([D], {F, K}) if a target call pattern diagram: Q j = ([D], {F, K}) can be selected, then wherein, is a variance of a first Gaussian distribution of Q j is a variance of a second Gaussian distribution of Q j related to D calling F, is a variance of a second Gaussian distribution of Q j related to D calling K.

[0086] And if only multiple similar call pattern diagrams: Q1= ([D], {E, F}), Q2= ([D], {E, F, G}), Q3= ([M], {E, K}) and Q4= ([J], {F, K}) are selected, then wherein, Com1', Com21', and Com22' can be specifically obtained by replacing the mean values in Com1, Com21, and Com22 with corresponding variance calculations, and the description will not be repeated here.

[0087] In step S512, the abnormal root cause of the target system is determined according to the abnormal scores of each microservice call in the service call graph.

[0088] In one embodiment, the abnormal root cause of the target system can be determined according to the microservice call with the highest corresponding abnormal score in each microservice call. For example, the target microservice of the microservice call with the highest abnormal score is determined as the abnormal microservice.

[0089] In another embodiment, the abnormal scores of each microservice call can also be corrected first, and then the abnormal root cause of the target system is determined based on the corrected scores of each microservice call.

[0090] Taking any microservice call (hereinafter referred to as a first microservice call) as an example, the corresponding correction process can be as follows:

[0091] According to the historical abnormality proportion of the first microservice call and its child microservice calls, and the historical average number of times of the target microservice call child microservice corresponding to the first call pattern diagram, a target transition parameter is determined, which indicates the probability that the abnormality of the first microservice call is caused by itself and unrelated to the call child microservice. At least based on the target transition parameter, the abnormal score of the first microservice call is corrected to obtain a corrected score.

[0092] ​Specifically, the observed response times of each historical microservice call mapped to the first call pattern graph (i.e., the call pattern graph determined based on the first microservice call) can be counted, and then based on the 3σ check method or the like, the abnormal observed response times can be determined therefrom, and the proportion of the abnormal observed response times can be counted as the historical abnormal proportion of the first microservice call. Similarly, the historical abnormal proportion of the sub-microservice call can be counted.

[0093] In one example, the target transfer number of any microservice call S i may be calculated according to the following formula:

[0094]

[0095] wherein p e (S i ) is the historical abnormal proportion of the microservice call S i , S j is the sub-microservice call of S i , p e (S j ) is the historical abnormal proportion of the sub-microservice call S j , P i is the call pattern graph corresponding to S i , and is the historical average number of times that the microservice ms(S i ) calls the sub-microservice ms(S j ) corresponding to P i .

[0096] After the corresponding target transfer number is calculated for the first microservice call, the abnormal score of the first microservice call can be taken as the initial score, and multiple score transfers can be iteratively performed; wherein a single score transfer includes determining a retention ratio according to the target transfer parameter, taking the product of the retention ratio and the current score of the first microservice call as the reserved score of the first microservice call, and transferring the remaining score obtained by subtracting the reserved score from the current score to the sub-microservice call.

[0097] As mentioned before, the number of sub-microservice calls of any microservice call can be multiple, and the number of sub-microservice calls is related to the out-degree of the target node of the microservice call. Thus, in the case that the number of sub-microservice calls is multiple, the transfer ratio corresponding to each sub-microservice call can be determined according to the out-degree. For example, when the out-degree is 5, the corresponding transfer ratio can be 20%. Then, the remaining score is transferred to each sub-microservice call according to the transfer ratio. For example, each microservice call can obtain 20% of the remaining score.

[0098] It should be understood that after the end of the above-mentioned one round of iteration, the current score of the first microservice call is updated to its reserved score, and the current score of each sub-microservice call is updated to its current score plus the transfer score, for the next round of iteration.

[0099] After multiple score transfers, the revised scores of the respective microservice calls can be obtained, and then the abnormal root cause of the target system can be determined according to the microservice call corresponding to the highest revised score.

[0100] In one example, determining the abnormal root cause of the target system includes determining the target microservice corresponding to the microservice call with the highest revised score as the abnormal microservice.

[0101] Of course, in practice, the abnormal root cause of the target system can also be determined according to each microservice call corresponding to a revised score greater than a preset threshold, which is not limited in the present specification.

[0102] In summary, the abnormal root cause positioning method for the target system provided by the embodiments of the present specification can mine the call pattern graph by analyzing the historical service call graph and calculate the corresponding historical execution time distribution. Then, when positioning the abnormal root cause of the new service call graph, the estimated execution time of each microservice call can be estimated by referring to the same or similar call pattern graph, which improves the algorithm performance and can also be used for comparison with the actual observed execution time to determine the abnormal score of the microservice call, so that the present scheme has the advantages of low performance overhead and high efficiency. In addition, the present scheme is based on the link call information and microservice call response time obtained in advance at the data level, without the need to extract data from log data, which improves the performance of the algorithm. Finally, the present scheme can realize root cause positioning of the target system based on a single service call graph, which makes the processing flow of root cause analysis of the target system more efficient and reduces the dependence on abnormal data.

[0103] Corresponding to the above-mentioned abnormal root cause positioning method for the target system, one embodiment of the present specification also provides an abnormal root cause positioning device for the target system, as shown in Figure 6 The device can include:

[0104] The acquisition unit 602 is configured to acquire a service call graph of a target system, wherein a single directed edge in the service call graph represents a single call between microservices, and a single node represents a microservice in a single call.

[0105] The determination unit 604 is configured to determine, for any first microservice call, a corresponding first call pattern graph, the first microservice call corresponding to a call directed to a first node through a first edge in the service call graph, the first call pattern graph including the predecessor nodes of the first node and the first-order sub-nodes.

[0106] The determination unit 604 is further configured to determine a target invocation pattern graph matching the first invocation pattern graph from the known invocation pattern graphs.

[0107] The acquisition unit 602 is further configured to obtain an estimated execution time of the first microservice invocation based on a historical execution time distribution of the target invocation pattern graph.

[0108] The determination unit 604 is further configured to determine a first anomaly score of the first microservice invocation according to a difference between the estimated execution time and an actual observed execution time of the first microservice invocation.

[0109] The determination unit 604 is further configured to determine an anomaly root cause of the target system according to the anomaly scores of the microservice invocations in the service invocation graph.

[0110] In an embodiment, the historical execution time distribution includes a first Gaussian distribution of a first sub-time irrelevant to invoking a sub-microservice and a second Gaussian distribution of a second sub-time relevant to invoking the sub-microservice.

[0111] The acquisition unit 602 is specifically configured to:

[0112] The estimated execution time of the first microservice invocation is obtained according to the means of the first and second Gaussian distributions and the number of times of invoking the sub-microservice by the target microservice corresponding to the first invocation pattern graph.

[0113] In an embodiment, the target invocation pattern graph is the same as the first invocation pattern graph.

[0114] The acquisition unit 602 is further specifically configured to:

[0115] The mean of the first Gaussian distribution is superimposed with the product of the mean of the second Gaussian distribution and the number of times to obtain the estimated execution time.

[0116] In another embodiment, the target invocation pattern graph includes a plurality of similar invocation pattern graphs of the first invocation pattern graph.

[0117] The acquisition unit 602 includes:

[0118] The selection submodule 6022 is configured to select, from the plurality of similar invocation pattern graphs, each first similar pattern graph containing the same target microservice as the first invocation pattern graph, and to weight and combine the means of the corresponding first Gaussian distributions to obtain a first combined value.

[0119] The selection submodule 6022 is further configured to select, from the plurality of similar invocation pattern graphs, each second similar pattern graph containing the sub-microservice of the first invocation pattern graph, to multiply the mean of the corresponding second Gaussian distribution by the number of times, and to weight and combine each product to obtain a second combined value.

[0120] The superposition submodule 6024 is configured to superimpose the first comprehensive value and the second comprehensive value as the estimated execution time.

[0121] In an embodiment, the similar invocation pattern graph satisfies the following conditions:

[0122] The target microservice of the similar invocation pattern graph is the same as the first invocation pattern graph, or the similar invocation pattern graph contains a sub-microservice of the first invocation pattern graph.

[0123] The similarity between the similar invocation pattern graph and the first invocation pattern graph is greater than a preset threshold.

[0124] In an embodiment,

[0125] The selection submodule 6022 is specifically configured to:

[0126] The similarity between the first invocation pattern graph and each first similar pattern graph is used as a weight coefficient to weight and superimpose the mean value of each first Gaussian distribution.

[0127] The selection submodule 6022 is specifically configured to:

[0128] The similarity between the first invocation pattern graph and each second similar pattern graph is used as a weight coefficient to weight and superimpose each product.

[0129] In an embodiment, the determination unit 604 includes:

[0130] The difference submodule 6042 is configured to obtain a first difference value by subtracting the observed execution time from the estimated execution time.

[0131] The acquisition submodule 6044 is configured to use the ratio of the first difference value to the target standard deviation as the first anomaly score.

[0132] The target standard deviation is determined according to the variances of the first and second Gaussian distributions and the number of times.

[0133] In an embodiment, the apparatus further includes:

[0134] The solving unit 606 is configured to use a maximum EM algorithm or a greedy EM algorithm to solve the mixed Gaussian distribution satisfied by the first sub-time and the second sub-time based on the observed execution times of the historical microservice invocations mapped to the target invocation pattern graph, to obtain the distribution parameters of the first and second Gaussian distributions.

[0135] In an embodiment, the determination unit 604 further includes:

[0136] The determining sub-module 6046 is configured to determine, for the first microservice call, a target transition parameter corresponding to the first microservice call according to the historical abnormality proportion of the first microservice call and its child microservice calls respectively, and the historical average number of times of the target microservice call child microservice corresponding to the first call mode graph, the target transition parameter indicating a probability that the abnormality of the first microservice call is caused by itself and irrelevant to the child microservice.

[0137] The correcting sub-module 6048 is configured to correct the abnormality score of the first microservice call based on at least the target transition parameter to obtain a corrected score.

[0138] The determining sub-module 6046 is further configured to determine the abnormality root cause of the target system based on the corrected scores of the microservice calls.

[0139] In one embodiment, the correcting sub-module 6048 is specifically configured to:

[0140] The abnormality score of the first microservice call is taken as an initial score, and multiple times of score transition are iteratively performed; wherein a single time of score transition includes determining a retention ratio according to the target transition parameter, taking a product of the retention ratio and a current score of the first microservice call as a retained score of the first microservice call, and transferring a residual score obtained by subtracting the retained score from the current score to the child microservice call.

[0141] In one embodiment, the number of child microservice calls is multiple, and the correcting sub-module 6048 is further configured to:

[0142] The transition ratio corresponding to each child microservice call is determined according to the out-degree of the first node;

[0143] The residual score is transferred to each child microservice call according to the transition ratio.

[0144] In one embodiment, the determining unit 604 is specifically configured to:

[0145] The target microservice of the microservice call with the highest abnormality score in each microservice call is determined as the abnormal microservice.

[0146] The functions of each functional unit of the above-mentioned embodiment device can be realized through each step of the above-mentioned method embodiment, therefore, the specific working process of the device provided by one embodiment of the present specification will not be repeated here.

[0147] The abnormality root cause positioning device for the target system provided by one embodiment of the present specification can quickly and accurately identify the abnormal microservice call.

[0148] According to an embodiment of the aspect, there is also provided a computer readable storage medium having stored thereon a computer program, which causes a computer to perform the method described in connection with Figure 5 the described method.

[0149] According to an embodiment of the aspect, there is also provided a computer readable storage medium having stored thereon a computer program, which causes a computer to perform the method described in connection with Figure 5 the described method.

[0150] The various embodiments in the specification are described in progressive manner, and the same or similar parts between the various embodiments can be mutually referred to. Each embodiment focuses on the difference from other embodiments. In particular, the medium or device embodiments are described more simply because they are basically similar to the method embodiments. The relevant parts can be referred to the part of the method embodiments.

[0151] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0152] The above detailed description of the specific embodiments of the specification further describes the purpose, technical solutions and beneficial effects of the specification. It should be understood that the above description is only for the specific embodiments of the specification and is not used to limit the protection scope of the specification. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the specification shall be included in the protection scope of the specification.

Claims

1. A method for locating the root cause of anomalies in a target system, comprising: Obtain the service call graph of the target system, where a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call; For any first microservice call, determine the corresponding first call pattern diagram. The first microservice call corresponds to the call in the service call diagram that points to the first node through the first edge. The first call pattern diagram includes the predecessor node of the first node and the first-order child node. From each known call pattern diagram, determine the target call pattern diagram that matches the first call pattern diagram; Based on the historical execution time distribution of the target call pattern graph, the estimated execution time of the first microservice call is obtained; Based on the difference between the estimated execution time and the actual observed execution time of the first microservice call, a first anomaly score for the first microservice call is determined; Based on the exception scores of each microservice call in the service call graph, the root cause of the exception in the target system is determined.

2. The method according to claim 1, wherein, The historical execution time distribution includes: a first Gaussian distribution of the first sub-time that is unrelated to the calling sub-microservice, and a second Gaussian distribution of the second sub-time that is related to the calling sub-microservice; Obtaining the estimated execution time of the first microservice call includes: The estimated execution time of the first microservice call is obtained based on the mean of the first and second Gaussian distributions and the number of times the target microservice calls the sub-microservices corresponding to the first call pattern diagram.

3. The method according to claim 2, wherein, The target call pattern diagram is the same as the first call pattern diagram; The step of obtaining the estimated execution time of the first microservice call based on the mean of the first and second Gaussian distributions and the number of times the target microservice calls the sub-microservices corresponding to the first call pattern graph includes: The estimated execution time is obtained by superimposing the product of the mean of the first Gaussian distribution, the mean of the second Gaussian distribution, and the number of times.

4. The method according to claim 2, wherein, The target call pattern graph includes multiple similar call pattern graphs of the first call pattern graph; The step of obtaining the estimated execution time of the first microservice call based on the mean of the first and second Gaussian distributions and the number of times the target microservice calls the sub-microservices corresponding to the first call pattern graph includes: From the multiple similar call pattern diagrams, select each first similar pattern diagram containing the same target microservice as the first call pattern diagram, and perform a weighted summation on the mean of each corresponding first Gaussian distribution to obtain a first summation value; From the plurality of similar call pattern diagrams, select each second similar pattern diagram of the sub-microservice that contains the first call pattern diagram, multiply the mean of the corresponding second Gaussian distribution by the number, and perform weighted summation on each product to obtain the second summation value; The sum of the first composite value and the second composite value is used as the estimated execution time.

5. The method according to claim 4, wherein, The similar call pattern graph satisfies the following conditions: The target microservice of the similar call pattern diagram is the same as that of the first call pattern diagram, or the similar call pattern diagram contains a sub-microservice of the first call pattern diagram; The similarity between the similar call pattern diagram and the first call pattern diagram is greater than a preset threshold.

6. The method according to claim 4, wherein, The weighted summation of the means of the corresponding first Gaussian distributions includes: The similarity between the first call pattern diagram and each first similar pattern diagram is used as a weight coefficient to perform a weighted summation of the mean values ​​of the corresponding first Gaussian distributions. The weighted summation of the products includes: The similarity between the first call pattern diagram and each of the second similar pattern diagrams is used as a weight coefficient to perform a weighted summation on the corresponding products.

7. The method according to claim 2, wherein, Determining the first anomaly score of the first microservice call includes: The first difference is obtained by subtracting the estimated execution time from the observed execution time. The ratio of the first difference to the target standard deviation is taken as the first anomaly score; The target standard deviation is determined based on the variances of the first and second Gaussian distributions and the degree.

8. The method according to claim 2, wherein, The distribution parameters, including the mean, of the first and second Gaussian distributions are obtained through the following steps: Using the Maximize EM algorithm or the Greedy EM algorithm, based on the observed execution times of each historical microservice call statistics mapped to the target call pattern graph, the mixture Gaussian distribution satisfied by the first sub-time and the second sub-time is solved to obtain the distribution parameters of the first and second Gaussian distributions.

9. The method according to claim 1, wherein, Determining the root cause of the anomaly in the target system includes: For the first microservice call, based on the historical anomaly percentage of the first microservice call and its sub-microservice calls, and the historical average number of times the target microservice calls the sub-microservices corresponding to the first call pattern diagram, a corresponding target transfer parameter is determined, which indicates the probability that the anomaly of the first microservice call is caused by itself and is unrelated to the call to the sub-microservices. Based at least on the target transfer parameters, the abnormal score of the first microservice call is corrected to obtain a corrected score; Based on the corrected scores of each microservice call, the root cause of the anomaly in the target system is determined.

10. The method according to claim 9, wherein, The correction of the abnormal score of the first microservice call includes: The abnormal score of the first microservice call is used as the initial score, and multiple score transfers are performed iteratively. Each score transfer includes determining a retention ratio based on the target transfer parameter, using the product of the retention ratio and the current score of the first microservice call as the retention score of the first microservice call, and transferring the remaining score obtained by subtracting the retention score from the current score to the sub-microservice call.

11. The method according to claim 10, wherein, The number of sub-microservice calls is multiple; the transfer to sub-microservice calls includes: Based on the out-degree of the first node, determine the transfer ratio corresponding to each sub-microservice call; According to the transfer ratio, the remaining score is transferred to each of the sub-microservice calls.

12. The method according to claim 1, wherein, Determining the root cause of the target anomaly includes: The target microservice of the microservice call with the highest anomaly score among all the microservice calls is identified as the abnormal microservice.

13. A device for locating the root cause of an anomaly in a target system, comprising: The acquisition unit is used to acquire the service call graph of the target system, where a single directed edge represents a single call between microservices, and a single node represents a microservice in a single call; The determining unit is used to determine the corresponding first call pattern diagram for any first microservice call, wherein the first microservice call corresponds to the call in the service call diagram that points to the first node through the first edge, and the first call pattern diagram includes the predecessor node of the first node and the first-order child node. The determining unit is further configured to determine a target call pattern diagram that matches the first call pattern diagram from each known call pattern diagram; The acquisition unit is further configured to obtain the estimated execution time of the first microservice call based on the historical execution time distribution of the target call pattern graph; The determining unit is further configured to determine a first anomaly score for the first microservice call based on the difference between the estimated execution time and the actual observed execution time of the first microservice call. The determining unit is further configured to determine the root cause of the anomaly in the target system based on the anomaly scores of each microservice call in the service call graph.

14. The apparatus according to claim 13, wherein, The historical execution time distribution includes: a first Gaussian distribution of the first sub-time that is unrelated to the calling sub-microservice, and a second Gaussian distribution of the second sub-time that is related to the calling sub-microservice; The acquisition unit is specifically used for: The estimated execution time of the first microservice call is obtained based on the mean of the first and second Gaussian distributions and the number of times the target microservice calls the sub-microservices corresponding to the first call pattern diagram.

15. The apparatus according to claim 14, wherein, The target call pattern diagram is the same as the first call pattern diagram; The acquisition unit is also specifically used for: The estimated execution time is obtained by superimposing the product of the mean of the first Gaussian distribution, the mean of the second Gaussian distribution, and the number of times.

16. The apparatus according to claim 14, wherein, The target call pattern graph includes multiple similar call pattern graphs of the first call pattern graph; The acquisition unit includes: The selection submodule is used to select from the multiple similar call pattern diagrams each first similar pattern diagram containing the same target microservice as the first call pattern diagram, and to perform weighted summation on the mean of each corresponding first Gaussian distribution to obtain a first summation value; The selection submodule is further configured to select each second similar pattern diagram of the sub-microservice containing the first call pattern diagram from the plurality of similar call pattern diagrams, multiply the mean of the corresponding second Gaussian distribution by the number, and perform weighted summation on each product to obtain a second comprehensive value; The superposition submodule is used to superimpose the first comprehensive value and the second comprehensive value as the estimated execution time.

17. The apparatus according to claim 14, wherein, The determining unit includes: The difference calculation submodule is used to calculate the difference between the estimated execution time and the observed execution time to obtain a first difference value; The acquisition submodule is used to take the ratio of the first difference to the target standard deviation as the first anomaly score; The target standard deviation is determined based on the variances of the first and second Gaussian distributions and the degree.

18. The apparatus of claim 14, further comprising: The solution unit is used to solve the mixture Gaussian distribution satisfied by the first sub-time and the second sub-time based on the observation execution time of each historical microservice call statistics mapped to the target call pattern graph using the maximization EM algorithm or the greedy EM algorithm, and obtain the distribution parameters of the first and second Gaussian distributions.

19. The apparatus according to claim 13, wherein, The determining unit includes: The determination submodule is used to determine the corresponding target transfer parameter for the first microservice call based on the historical anomaly ratio of the first microservice call and its sub-microservice calls, as well as the historical average number of times the target microservice calls the sub-microservices corresponding to the first call pattern diagram. The parameter indicates the probability that the anomaly of the first microservice call is caused by itself and is unrelated to the call to the sub-microservices. The correction submodule is used to correct the abnormal score of the first microservice call based at least on the target transfer parameters, and obtain a corrected score; The determining submodule is also used to determine the root cause of the anomaly in the target system based on the corrected scores of each microservice call.

20. The apparatus according to claim 19, wherein, The correction submodule is specifically used for: The abnormal score of the first microservice call is used as the initial score, and multiple score transfers are performed iteratively. Each score transfer includes determining a retention ratio based on the target transfer parameter, using the product of the retention ratio and the current score of the first microservice call as the retention score of the first microservice call, and transferring the remaining score obtained by subtracting the retention score from the current score to the sub-microservice call.

21. The apparatus according to claim 20, wherein, The number of sub-micro-service calls is multiple; the correction sub-module is also specifically used for: Based on the out-degree of the first node, determine the transfer ratio corresponding to each sub-microservice call; According to the transfer ratio, the remaining score is transferred to each of the sub-microservice calls.

22. The apparatus according to claim 13, wherein, The determining unit is specifically used for: The target microservice of the microservice call with the highest anomaly score among all the microservice calls is identified as the abnormal microservice.

23. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method according to any one of claims 1-12.

24. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-12.

Images