Microservice abnormal root cause positioning method and system

By obtaining Trace from the log library, building a Trace tree, calculating exception scores, and filtering out the Span with the highest exception scores, it solves the problem of slow positioning speed and low accuracy of microservice exception root in the existing technology, and achieves fast and accurate Abnormal root cause location.

CN119025310AActive Publication Date: 2024-11-26ZHEJIANG UNIV

Patent Information

Application Number
CN202411005152.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-26
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing microservice abnormal root cause location methods are slow and have low accuracy, making it difficult to quickly and accurately locate abnormal root cause in distributed systems.

Method used

By obtaining Trace from the log library, eliminating exception Trace, building a Trace tree, calculating the execution time of Span, grouping statistics, calculating exception scores, filtering out the Span with the highest exception score as the root cause Span, and analyzing it to realize microservices Abnormal root cause location.

Benefits of technology

It realizes rapid and accurate positioning of the root cause of microservice abnormalities, and improves the efficiency and accuracy of abnormal positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025310A_ABST
    Figure CN119025310A_ABST
Patent Text Reader

Abstract

The invention discloses a micro-service abnormal root cause positioning method, which comprises the following steps of: obtaining a plurality of Traces from a log library, and removing abnormal Traces from the plurality of Traces based on a preset abnormal Trace set to obtain a first preliminary normal Trace set; checking whether each Trace in the first preliminary normal Trace set is complete or not, and rejecting incomplete Traces to obtain a second preliminary normal Trace set; constructing a Trace tree for each Trace in the second preliminary normal Trace set, and calculating the own execution time of each Span in each Trace tree to obtain a third preliminary normal Trace set; based on the service name and the Span name, performing grouping statistics on the third preliminary normal Trace set to obtain abnormal Trace in the third preliminary normal Trace set; the abnormal score of each Span in the abnormal Trace is calculated; screening out the Span with the highest abnormal score from the abnormal scores as a root cause Span; and analyzing the root cause Span to obtain the abnormal root cause of the micro-service, so that the abnormal root cause can be quickly and accurately positioned.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of microservice technology, and more particularly to a method and system for locating root causes of microservice anomalies. Background Art

[0002] In today's information technology field, cloud native and microservice systems have become the main trends in business development and deployment. The advantage of this architecture is that it provides greater flexibility and scalability, allowing enterprises to better adapt to rapidly changing market demands. However, as microservice systems continue to expand, the maintenance of microservices and the location of abnormal root causes have become increasingly complex. In a microservice system, a business request usually traverses multiple microservices, which are distributed in different containers, virtual machines, or physical machines. Although the decomposition of microservices improves the modularity and maintainability of the system, it also introduces new challenges, especially in locating the root causes of abnormalities.

[0003] Most of the existing methods for locating the root cause of microservice anomalies have the problems of slow speed and low accuracy.

[0004] Therefore, how to provide a microservice abnormality root cause location method and system, which can quickly and accurately locate the abnormality root cause is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a method and system for locating the root cause of a microservice anomaly, which can quickly and accurately locate the root cause of the anomaly.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for locating the root cause of a microservice exception includes the following steps:

[0008] S1: Obtain several traces from the log library, and remove abnormal traces from the several traces based on a preset abnormal trace set to obtain a first preliminary normal trace set;

[0009] S2: Check whether each trace in the first preliminary normal trace set is complete, remove incomplete traces, and obtain a second preliminary normal trace set; wherein, whether each trace is complete includes whether each span in the trace is repeated or missing;

[0010] S3: constructing a trace tree for each trace in the second preliminary normal trace set, and calculating the execution time of each span in each trace tree, to obtain a third preliminary normal trace set;

[0011] S4: performing grouping and statistics on the third preliminary normal Trace set based on the service name and the span name to obtain abnormal traces in the third preliminary normal Trace set;

[0012] S5: Calculate the abnormal score of each span in the abnormal trace;

[0013] S6: Filter out the span with the highest anomaly score from each anomaly score as the root cause span;

[0014] S7: Analyze the root cause Span to obtain the root cause of the microservice exception.

[0015] Preferably, S4 specifically includes:

[0016] S41: Group all traces with the same service name and span name into one group;

[0017] S42: Statistically calculate the duration of all spans included in each group and the execution time of all spans themselves to obtain the central trend, distribution and abnormality of the execution of each trace under different services and operation types;

[0018] S43: Filter out abnormal traces in the third preliminary normal trace set according to abnormal situations.

[0019] Preferably, the statistical calculation in S42 includes calculating the mean, variance and quartile of the duration, and calculating the mean, variance and quartile of the execution time itself.

[0020] Preferably, the anomaly score of each span in S5 is obtained based on the following formula:

[0021] F=self_duration-(q3-1.5*(q3-q1));

[0022] Among them, F represents the abnormal score of Span; self_duration represents the execution time of Span itself; q3 represents the third quartile; and q1 represents the first quartile.

[0023] Preferably, S7 specifically includes: analyzing the execution path, dependency, and possible abnormal conditions of the root cause Span to obtain the root cause of the microservice abnormality.

[0024] A microservice anomaly root cause location system, comprising a first data acquisition unit, a second data acquisition unit, a third data acquisition unit, a fourth data acquisition unit, an anomaly score calculation unit, a root cause span acquisition unit and a root cause span analysis unit;

[0025] The first data acquisition unit is used to acquire a plurality of traces from the log library, and remove abnormal traces from the plurality of traces based on a preset abnormal trace set to obtain a first preliminary normal trace set;

[0026] The second data acquisition unit is used to check whether each trace in the first preliminary normal trace set is complete, remove incomplete traces, and obtain a second preliminary normal trace set; wherein, whether each trace is complete includes whether each span in the trace is repeated or missing;

[0027] The third data acquisition unit is used to construct a trace tree for each trace in the second preliminary normal trace set, and calculate the execution time of each span in each trace tree to obtain a third preliminary normal trace set;

[0028] The fourth data acquisition unit is used to perform grouping statistics on the third preliminary normal Trace set based on the service name and the Span name to obtain the abnormal Trace in the third preliminary normal Trace set;

[0029] The anomaly score calculation unit is used to calculate the anomaly score of each span in the abnormal trace;

[0030] The root cause span acquisition unit is used to select the span with the highest anomaly score from each anomaly score as the root cause span;

[0031] The root cause span analysis unit is used to analyze the root cause span to obtain the root cause of the microservice exception.

[0032] Preferably, the fourth data acquisition unit includes a grouping module, a statistical calculation module and an abnormal trace screening module;

[0033] The grouping module is used to group all traces with the same service name and span name into one group;

[0034] The statistical calculation module is used to perform statistical calculations on the duration of all spans included in each group and the execution time of all spans themselves, so as to obtain the central trend, distribution and abnormality of the execution of each trace under different services and operation types;

[0035] The abnormal trace screening module is used to screen out abnormal traces in the third preliminary normal trace set according to abnormal conditions.

[0036] Preferably, the statistical calculation module includes a first statistical calculation module and a second statistical calculation module;

[0037] The first statistical calculation module is used to calculate the mean, variance and quartile of the duration;

[0038] The second statistical calculation module is used to calculate the mean, variance and quartile of its own execution time.

[0039] Preferably, the anomaly score of each span is obtained based on the following formula:

[0040] F=self_duration-(q3-1.5*(q3-q1));

[0041] Among them, F represents the abnormal score of Span; self_duration represents the execution time of Span itself; q3 represents the third quartile; and q1 represents the first quartile.

[0042] Preferably, the root cause Span analysis unit is used to analyze the execution path, dependency relationship and possible abnormal conditions of the root cause Span to obtain the root cause of the microservice abnormality.

[0043] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for locating the root cause of a microservice anomaly, which can quickly and accurately locate the root cause of the anomaly. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A flowchart of a method for locating the root cause of a microservice anomaly provided in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of group statistical calculation results provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] It should be noted that according to the OpenTracing specification, Trace is a method for describing the flow of transactions (i.e., the execution process of requests) in a distributed system. Trace can intuitively show the transmission process of requests between services, usually represented by a tree structure, consisting of a series of service calls. When a request arrives at a microservice system, a unique Trace ID is generated. This ID is carried and propagated throughout the life cycle of the request to ensure that each service can identify and associate with the same transaction.

[0049] During the entire request processing, each time a user executes a command request or makes a remote procedure call (RPC), a span is generated. Each span has a unique identifier, called a span ID. These spans can be associated through parent-child relationships to form a hierarchical structure that can describe the execution path of the request and the performance of each step in detail.

[0050] As shown in Table 1, each Span also contains relevant information about the service call, which is used to further enrich and refine the monitoring and analysis of the request execution process.

[0051] Table 1 Related information of a Span

[0052]

[0053]

[0054] Through the combination of Trace and Span, the OpenTracing specification provides a powerful tool to help developers and operations personnel deeply understand and manage the complexity of distributed systems, identify performance bottlenecks, track the source of errors, and effectively implement distributed tracing and monitoring.

[0055] like Figure 1 As shown, an embodiment of the present invention discloses a method for locating the root cause of a microservice anomaly, comprising the following steps:

[0056] S1: Obtain several traces from the log library, and remove abnormal traces from the several traces based on a preset abnormal trace set to obtain a first preliminary normal trace set;

[0057] Specifically: 10,000 traces were randomly selected from the log library, and 6,271 abnormal traces were removed; the remaining traces were used as the first preliminary normal trace set;

[0058] S2: Check whether each trace in the first preliminary normal trace set is complete, remove incomplete traces, and obtain a second preliminary normal trace set; wherein, whether each trace is complete includes whether each span in the trace is repeated or missing;

[0059] It is understandable that we need to ensure that each trace is complete, that is, it contains all relevant span information. Therefore, we need to check whether there are any missing or duplicate spans in each trace. Only a complete trace can provide accurate analysis results.

[0060] S3: constructing a trace tree for each trace in the second preliminary normal trace set, and calculating the execution time of each span in each trace tree, to obtain a third preliminary normal trace set;

[0061] When calculating the execution time of each span, you need to consider concurrency issues, because the time periods of sub-spans may overlap. Therefore, when calculating self_duration (self-execution time), you need to accurately consider the execution time of each span and the execution time of its sub-spans to ensure that the calculation result is accurate.

[0062] S4: performing grouping and statistics on the third preliminary normal Trace set based on the service name and the span name to obtain abnormal traces in the third preliminary normal Trace set;

[0063] S5: Calculate the abnormal score of each span in the abnormal trace;

[0064] S6: Filter out the span with the highest anomaly score from each anomaly score as the root cause span;

[0065] S7: Analyze the root cause Span to obtain the root cause of the microservice exception.

[0066] Furthermore, S4 specifically includes:

[0067] S41: group all traces with the same service name and span name;

[0068] S42: Statistically calculate the duration of all spans included in each group and the execution time of all spans themselves to obtain the central trend, distribution and abnormality of the execution of each trace under different services and operation types;

[0069] like Figure 2As shown: The present invention displays the group statistical calculation results in the form of a table, which allows users to intuitively understand the execution status of Trace under different services and operation types, and can more easily find abnormalities or optimization space.

[0070] S43: Filter out abnormal traces in the third preliminary normal trace set according to abnormal situations.

[0071] Furthermore, the statistical calculation in S42 includes calculating the mean, variance and quartile of the duration, and calculating the mean, variance and quartile of the execution time itself.

[0072] It is understandable that the above statistical calculations can be completed by statistical methods or numerical calculation methods to ensure the accuracy and reliability of the results.

[0073] Furthermore, the anomaly score of each span in S5 is obtained based on the following formula:

[0074] F=self_duration-(q3-1.5*(q3-q1));

[0075] Among them, F represents the abnormal score of Span; self_duration represents the execution time of Span itself; q3 represents the third quartile; and q1 represents the first quartile.

[0076] It can be understood that the calculation method of the anomaly score of the present invention takes into account the execution time of the span itself and the deviation from the overall distribution, and thus can better reflect the degree of anomaly of the span.

[0077] Furthermore, S7 specifically includes: analyzing the execution path, dependency, and possible abnormal conditions of the root cause Span to obtain the root cause of the microservice abnormality.

[0078] A microservice anomaly root cause location system, comprising a first data acquisition unit, a second data acquisition unit, a third data acquisition unit, a fourth data acquisition unit, an anomaly score calculation unit, a root cause span acquisition unit and a root cause span analysis unit;

[0079] The first data acquisition unit is used to acquire a plurality of traces from the log library, and remove abnormal traces from the plurality of traces based on a preset abnormal trace set to obtain a first preliminary normal trace set;

[0080] The second data acquisition unit is used to check whether each trace in the first preliminary normal trace set is complete, remove incomplete traces, and obtain a second preliminary normal trace set; wherein, whether each trace is complete includes whether each span in the trace is repeated or missing;

[0081] The third data acquisition unit is used to construct a trace tree for each trace in the second preliminary normal trace set, and calculate the execution time of each span in each trace tree to obtain a third preliminary normal trace set;

[0082] The fourth data acquisition unit is used to perform grouping statistics on the third preliminary normal Trace set based on the service name and the Span name to obtain the abnormal Trace in the third preliminary normal Trace set;

[0083] The anomaly score calculation unit is used to calculate the anomaly score of each span in the abnormal trace;

[0084] The root cause span acquisition unit is used to select the span with the highest anomaly score from each anomaly score as the root cause span;

[0085] The root cause span analysis unit is used to analyze the root cause span to obtain the root cause of the microservice exception.

[0086] Furthermore, the fourth data acquisition unit includes a grouping module, a statistical calculation module and an abnormal trace screening module;

[0087] The grouping module is used to group all traces with the same service name and span name into one group;

[0088] The statistical calculation module is used to perform statistical calculations on the duration of all spans included in each group and the execution time of all spans themselves, so as to obtain the central trend, distribution and abnormality of the execution of each trace under different services and operation types;

[0089] The abnormal trace screening module is used to screen out abnormal traces in the third preliminary normal trace set according to abnormal conditions.

[0090] Further, the statistical calculation module includes a first statistical calculation module and a second statistical calculation module;

[0091] The first statistical calculation module is used to calculate the mean, variance and quartile of the duration;

[0092] The second statistical calculation module is used to calculate the mean, variance and quartile of its own execution time.

[0093] Furthermore, the anomaly score of each span is obtained based on the following formula:

[0094] F=self_duration-(q3-1.5*(q3-q1));

[0095] Among them, F represents the abnormal score of Span; self_duration represents the execution time of Span itself; q3 represents the third quartile; and q1 represents the first quartile.

[0096] Furthermore, the root cause Span analysis unit is used to analyze the execution path, dependency, and possible abnormal conditions of the root cause Span to obtain the root cause of the microservice abnormality.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for locating the root cause of microservice anomalies, characterized in that: The following steps are involved: S1: Obtain several traces from the log library, and remove abnormal traces from the several traces based on a preset abnormal trace set to obtain a first preliminary normal trace set; S2: Check whether each trace in the first preliminary normal trace set is complete, remove incomplete traces, and obtain a second preliminary normal trace set; wherein, whether each trace is complete includes whether each span in the trace is repeated or missing; S3: constructing a trace tree for each trace in the second preliminary normal trace set, and calculating the execution time of each span in each trace tree, to obtain a third preliminary normal trace set; S4: performing grouping and statistics on the third preliminary normal Trace set based on the service name and the span name to obtain abnormal traces in the third preliminary normal Trace set; S5: Calculate the abnormal score of each span in the abnormal trace; S6: Filter out the span with the highest anomaly score from each anomaly score as the root cause span; S7: Analyze the root cause Span to obtain the root cause of the microservice exception.

2. According to a method for locating the root cause of microservice anomalies according to claim 1, it is characterized in that: S4 specifically includes: S41: Group all traces with the same service name and span name into one group; S42: Statistically calculate the duration of all spans included in each group and the execution time of all spans themselves to obtain the central trend, distribution and abnormality of the execution of each trace under different services and operation types; S43: Filter out abnormal traces in the third preliminary normal trace set according to abnormal situations.

3. A method for locating the root cause of microservice anomaly according to claim 2, characterized in that: The statistical calculation in S42 includes calculating the mean, variance and quartile of the duration, and calculating the mean, variance and quartile of the execution time itself.

4. A method for locating the root cause of microservice anomaly according to claim 1, characterized in that: The anomaly score of each span in S5 is obtained based on the following formula: F=self_duration-(q3-1.5*(q3-q1)); Among them, F represents the abnormal score of Span; self_duration represents the execution time of Span itself; q3 represents the third quartile; and q1 represents the first quartile.

5. A method for locating the root cause of microservice anomaly according to claim 1, characterized in that: S7 specifically includes: analyzing the execution path, dependency, and possible abnormal situations of the root cause Span to obtain the root cause of the microservice abnormality.

6. A microservice abnormality root cause location system, characterized in that: It includes a first data acquisition unit, a second data acquisition unit, a third data acquisition unit, a fourth data acquisition unit, an abnormal score calculation unit, a root cause span acquisition unit and a root cause span analysis unit; The first data acquisition unit is used to acquire a plurality of traces from the log library, and remove abnormal traces from the plurality of traces based on a preset abnormal trace set to obtain a first preliminary normal trace set; The second data acquisition unit is used to check whether each trace in the first preliminary normal trace set is complete, remove incomplete traces, and obtain a second preliminary normal trace set; wherein, whether each trace is complete includes whether each span in the trace is repeated or missing; The third data acquisition unit is used to construct a trace tree for each trace in the second preliminary normal trace set, and calculate the execution time of each span in each trace tree to obtain a third preliminary normal trace set; The fourth data acquisition unit is used to perform grouping statistics on the third preliminary normal Trace set based on the service name and the Span name to obtain the abnormal Trace in the third preliminary normal Trace set; The anomaly score calculation unit is used to calculate the anomaly score of each span in the abnormal trace; The root cause span acquisition unit is used to select the span with the highest anomaly score from each anomaly score as the root cause span; The root cause span analysis unit is used to analyze the root cause span to obtain the root cause of the microservice exception.

7. A microservice abnormality root cause location system according to claim 6, characterized in that: The fourth data acquisition unit includes a grouping module, a statistical calculation module and an abnormal trace screening module; The grouping module is used to group all traces with the same service name and span name into one group; The statistical calculation module is used to perform statistical calculations on the duration of all spans included in each group and the execution time of all spans themselves, so as to obtain the central trend, distribution and abnormality of the execution of each trace under different services and operation types; The abnormal trace screening module is used to screen out abnormal traces in the third preliminary normal trace set according to abnormal conditions.

8. A microservice abnormality root cause location system according to claim 7, characterized in that: The statistical calculation module includes a first statistical calculation module and a second statistical calculation module; The first statistical calculation module is used to calculate the mean, variance and quartile of the duration; The second statistical calculation module is used to calculate the mean, variance and quartile of its own execution time.

9. A microservice abnormality root cause location system according to claim 6, characterized in that: The anomaly score of each span is obtained based on the following formula: F=self_duration-(q3-1.5*(q3-q1)); Among them, F represents the abnormal score of Span; self_duration represents the execution time of Span itself; q3 represents the third quartile; and q1 represents the first quartile.

10. A microservice abnormality root cause location system according to claim 6, characterized in that: The root cause span analysis unit is used to analyze the execution path, dependency and possible abnormal conditions of the root cause span to obtain the root cause of the microservice abnormality.

Citation Information

Patent Citations

  • Transaction analysis method, device and equipment and storage medium

    CN109684052A

  • Fault positioning method and device and electronic equipment

    CN110166264A

  • Index abnormal data tracing method and device, equipment and storage medium

    CN113360359A

  • Call link abnormity detection method, computer equipment and readable storage medium

    CN113516174A

  • Fault root cause positioning method and system based on service code level and storage medium

    CN113900844A

Cited By

  • Multi-dimensional root cause positioning method and device based on multi-modal learning

    CN120151186A

  • A multi-dimensional root cause positioning method and device based on multi-modal learning

    CN120151186B