Alarm root cause recommendation method and device and computer readable storage medium

Abstract

The invention provides an alarm root cause recommendation method and device and a computer readable storage medium, which can be applied to a micro-service system comprising a plurality of micro-services, and the method comprises the following steps: obtaining a plurality of normal call chains and a plurality of call chains to be tested, the plurality of normal call chains being call chains of a plurality of normal micro-service interfaces in the micro-service system, and the plurality of call chains to be tested being call chains of a plurality of normal micro-service interfaces in the micro-service system; the plurality of to-be-tested call chains are call chains of an abnormal micro-service interface in the micro-service system, then determining a plurality of candidate root causes based on the plurality of to-be-tested call chains and the plurality of normal call chains, and carrying out statistics on information of the plurality of to-be-tested call chains to obtain multi-dimensional statistical information corresponding to the plurality of candidate root causes; and then sorting the plurality of candidate root causes based on the multi-dimensional statistical information corresponding to the plurality of candidate root causes, and providing the sorted plurality of candidate root causes to the user. According to the method, root cause recommendation can be performed, and operation and maintenance personnel can select the target root cause from the recommended root causes to perform exception repair, so that the root cause positioning efficiency and the exception repair efficiency are improved.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 26, 2023, with application number 202311083390.9 and invention name “Fault root cause analysis method, device and computer-readable storage medium”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of operation and maintenance technology, and in particular to a method, device and computer-readable storage medium for recommending root causes of alarms. Background Art

[0003] With the continuous development of cloud-native technologies, microservice architecture has become the preferred architecture for modern business systems. In the microservice architecture, each microservice is independent and can be deployed, expanded, and maintained independently. This design makes the system more flexible and can respond to changes in business needs more quickly.

[0004] However, the number of microservices in a microservice system is huge, each microservice is independent and heterogeneous, and each microservice is usually deployed in a distributed manner on different nodes (such as servers, virtual machines, containers, chips, etc.). Therefore, the call relationship in the microservice system is very complex, and the occurrence of a fault is not limited to a certain node. It will propagate along the call relationship and spread to other nodes, so that a single point of failure may trigger a large-scale fault alarm. At present, when a large number of abnormalities occur in a microservice system at the same time, the operation and maintenance personnel will locate the root cause. The analysis pressure of the operation and maintenance personnel is very high. It often takes hours or even days to locate the actual root cause (such as abnormal request time consumption of a microservice interface, missing of a microservice interface, increased network delay of a microservice interface, etc.), resulting in missing the best time for fault handling and causing network accidents. Summary of the invention

[0005] The present application provides an alarm root cause recommendation method, device and computer-readable storage medium, which can recommend root causes when an abnormality occurs in a microservice system. The recommended root causes can provide a good reference for operation and maintenance personnel. The operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes, and repair the abnormality according to the selected root cause, thereby improving the efficiency of root cause location and abnormality repair.

[0006] In a first aspect, a method for recommending an alarm root cause is provided, which is applied to a microservice system including multiple microservices, and the microservice system also includes a root cause recommendation device, the method including the following steps: the root cause recommendation device obtains multiple normal call chains and multiple call chains to be tested, the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system; multiple first candidate root causes are determined based on the multiple call chains to be tested and the multiple normal call chains; information of the multiple call chains to be tested is counted to obtain multi-dimensional statistical information corresponding to the multiple first candidate root causes; multiple second candidate root causes are provided to the user, or multiple second candidate root causes and repair suggestions for each second candidate root cause are provided to the user, wherein the multiple second candidate root causes are obtained by sorting the multiple first candidate root causes based on the multi-dimensional statistical information corresponding to the multiple first candidate root causes.

[0007] In the above scheme, since the root cause recommendation device can obtain multiple normal call chains and multiple call chains to be tested of the microservice system, after determining multiple candidate root causes based on the multiple call chains to be tested and the multiple normal call chains, the multiple candidate root causes are sorted based on the multi-dimensional statistical information corresponding to the multiple candidate root causes, and the sorting results are provided to the operation and maintenance personnel of the microservice system for selection. The sorting results can provide a good reference for the operation and maintenance personnel. The operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes, and perform abnormal repair according to the selected root cause, thereby improving the efficiency of root cause location and abnormal repair.

[0008] In addition, since the root cause recommendation device can provide repair suggestions for each candidate root cause when providing the operation and maintenance personnel with multiple ranked candidate root causes, these repair suggestions can provide the operation and maintenance personnel with a better reference. After the operation and maintenance personnel select the root cause, they can refer to the repair suggestions corresponding to the selected root cause to perform abnormal repair, thereby further improving the efficiency of abnormal repair.

[0009] In one possible implementation, the multi-dimensional statistical information corresponding to each first candidate root cause among the above-mentioned multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

[0010] By implementing the above implementation method, since multiple first candidate root causes are sorted with reference to the call event dimension information of each first candidate root cause and the resource dimension information called by the call event, the candidate root causes with higher probability among the multiple first candidate root causes can be arranged in front, and the candidate root causes with lower probability can be arranged in the back, thereby providing users with more accurate recommendation results.

[0011] In one possible implementation, the first call event is a call event in which each first candidate root cause exists, and the first dimension statistical information corresponding to each of the above-mentioned first candidate root causes includes at least one of the following: the frequency, average duration and end time of the first call event in the call chain to be tested to which the first call event belongs; the second dimension statistical information corresponding to each of the above-mentioned first candidate root causes includes at least one of the following: the resource type called in multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource type called by the first call event and the nodes where the resources are located.

[0012] By implementing the above implementation, since richer information is referred to when sorting the plurality of first candidate root causes, the sorting result can be made more accurate.

[0013] In a possible implementation, the root cause recommendation device determines a plurality of first candidate root causes based on a plurality of call chains to be tested and a plurality of normal call chains, including:

[0014] The root cause recommendation device compares the root microservice interface of each call chain to be tested in multiple call chains to be tested with the root microservice interfaces of multiple normal call chains, compares the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and compares the log of each call chain to be tested with the log of multiple normal call chains to determine multiple first candidate root causes.

[0015] When implementing the above-mentioned implementation method, the root cause recommendation device compares the root microservice interface of each call chain to be tested in multiple call chains to be tested with the root microservice interfaces of multiple normal call chains, compares the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and compares the log of each call chain to be tested with the log of multiple normal call chains when determining the candidate root cause. Compared with some current technologies that only compare the root microservice interface of the call chain to be tested with the root microservice interface of the normal call chain when determining the candidate root cause, since the comparison granularity of the present application is finer, the accuracy of the determined candidate root cause can be improved.

[0016] In a possible implementation, the root cause recommendation device obtains multiple call chains to be tested, including: the root cause recommendation device counts the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the exception to the current moment; based on the number of abnormal microservice interfaces of the microservice system in multiple time periods, a target time period is selected from multiple time periods, and the rate of change between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; and the call chains of the abnormal microservice interfaces of the microservice system from the occurrence of the exception to the target time period are obtained as multiple call chains to be tested.

[0017] In a possible implementation, the method further includes: the root cause recommendation device obtains a target candidate root cause from among the multiple second candidate root causes selected by the user; based on the target candidate root cause, multiple weights are optimized, wherein the multiple weights are weights corresponding to the multi-dimensional statistical information when sorting the multiple first candidate root causes. In this way, the accuracy of the root cause recommendation can be improved when the root cause recommendation method provided by the present application is used to recommend the root cause in the future.

[0018] In a second aspect, a device for recommending root causes of alarms is provided, which is applied to a microservice system including multiple microservices, and the device includes:

[0019] An acquisition module, used to acquire multiple normal call chains and multiple call chains to be tested, wherein the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system;

[0020] A processing module, configured to determine a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains;

[0021] The processing module is further used to collect statistics on the information of the multiple call chains to be tested to obtain multi-dimensional statistical information corresponding to the multiple first candidate root causes;

[0022] A recommendation module is used to provide a plurality of second candidate root causes to a user, or to provide the plurality of second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the plurality of second candidate root causes are obtained by sorting the plurality of first candidate root causes based on multi-dimensional statistical information corresponding to the plurality of first candidate root causes.

[0023] In one possible implementation, the multi-dimensional statistical information corresponding to each first candidate root cause among the multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

[0024] In a possible implementation, the first call event is a call event in which each of the first candidate root causes exists, and the first dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the frequency, average duration and end time of the first call event in the call chain to be tested to which the first call event belongs; the second dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the resource type called and the nodes where the resources are located in the multiple call chains to be tested, and the frequency, average duration and earliest start time of the call events that are the same as the resource type called and the nodes where the resources are located by the first call event.

[0025] In one possible implementation, the processing module is specifically used to: compare the root microservice interface of each of the multiple call chains to be tested with the root microservice interface of the multiple normal call chains, compare the call relationship in each call chain to be tested with the call relationship in the multiple normal call chains, and compare the log of each call chain to be tested with the log of the multiple normal call chains to determine multiple first candidate root causes.

[0026] In a possible implementation, the processing module is further used to count the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the exception to the current moment; the processing module is further used to select a target time period from the multiple time periods based on the number of abnormal microservice interfaces of the microservice system in the multiple time periods, and the rate of change between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; the acquisition module is used to acquire the call chain of the abnormal microservice interface of the microservice system from the occurrence of the exception to the target time period as the multiple call chains to be tested.

[0027] In a possible implementation, the acquisition module is also used to obtain a target candidate root cause among the multiple second candidate root causes selected by the user; the processing module is also used to optimize multiple weights based on the target candidate root cause, wherein the multiple weights are weights corresponding to the multi-dimensional statistical information when sorting the multiple first candidate root causes.

[0028] According to a third aspect, a computing device cluster is provided, comprising a processor and a memory; the processor is used to execute instructions stored in the memory, so that the computing device cluster implements the method provided in the first aspect or any possible implementation of the first aspect.

[0029] According to a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions for implementing the method provided in the first aspect or any possible implementation of the first aspect.

[0030] According to a fifth aspect, a computer program product is provided, comprising a computer program. When the computer program is read and executed by a computing device, the computing device executes the method provided in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of a call chain and a call chain log provided in an embodiment of the present application;

[0032] Figure 2 It is a structural diagram of a microservice system provided in an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of the structure of another microservice system provided in an embodiment of the present application;

[0034] Figure 4 It is a flowchart of a method for recommending root causes of alarms provided in an embodiment of the present application;

[0035] Figure 5 This is a distribution diagram of the number of abnormal microservice interfaces of the microservice system from 21:00:00 to 21:30:00, as exemplified in an embodiment of the present application;

[0036] Figure 6 It is a flow chart of comparing the i-th call chain to be tested with multiple normal call chains provided by an embodiment of the present application;

[0037] Figure 7 is a schematic diagram of the i-th call chain to be tested and multiple normal call chains and call chain logs exemplarily shown in an embodiment of the present application;

[0038] Figure 8 This is a schematic diagram of a process for obtaining first dimension statistical information and second dimension statistical information according to an embodiment of the present application;

[0039] Fig. 9 is a schematic diagram of multiple second sets exemplarily shown in an embodiment of the present application;

[0040] Fig.10 It is the first dimension statistical information and the second dimension statistical information corresponding to the multiple first candidate root causes exemplarily shown in this application;

[0041] Fig.11 is a structural schematic diagram of a root cause recommendation device provided in an embodiment of the present application;

[0042] Fig.12 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0043] Fig.13 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0044] Fig.14 It is a structural diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0046] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0047] In order to facilitate a clear understanding of the technical solution provided by this application, some nouns and terms involved in the technical solution provided by this application are first explained.

[0048] (1) Microservice is an architectural pattern that splits an application into a group of small, independently deployed services. Each microservice runs in its own process and can be implemented using different technology stacks. Each microservice is responsible for completing a specific business function. Each microservice can be deployed and upgraded independently without affecting the operation of other microservices. Microservices can communicate with each other through lightweight communication mechanisms, such as the Hypertext Transfer Protocol (HTTP) and message queues. Users can independently scale each microservice according to actual needs to improve the elasticity and scalability of the system.

[0049] (2) Call chain: When a request enters the system, it may need to be processed and called by multiple services (also called service interfaces) in the system before it can get a response. The service calls that implement the same request are called call events, and these call events form a call chain. Calling a service means calling the corresponding method in the interface that provides the corresponding service to implement the service, such as calling the Mysql method in interface A to access the Mysql database resource.

[0050] Each call chain has a unique identifier. When a request enters the system, a globally unique trace ID is generated to identify the request. All call log information of the request will record the ID, so that the call records of the request can be associated.

[0051] Each call has a unique identifier: After receiving the request, the server generates a unique identifier to mark the call. On the one hand, the identifier can be used to distinguish different calls in the call chain (it is possible that the same interface is called multiple times). On the other hand, the identifier is also passed to the sub-call (next service) that needs to be executed during this service process, and serves as the parent call identifier of the sub-call, thereby forming a superior-subordinate relationship between adjacent service calls (referred to as a call relationship). Each call can be called a time span (span), and the unique identifier of the call is called a time span identifier (span_id), which is generated by the server and passed to the next call for use.

[0052] When receiving a request, the server will record the trace id, span_id, and start time (starttimestamp) of the call in the call log, and record the end time (finish timestamp) in the call log when the call ends. This allows the order of different service calls to be sorted and the time consumption of service calls to be analyzed.

[0053] Through the above information, the entire call chain of a request can be associated: all call logs can be found through trace_id, the call relationship can be formed through span_id and the parent span_id of each call, and the order of calls can be formed through timestamp, thus forming a directed acyclic graph, which is the call chain, see Figure 1 An exemplary call chain (indicated by a solid arrow) and a log of the call chain are shown.

[0054] exist Figure 1 In the figure, the microservice interface details corresponds to two call logs, which means that the microservice interface details is called twice by the microservice interface productpage.

[0055] The application scenarios of the embodiments of the present application are introduced below.

[0056] With the continuous development of cloud-native technology, microservice architecture has become the preferred architecture for modern business systems. Compared with the traditional monolithic application architecture, microservice architecture has higher scalability, maintainability and reliability. In the microservice architecture, each microservice is independent and can be deployed, expanded and maintained independently. This design makes the system more flexible and can respond to changes in business needs more quickly.

[0057] See also Figure 2 The microservice system 200 shown includes microservices such as product microservices, order microservices, and user microservices. These microservices are independently deployed on different computing devices (also referred to as computing nodes).

[0058] The user can operate the local device 100 and send a request to the microservice system 200 through the communication network, requesting to use the microservices such as product microservices, order microservices, user microservices, etc. provided by the microservice system 200. The local device 100, which can also be called the terminal device 100, can be any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a virtual reality (VR) device, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0059] The computing devices used to provide microservices in the microservice system 200 may include but are not limited to physical servers, virtual machines, containers, chips, etc., which are not specifically limited here. Among them, physical servers may include but are not limited to ARM servers, X86 servers, etc.; virtual machines refer to complete computer systems with complete hardware system functions implemented by network function virtualization (NFV) technology and simulated by software, running in a completely isolated environment; containers refer to a group of processes that are resource-constrained and isolated from each other.

[0060] However, the number of microservices in the microservice system 200 is huge, each microservice is independent and heterogeneous, and each microservice is usually deployed in a distributed manner on different nodes (such as servers, virtual machines, containers, chips, etc.). Therefore, the call relationship in the microservice system 200 is very complex, and the occurrence of a fault is not limited to a certain node. It will propagate along the call relationship and spread to other nodes, so that a single point failure may trigger a large-scale fault alarm. At present, when a large number of abnormalities occur in the microservice system 200 at the same time, the operation and maintenance personnel will locate the root cause. The analysis pressure of the operation and maintenance personnel is very high. It often takes several hours or even days to locate the actual root cause, resulting in missing the best time for fault handling and causing network accidents.

[0061] In order to solve the above problems, the embodiments of the present application provide an alarm root cause recommendation method and device, etc., which can recommend root causes when an abnormality occurs in the microservice system 200. The recommended root causes can provide a good reference for the operation and maintenance personnel of the microservice system 200. The operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes, and repair the abnormality according to the selected root cause, thereby improving the efficiency of root cause locating and abnormality repairing.

[0062] First, see Figure 3 , Figure 3 is a schematic diagram of the structure of a microservice system 200 provided in an embodiment of the present application, such as Figure 3 As shown, the system 200 includes not only a computing device that provides various microservices, but also a root cause recommendation device 210 .

[0063] The root cause recommendation device 210, which may also be referred to as the alarm root cause recommendation device, may be a software module deployed on a single computing device. Alternatively, the root cause recommendation device 210 may also be implemented by hardware in the computing device, for example, a processor in the computing device implements the function of the root cause recommendation device 210. Optionally, the root cause recommendation device 210 may also be a software module deployed in a computing device that provides each microservice, or may be deployed on a single or multiple computing devices that are independent of the computing devices that provide each microservice (in Figure 3 The deployment mode is taken as an example in the example above), and the computing devices providing each microservice can be connected through a communication network with any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0064] Figure 3The computing devices in the microservice system 200 may include but are not limited to physical servers, virtual machines, containers, chips, etc., which are not specifically limited here. Among them, the physical server may include but are not limited to ARM servers, X86 servers, etc. The root cause recommendation device 210 is responsible for recommending the root cause when an abnormality occurs in the microservice system 200 (for example, an alarm message is generated or the response time of a microservice interface exceeds the normal response time).

[0065] To facilitate a clearer understanding Figure 3 The root cause recommendation device 210 in the microservice system 200 performs the root cause recommendation process. Figure 4 A flow chart of an alarm root cause recommendation method provided in an embodiment of the present application is shown for detailed introduction.

[0066] like Figure 4 As shown, the method may include the following steps:

[0067] S401: the root cause recommendation device 210 obtains a plurality of normal call chains, where the plurality of normal call chains are call chains of a plurality of normal microservice interfaces in the microservice system 200 .

[0068] S402: The root cause recommendation device 210 obtains a plurality of call chains to be tested, where the plurality of call chains to be tested are call chains of abnormal microservice interfaces in the microservice system 200 .

[0069] S403: The root cause recommendation device 210 determines a plurality of first candidate root causes based on a plurality of call chains to be tested and a plurality of normal call chains.

[0070] S404: The root cause recommendation device 210 collects statistics on information of multiple call chains to be tested, and obtains multi-dimensional statistical information corresponding to multiple first candidate root causes.

[0071] S405: The root cause recommendation device 210 sorts the plurality of first candidate root causes based on the multi-dimensional statistical information corresponding to the plurality of first candidate root causes to obtain sorted plurality of first candidate root causes.

[0072] S406: The root cause recommendation device 210 provides the user with a plurality of sorted first candidate root causes, or provides the user with a plurality of sorted first candidate root causes and a repair suggestion for each first candidate root cause.

[0073] In order to facilitate understanding of the embodiments of the present application, the microservice interfaces (including normal microservice interfaces and abnormal microservice interfaces) involved in the embodiments of the present application and the call events with candidate root causes are first introduced.

[0074] Microservice interface is an interface that provides users with various microservices, such as Figure 2The interfaces of the product microservice / order microservice / user microservice shown include interfaces for accessing database resources, interfaces for accessing network resources, etc.

[0075] The embodiments of the present application relate to a normal microservice interface and an abnormal microservice interface. A normal microservice interface refers to a microservice interface with a normal response time, and an abnormal microservice interface refers to a microservice interface with an abnormal response time. The microservice interface response time refers to the period from when the microservice interface receives a call request to when the call ends. If the microservice interface response time is less than or equal to the normal response time, the microservice interface response time is determined to be normal, thereby determining that the microservice interface is normal. Otherwise, the microservice interface response time is determined to be abnormal, thereby determining that the microservice interface is abnormal. The normal response time corresponding to each microservice interface can be set according to the actual scenario, and this application does not make specific limitations. In this application, algorithms such as boxplot, histogram, and scatter plot can be used to detect whether the microservice interface response time is normal. For the sake of brevity of the specification, it will not be elaborated here.

[0076] For call events with candidate root causes, for example, assuming that call event C is a call to a Mysql method in microservice interface B on node N1, subsequent analysis shows that call event C has candidate root cause RC1 "the time consumption of calling the Mysql method in microservice interface B on node N1 has increased", then call event C "calling the Mysql method in microservice interface B on node N1" is called a call event with candidate root cause RC1 "the time consumption of calling the Mysql method in microservice interface B on node N1 has increased".

[0077] Below Figure 4 Each step shown is described in detail.

[0078] In S401, the multiple normal call chains corresponding to the multiple normal microservice interfaces in the microservice system 200 are the call chains of all the microservice interfaces in the microservice system 200 when they are normal. A normal microservice interface may correspond to at least one normal call chain among the multiple normal call chains, and the normal call chains corresponding to any two normal microservice interfaces are different.

[0079] The root cause recommendation device 210 may obtain multiple normal call chains in the following manner: the root cause recommendation device 210 receives logs of multiple normal call chains sent by the microservice system 200, and obtains multiple normal call chains based on the logs of the multiple normal call chains.

[0080] In the embodiment of the present application, the multiple normal call chains acquired by the root cause recommendation device 210 include the structure of the normal call chain (similar to Figure 1 In addition to the logs of the normal call chain, the logs of the normal call chain are also included.

[0081] In S402, the root cause recommendation device 210 may obtain multiple call chains to be tested in the following manner: the root cause recommendation device 210 receives logs of multiple call chains to be tested sent by the microservice system 200, and obtains multiple call chains to be tested based on the logs of the multiple call chains to be tested.

[0082] In the embodiment of the present application, the multiple call chains to be tested obtained by the root cause recommendation device 210 include the structure of the call chain to be tested (similar to Figure 1 In addition to the logs of the call chain to be tested (shown by the solid arrow), it also includes the logs of the call chain to be tested.

[0083] It can be understood that in actual scenarios, there may be one or more abnormal microservice interfaces in the microservice system 200, and each abnormal microservice interface may generate one or more call chains during the abnormal period. In a possible embodiment, the root cause recommendation device 210 can obtain all call chains of all abnormal microservice interfaces in the microservice system 200 from the time when the abnormality occurs in the microservice system 200 to the current moment, as multiple call chains to be tested. Optionally, the root cause recommendation device 210 can also select some call chains from all call chains as multiple call chains to be tested. Among them, when an abnormality occurs in the microservice system 200, an alarm message can be generated for the microservice system 200, indicating that an abnormality occurs in the microservice system 200, or an abnormal microservice interface can be detected in the microservice system 200, and then it is determined that the microservice system 200 is abnormal.

[0084] Next, the implementation process of the root cause recommendation device 210 selecting some call chains from all the above call chains as multiple call chains to be tested is described:

[0085] S1: Count the number of abnormal microservice interfaces of the microservice system 200 in multiple time periods from when the abnormality occurs to the current moment.

[0086] The multiple time periods may be time periods of equal length. For example, assuming that the microservice system 200 has an exception at 21:00:00 and the current time is 21:30:00, the multiple time periods may be every minute between 21:00:00 and 21:30:00, or every two minutes, etc., which is not specifically limited in this application. Preferably, the length of the multiple time periods is one minute.

[0087] S2: Based on the number of abnormal microservice interfaces of the microservice system 200 in multiple time periods, a target time period is selected from multiple time periods, wherein the rate of change between the number of abnormal microservice interfaces of the microservice system 200 in the target time period and the number of abnormal microservice interfaces of the microservice system 200 in a time period before the target time period exceeds a threshold.

[0088] That is, the target time period is a time period in which the number of abnormal microservice interfaces corresponding to multiple time periods changes suddenly compared with the number of abnormal microservice interfaces corresponding to the previous time period.

[0089] The above-mentioned change rate can be a growth rate or a reduction rate, and the threshold value can be customized according to the actual scenario, and this application does not make any specific limitation.

[0090] In a possible embodiment, when the change rate is a growth rate, the threshold value may be set to a maximum growth rate / 2, where the maximum growth rate is the maximum value of the number of abnormal microservice interfaces of the microservice system 200 in each time period and the growth rate of the number of abnormal microservice interfaces of the microservice system 200 in the previous time period of each time period; when the change rate is a reduction rate, the threshold value may be set to a maximum reduction rate / 3, where the maximum reduction rate is the maximum value of the number of abnormal microservice interfaces of the microservice system 200 in each time period and the reduction rate of the number of abnormal microservice interfaces of the microservice system 200 in the previous time period of each time period.

[0091] It can be understood that if the growth rate corresponding to the target time period exceeds the threshold, it means that the number of abnormal microservice interfaces of the microservice system 200 increases suddenly during the target time period, which usually means that the microservice system 200 fails before the target time period. Root cause analysis is mainly used to detect the cause of the failure at the beginning of the failure. Therefore, the root cause should be located before the target time period. If the decline rate corresponding to the target time period exceeds the threshold, it means that the number of abnormal microservice interfaces of the microservice system 200 decreases suddenly during the target time period, which usually means that the failure of the microservice system 200 is restored during the target time period, and the root cause should be located before the target time period.

[0092] S3: Acquire the call chains of the abnormal microservice interfaces of the microservice system 200 from the occurrence of the abnormality to the target time period as multiple call chains to be tested.

[0093] In a specific implementation, if each abnormal microservice interface generates a large number of call chains from the time when the microservice system 200 is abnormal to the target time period, some call chains can be selected from the large number of call chains generated by the abnormal microservice interface as the call chains to be tested corresponding to the abnormal microservice interface, and the selected part of the call chain can be the call chain with abnormal response time in the call chain. Further, the call chain with an early start time and a large response time can be selected from the large number of call chains. This is because the call chain with an early start time and a large response time is usually a call chain with a root cause (such as a call event with a time-consuming abnormality in the call chain). Among them, the response time of the call chain can be the period from the start call time to the end call time of the call chain. The abnormal response time of the call chain can be comparing the response time of the call chain with the normal response time of the call chain. When the response time of the call chain is less than or equal to the normal response time of the call chain, it is determined that the response time of the call chain is normal, otherwise, it is determined that the response time of the call chain is abnormal. Among them, the normal response time of the call chain can be customized according to the actual scenario, and this application does not make specific restrictions on this.

[0094] In a possible embodiment, if the time period from the occurrence of an exception in the microservice system 200 to the target time period is a short period of time, such as a few minutes, then the call chain of the abnormal microservice interface of the microservice system 200 within these few minutes can be directly obtained as multiple call chains to be tested; if the time period from the occurrence of an exception in the microservice system 200 to the target time period is a very long period of time, more than ten minutes, or even more than half an hour, then a continuous time (such as a few minutes) including the target time period can be intercepted from the longer time period, and the call chain of the abnormal microservice interface of the microservice system 200 within the intercepted time period is obtained as multiple call chains to be tested. This application does not specifically limit the intercepted time period.

[0095] For example, see Figure 5 , Figure 5 is a distribution diagram of the number of abnormal microservice interfaces of the microservice system 200 from 21:00:00 to 21:30:00, as shown in the embodiment of the present application. Figure 5 It can be seen that the number of abnormal microservice interfaces in microservice system 200 suddenly increased at 21:10:00. Therefore, it can be inferred that the resources of microservice system 200 reached the upper limit at 21:10:00, which led to a sharp increase in the number of abnormal microservice interfaces. Since root cause analysis is mainly used to detect the cause of the failure at the beginning of the failure, the call chain of the abnormal microservice interface before 21:10:00 can be obtained for analysis, such as obtaining the call chain of the abnormal microservice interface between 21:03:00-21:10:00 as the call chain to be tested for analysis. Figure 5It can also be seen that the number of abnormal microservice interfaces in microservice system 200 suddenly drops around 21:22:00. Therefore, it can be inferred that the failure of microservice system 200 ends at 21:22:00. Optionally, the call chain of the abnormal microservice interface between 21:00:00 and 21:22:00 can also be obtained as the call chain to be tested for analysis.

[0096] It should be understood that the above S1 to S3 are merely regarded as exemplary implementations of the root cause recommendation device 210 to obtain multiple call chains to be tested, and this application does not make specific limitations. For example, in a specific implementation, S1 can be the ratio of the number of abnormal microservice interfaces in the microservice system 200 from the occurrence of an exception to the current moment to the total number of microservice interfaces in the microservice system 200, and S2 can be based on the ratio of the number of abnormal microservice interfaces in the microservice system 200 in multiple time periods to the total number of microservice interfaces, and a target time period is selected from multiple time periods, wherein the ratio of the number of abnormal microservice interfaces in the microservice system 200 in the target time period to the total number of microservice interfaces, and the ratio of the number of abnormal microservice interfaces in the microservice system 200 in the time period before the target time period to the total number of microservice interfaces, and the ratio of the number of abnormal microservice interfaces in the microservice system 200 in the time period before the target time period to the total number of microservice interfaces. The ratio of the ratio exceeds the threshold, and S3 remains unchanged.

[0097] In S403, the root cause recommendation device 210 may determine a plurality of first candidate root causes by comparing a plurality of call chains to be tested with a plurality of normal call chains.

[0098] Specifically, multiple first candidate root causes can be determined by comparing the root microservice interface of each call chain to be tested (referring to the microservice interface that is called earliest in the call chain) in the multiple call chains to be tested with the root microservice interfaces of multiple normal call chains, comparing the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and comparing the log of each call chain to be tested with the log of multiple normal call chains.

[0099] Take the i-th call chain to be tested as an example, see Figure 6 The flowchart shown is a flow chart of comparing the i-th call chain to be tested with multiple normal call chains.

[0100] S601: Determine whether there is a root microservice interface that is the same as the root microservice interface of the i-th call chain to be tested among the root microservice interfaces of multiple normal call chains. If it is determined that the same root microservice interface exists, execute S602 and S603. If it is determined that the same root microservice interface does not exist, execute S607, that is, analyze the root microservice interface of the i-th call chain to be tested to obtain the first candidate root cause.

[0101] The above-mentioned same root microservice interface means that the root microservice interface identifier and the root microservice interface calling method are the same.

[0102] by Figure 7 Take the i-th test call chain and multiple normal call chains as examples. Figure 7 In the figure, the root microservice interface of the i-th call chain to be tested is the microservice interface productpage, and the calling method is GET. Multiple normal call chains include normal call chain 1, normal call chain 2..., among which the root microservice interface of normal call chain 1 is the microservice interface productpage, and the calling method is GET. The root microservice interface of normal call chain 2 is the microservice interface bluewhare, and the calling method is POST. Therefore, it can be determined that the root microservice interface of normal call chain 1 is the same as the root microservice interface of the i-th call chain to be tested, and the root microservice interface of normal call chain 2 is different from the root microservice interface of the i-th call chain to be tested.

[0103] There is no normal call chain in multiple normal call chains that is the same as the root microservice interface of the i-th call chain to be tested. This may include the following situations:

[0104] Case 1. There is no root microservice interface identifier X (such as Figure 7 In this case, the first candidate root cause may be: microservice interface X does not exist, or microservice interface X requested by the client does not exist, or microservice interface X does not exist in microservice system 200, and so on.

[0105] Case 2. Among multiple normal call chains, there is a normal call chain with the same root microservice interface identifier X as the i-th call chain to be tested, but with a different root microservice interface calling method. In this case, the first candidate root cause may be: microservice interface X does not have a calling method Y (referring to the calling method of the root microservice interface X of the i-th call chain to be tested, such as Figure 7 The calling method of the root microservice interface productpage of the i-th call chain to be tested is shown as GET), or the client calls the microservice interface productpage in an incorrect way, or the client requests an HTTP method incorrectly, and so on.

[0106] S602: Determine, from the calling relationships of multiple normal calling chains, a calling relationship that best matches the calling relationship in the i-th calling chain to be tested.

[0107] Specifically, the similarity between the call relationship in the i-th call chain to be tested and the call relationship in multiple normal call chains can be calculated, and the call relationship with the greatest similarity between the call relationship in multiple normal call chains and the call relationship in the i-th call chain to be tested is determined as the most matching call relationship. The similarity calculation method can be Euclidean distance, cosine similarity, Pearson similarity or Jaccard similarity, etc., which is not specifically limited in this application.

[0108] S603: Determine whether the best-matching call relationship is consistent with the call relationship in the i-th call chain to be tested. If the call relationships are consistent, execute S604 and S605. If the call relationships are inconsistent, execute S608, that is, perform a difference analysis on the best-matching call relationship and the call relationship in the i-th call chain to be tested to obtain the first candidate root cause.

[0109] Continue with Figure 7 For example, from Figure 7 It can be seen that the calling relationship of the normal call chain 1 is the same as the calling relationship in the i-th call chain to be tested.

[0110] Assume that the call relationship of normal call chain 1 is the call relationship that best matches the call relationship in the i-th call chain to be tested. The difference between the call relationship of normal call chain 1 and the i-th call chain to be tested is that: in the call relationship of normal call chain 1, there is no microservice interface reviewsv2 calling microservice interface ratings, while in the call relationship in the i-th call chain to be tested, there is microservice interface reviewsv2 calling microservice interface ratings. At this time, the first candidate root cause can be: microservice interface ratings is missing, or microservice interface ratings is wrong, and so on.

[0111] S604: Determine, from the logs of multiple normal call chains, a log that best matches the log of the i-th call chain to be tested.

[0112] Specifically, the similarity between the log of the i-th call chain to be tested and the logs of multiple normal call chains can be calculated, and the log with the greatest similarity between the logs of multiple normal call chains and the log of the i-th call chain to be tested is determined as the most matching log. The similarity calculation method can be Euclidean distance, cosine similarity, Pearson similarity or Jaccard similarity, etc., which is not specifically limited in this application.

[0113] S605: Determine whether the best-matching log is consistent with the log of the i-th call chain to be tested. If the logs are consistent, execute S606, i.e., determine that the first candidate root cause does not exist in the i-th call chain to be tested. If the logs are inconsistent, execute S609, i.e., perform a difference analysis on the best-matching log and the log of the i-th call chain to be tested to obtain the first candidate root cause.

[0114] The above-mentioned log consistency refers to the consistency of attributes of the same call event recorded in the log, wherein the event attribute may be event duration and / or event frequency, etc.

[0115] Taking the event attribute as event duration as an example, if the difference between the duration of all call events recorded in the most matching log and the duration of the same call event recorded in the log of the i-th call chain to be tested is less than the duration threshold, then the two logs are determined to be consistent, otherwise, the two logs are determined to be inconsistent. Assuming that the duration of call event C recorded in the most matching log is much greater than the duration of call event C recorded in the log of the i-th call chain to be tested, that is, the difference between the duration of call event C recorded in the most matching log and the duration of call event C recorded in the log of the i-th call chain to be tested is greater than the duration threshold, assuming that call event C is a Mysql method that calls microservice interface B, at this time, the first candidate root cause can be: the Mysql method duration of microservice interface B is aggravated.

[0116] Taking the event attribute as event frequency, if the difference between the frequency of all call events recorded in the most matching log and the frequency of the same call event recorded in the log of the i-th call chain to be tested is less than the frequency threshold, then the two logs are determined to be consistent, otherwise, the two logs are determined to be inconsistent. Assuming that the frequency of call event C recorded in the most matching log is much greater than the frequency of call event C recorded in the log of the i-th call chain to be tested, that is, the difference between the frequency of call event C recorded in the most matching log and the frequency of call event C recorded in the log of the i-th call chain to be tested is greater than the frequency threshold, assuming that call event C is a Mysql method that calls microservice interface B, at this time, the first candidate root cause can be: the Mysql method of microservice interface B is called too frequently.

[0117] It should be understood that the description information of the first candidate root cause listed above is only an example, and the present application does not specifically limit the description information of the first candidate root cause.

[0118] In S404, based on the above content, after obtaining multiple first candidate root causes, the root cause recommendation device 210 can collect statistics on information of multiple call chains to be tested based on the multiple first candidate root causes to obtain multi-dimensional statistical information corresponding to each of the multiple first candidate root causes.

[0119] The multi-dimensional statistical information corresponding to each first candidate root cause includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension where each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event where each first candidate root cause exists.

[0120] For ease of description, taking the i-th first candidate root cause as an example, call event i represents the call event with the i-th first candidate root cause, and the first dimension statistical information corresponding to the i-th first candidate root cause includes at least one of the following: the frequency, average time and end time of call event i in the call chain to be tested to which call event i belongs; the second dimension statistical information corresponding to the i-th first candidate root cause includes at least one of the following: the frequency, average time and earliest start time of call events in which the resource type called in multiple call chains to be tested and the resource node where the resource is located are the same as the resource type called by call event i and the resource node where the resource is located. Among them, resource types can include Mysql, message queues, input / output (IO) and network bandwidth. It should be understood that the specific division method of resource types can be set according to the actual scenario, and this application does not make specific limitations on this.

[0121] It should be understood that the information types included in the first dimension statistical information corresponding to the i-th first candidate root cause and the information types included in the second dimension statistical information are merely examples and should not be considered as specific limitations. For example, in one possible embodiment, the first dimension statistical information also includes the start time of the call event i in the call chain to be tested to which the call event i belongs, and the second dimension statistical information also includes the resource types called in multiple call chains to be tested and the nodes where the resources are located, and the latest end time of the call event that has the same resource type and resource node as the call event i.

[0122] The following is a detailed description of the process of obtaining the statistical information of the first dimension and the statistical information of the second dimension with a specific example. Figure 8 The process diagram is shown.

[0123] S801: Obtain the frequency, average duration, and end time of the call event i in the call chain to be tested to which the call event i belongs.

[0124] In the call chain to be tested to which the above-mentioned call event i belongs, call event i may occur once or multiple times. When call event i occurs once, the end time is the end time of call event i. When call event i occurs multiple times, the end time can be the end time of the call event i that starts first among the multiple call events i. Optionally, it can also be the end time of any call event i.

[0125] The end time of call event i is obtained instead of the start time. This is because for a call chain to be tested, the exception that is triggered first is likely to cause the occurrence of subsequent exceptions (for example, the failure of the database read and write event operation during the request processing leads to the failure of the data synchronization event). Usually, there is no situation where the exception that occurs later causes the exception that occurs earlier. In other words, the candidate root cause corresponding to the call event with an earlier end time is more likely to become the real root cause. Therefore, in this application, the end time of the call event is obtained as a reference factor for sorting multiple first candidate root causes.

[0126] S802: Divide the call events in the multiple call chains to be tested according to the occurrence time and the end time to obtain multiple first sets, wherein any two first sets in the multiple first sets correspond to non-overlapping time periods.

[0127] Taking 1143 call events in multiple call chains to be tested as an example, assuming that 57 of them occur between 21:09:59 and 21:10:00, 86 of them occur between 21:10:31 and 21:10:32, and the remaining 1000 occur between 21:10:35 and 21:13:31, the three first sets shown in Table 1 can be obtained.

[0128] Table 1 Multiple first sets

[0129] First set serial number Fault range Number of call events 1 21:09:59-21:10:00 57 2 21:10:31-21:10:32 86 3 At 21:10:35-21:13:31 1000

[0130] It should be understood that Table 1 is merely an example of multiple first sets and should not be considered as a specific limitation.

[0131] S803: Divide the call events in each first set according to the called resource type and the node where the resource is located to obtain multiple second sets corresponding to each first set, and any two second sets in the multiple second sets correspond to different resource types and different nodes.

[0132] Taking resource types including Mysql, message queue, IO and network bandwidth as an example, assuming that among the 1000 call events contained in the first set 3 of Table 1, 300 call events call resource types of Mysql type, 55 call events call resource types of message queue type, 600 call events call resource types of IO type, and 45 call events call resource types of network bandwidth type. Among the 300 call events of Mysql type, 282 call events call resource nodes of node N1, and the remaining 18 call events call resource nodes of node N2. Among the 55 call events of the type, 43 of the called resources are located in node N1, and the remaining 12 of the called resources are located in node N2. Among the 600 call events of the above-mentioned call resource type of IO type, 545 of the called resources are located in node N1, and the remaining 55 of the called resources are located in node N2. Among the 45 call events of the above-mentioned call resource type of network bandwidth type, 38 of the called resources are located in node N1, and the remaining 7 of the called resources are located in node N2. The 908 call events in the first set 3 are divided according to the called resource type and the resource node, and the obtained multiple second sets are as follows: Fig. 9 shown.

[0133] S804: Count the average time consumption of all call events in each second set (see Fig. 9 The average time taken in each second set shown) and the earliest start time ( Fig. 9 not shown).

[0134] S805: Map each second set to a corresponding first candidate root cause among the multiple first candidate root causes according to the resource type called by the calling event and the node where the resource is located.

[0135] Taking the i-th first candidate root cause as an example, assume that there is a call event i of the i-th first candidate root cause that calls the Mysql method of the microservice interface B on the node N1. Fig. 9 Among the multiple second sets shown, the second set 1 corresponds to the node N1 and corresponds to the Mysql resource type, and the second set 1 can be mapped to the i-th first candidate root cause.

[0136] It can be seen that based on Figure 8 In the example shown, the first dimension statistical information and the second dimension statistical information corresponding to the i-th first candidate root cause are obtained. It can be understood that Figure 8 The method of obtaining the first dimension statistical information and the second dimension statistical information corresponding to the i-th first candidate root cause shown in the figure can obtain the first dimension statistical information and the second dimension statistical information corresponding to more first candidate root causes. For the sake of brevity of the specification, it will not be elaborated here.

[0137] See also Fig.10 , Fig.10 is the first dimension statistical information and the second dimension statistical information corresponding to the first candidate root causes exemplarily shown in this application. It should be noted that Fig.10 In this article, interface is the abbreviation of microservice interface. Fig.10 Taking the first candidate root cause "the increased time consumption of the Mysql method of interface B" in the row with sequence number 1 and the first candidate root cause "the increased time consumption of the Mysql method of interface B" in the row with sequence number 2 as examples, it should be noted that these two first candidate root causes are obtained based on the analysis of different call chains to be tested. For example, the first candidate root cause "the increased time consumption of the Mysql method of interface B" in the row with sequence number 1 is obtained based on the analysis of the call chain to be tested with trace id f1041, and the first candidate root cause "the increased time consumption of the Mysql method of interface B" in the row with sequence number 2 is obtained based on the analysis of the call chain to be tested with trace id f1042.

[0138] exist Fig.10 In the figure, “Same Type Call Exception Frequency” indicates the frequency of calling event i in the call chain to be tested to which calling event i belongs, and “Same Type Resource Call Count” indicates the frequency of calling events in which the resource types and resource nodes called in multiple call chains to be tested are the same as those called by calling event i and the resource types and resource nodes are the same.

[0139] Continue with Fig.10 Take the first candidate root cause "Interface B's MySQL method takes more time" in the row with sequence number 1 as an example. Fig.10 It can be seen that the first dimension statistical information corresponding to the first candidate root cause includes: the frequency of the call event (i.e., calling the Mysql method of interface B) in the call chain to be tested to which the call event of the first candidate root cause (i.e., calling the Mysql method of interface B) belongs is 3, the average time consumption is 40666.67 (in milliseconds), and the end time is 21:11:06 on August 14, 2023; the second dimension statistical information includes: the resource type called in multiple call chains to be tested and the node where the resource is located are the same as the resource type (i.e., Mysql type) called by the call event (i.e., calling the Mysql method of interface B) and the node where the resource is located (i.e., node N1) The frequency of the call event is 282, the average time consumption is 1265.96 (in milliseconds), and the earliest start time is 21:11:02 on August 14, 2023.

[0140] It should be understood that Fig. 9 as well as Fig.10 The contents in the description are merely examples provided in this application and should not be considered as specific limitations.

[0141] In S405, the root cause recommendation device 210 can apply weights to each dimension of the multidimensional statistical information, and then encode the multidimensional statistical information corresponding to each first candidate root cause into a multidimensional vector, and multiply each element in the multidimensional vector by the corresponding weight, and then add the results to obtain a weighted sum (also referred to as a score). After calculating and obtaining multiple weighted sums corresponding to multiple first candidate root causes, the multiple first candidate root causes can be sorted based on the multiple weighted sums, see the sorting scores corresponding to the multiple first candidate root causes shown in the last column of Table 2.

[0142] In S406, Fig.10 Taking the first candidate root cause "Mysql method of interface B consumes more time" as an example, the repair suggestion for the candidate root cause provided by the root cause recommendation device 210 may be:

[0143] Repair suggestion (1), troubleshooting: a. Use monitoring tools (such as Datadog) or MySQL's slow query log tool to find out the query statements that take a long time to execute; b. Analyze the query statements to determine whether there are potential performance issues, such as not using indexes, table scans, complex join operations, etc.

[0144] Repair suggestion (2), optimize query statements: a. Review query statements to ensure that correct table joins, group by, and order by conditions are used, and avoid using select*; b. Use the explain command to analyze query statements and understand their execution plans so that corresponding optimizations can be performed.

[0145] It can be understood that when the root cause recommendation device 210 provides the user with a plurality of sorted first candidate root causes, if it also provides the user with repair suggestions for each first candidate root cause, compared to only providing the sorted plurality of first candidate root causes, it can provide the user with a better reference, making it easier for the user to refer to the repair suggestions to repair the abnormality.

[0146] In a possible embodiment, when providing the ranked multiple first candidate root causes to the user, the root cause recommendation device 210 may also provide the ranking scores corresponding to the multiple first candidate root causes for the user's reference.

[0147] In a possible embodiment, after providing the sorted multiple first candidate root causes to the user, the root cause recommendation device 210 can also obtain the target candidate root cause selected by the user from the sorted multiple first candidate root causes, and then optimize the weight corresponding to the multi-dimensional statistical information when sorting the multiple first candidate root causes based on the target candidate root cause, so as to improve the accuracy of the root cause recommendation when the alarm root cause recommendation method provided in the embodiment of the present application is used for root cause recommendation in the future. The algorithm for optimizing the weight can be Bayesian optimization (such as tree-structured parzen estimator (TPE) algorithm), random search, genetic algorithm, simulated annealing algorithm, reinforcement learning algorithm, etc., which is not specifically limited in the present application.

[0148] In summary, in the alarm root cause recommendation method provided by the present application, the root cause recommendation device 210 can obtain multiple normal call chains and multiple call chains to be tested of the microservice system 200. After determining multiple candidate root causes based on the multiple call chains to be tested and the multiple normal call chains, the multiple candidate root causes are sorted based on the multi-dimensional statistical information corresponding to the multiple candidate root causes, and the sorting results are provided to the operation and maintenance personnel of the microservice system 200. The sorting results can provide a good reference for the operation and maintenance personnel. The operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes, and perform abnormal repair according to the selected root cause, thereby improving the root cause location efficiency and the abnormal repair efficiency.

[0149] When providing the sorted multiple candidate root causes to the operation and maintenance personnel, the root cause recommendation device 210 can provide repair suggestions for each candidate root cause. These repair suggestions can provide a better reference for the operation and maintenance personnel. After selecting the root cause, the operation and maintenance personnel can refer to the repair suggestions corresponding to the selected root cause to perform abnormal repair, thereby further improving the efficiency of abnormal repair.

[0150] The above describes in detail the method of the embodiment of the present application. In order to facilitate better implementation of the above scheme of the embodiment of the present application, correspondingly, the following also provides relevant devices and equipment for cooperating in implementing the above scheme.

[0151] See also Fig.11 , Fig.11 2 is a schematic diagram of the structure of a root cause recommendation device 210 provided in an embodiment of the present application. Fig.11 As shown, the device 210 includes: an acquisition module 211 , a processing module 212 and a recommendation module 213 .

[0152] The acquisition module 211 is used to acquire multiple normal call chains and multiple call chains to be tested, where the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system;

[0153] A processing module 212, configured to determine a plurality of first candidate root causes based on a plurality of call chains to be tested and a plurality of normal call chains;

[0154] The processing module 212 is further used to collect statistics on the information of the multiple call chains to be tested, and obtain multi-dimensional statistical information corresponding to the multiple first candidate root causes;

[0155] The recommendation module 213 is used to provide multiple second candidate root causes to the user, or to provide multiple second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the multiple second candidate root causes are obtained by sorting the multiple first candidate root causes based on the multi-dimensional statistical information corresponding to the multiple first candidate root causes.

[0156] In some possible implementations, the multi-dimensional statistical information corresponding to each first candidate root cause among the above-mentioned multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

[0157] In some possible implementations, the first call event is a call event in which each first candidate root cause exists, and the first dimension statistical information corresponding to each of the above-mentioned first candidate root causes includes at least one of the following: the frequency, average duration and end time of the first call event in the call chain to be tested to which the first call event belongs; the second dimension statistical information corresponding to each of the above-mentioned first candidate root causes includes at least one of the following: the resource type called in multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource type called by the first call event and the nodes where the resources are located.

[0158] In some possible implementations, the processing module 212 is specifically used to: compare the root microservice interface of each call chain to be tested in multiple call chains to be tested with the root microservice interfaces of multiple normal call chains, compare the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and compare the log of each call chain to be tested with the log of multiple normal call chains to determine multiple first candidate root causes.

[0159] In some possible implementations, the processing module 212 is also used to count the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the exception to the current moment; the processing module 212 is also used to select a target time period from multiple time periods based on the number of abnormal microservice interfaces of the microservice system in multiple time periods, and the rate of change between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; the acquisition module 211 is used to obtain the call chain of the abnormal microservice interface of the microservice system from the occurrence of the exception to the target time period as multiple call chains to be tested.

[0160] In some possible implementations, the acquisition module 211 is also used to obtain a target candidate root cause from multiple second candidate root causes selected by a user; the processing module 212 is also used to optimize multiple weights based on the target candidate root cause, wherein the multiple weights are weights corresponding to the multi-dimensional statistical information when sorting multiple first candidate root causes.

[0161] The acquisition module 211, the processing module 212, and the recommendation module 213 in all possible implementations described above can be implemented by software or by hardware. Exemplarily, the implementation of the acquisition module 211 is described below by taking the acquisition module 211 as an example. Similarly, the implementations of the processing module 212 and the recommendation module 213 can refer to the implementation of the acquisition module 211.

[0162] As an example of a software functional unit, the acquisition module 211 may include a program running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 211 may include a program running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the program may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the program may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0163] Similarly, multiple hosts / virtual machines / containers used to run the program can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0164] As an example of a hardware functional unit, the acquisition module 211 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 211 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0165] The multiple computing devices included in the acquisition module 211 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 211 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 211 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0166] It should be noted that the steps implemented by the acquisition module 211, the processing module 212, and the recommendation module 213 can be specified as needed, and all functions of the device 210 can be realized by respectively implementing different steps in the alarm root cause recommendation method through the acquisition module 211, the processing module 212, and the recommendation module 213.

[0167] See also Fig.12 , Fig.12 is a schematic diagram of a computing device provided in an embodiment of the present application. The computing device can be used to deploy Fig.11 The root cause recommendation device 210 is shown to implement the alarm root cause recommendation method described in the above method embodiment.

[0168] like Fig.12 As shown, the computing device 1200 includes: a bus 1201, a processor 1202, a memory 1203, a network interface 1204 and an output interface 1205. The processor 1202, the memory 1203, the network interface 1204 and the output interface 1205 communicate through the bus 1201. The computing device 1200 can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device 1200 can also be a terminal device such as a desktop, a laptop or a smart phone. It should be understood that the present application does not limit the number of processors and memories in the computing device 1200.

[0169] The bus 1201 may be a peripheral component interconnection standard bus or an extended industry standard structure bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 The bus 1201 may include a path for transmitting information between various components of the computing device 1200 (eg, the processor 1202, the memory 1203, the network interface 1204, and the output interface 1205).

[0170] The processor 1202 is composed of one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip is an ASIC, a PLD, or a combination thereof. The PLD is a complex programming logic device CPLD, an FPGA, a GAL, a data processing unit (DPU), a system on chip (SoC), or any combination thereof. The processor 1202 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 1203, which enables the computing device 1200 to provide a wide variety of services.

[0171] The memory 1203 may include a volatile memory, such as a random access memory, or a non-volatile memory, such as a read-only memory, a flash memory, a mechanical hard disk, or a solid state drive.

[0172] The memory 1203 stores an executable program, and the processor 1202 executes the executable program to respectively implement the functions of the acquisition module 211, the processing module 212, and the recommendation module 213, thereby implementing the alarm root cause recommendation method described in the above method embodiment. That is, the memory 1203 stores instructions for executing the alarm root cause recommendation method described in the above method embodiment.

[0173] The network interface 1204 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to obtain files or network data.

[0174] The output interface 1205 uses, for example but not limited to, a wired interface to enable communication between the computing device 1200 and other devices or a communication network.

[0175] See also Fig.13 , Fig.13 1 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. The computing device cluster provided in the present application includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0176] like Fig.13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1203 in one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the alarm root cause recommendation method.

[0177] In some possible implementations, the memory 1203 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the alarm root cause recommendation method. In other words, the combination of one or more computing devices 1200 may jointly execute instructions for executing the alarm root cause recommendation method.

[0178] It should be noted that the memory 1203 in different computing devices 1200 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the root cause recommendation apparatus 210. That is, the instructions stored in the memory 1203 in different computing devices 1200 may implement the functions of one or more of the aforementioned acquisition module 211, processing module 212, and recommendation module 213.

[0179] See also Fig.14 , Fig.14 FIG. 1 is a schematic diagram of another computing device cluster provided in an embodiment of the present application. In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. Fig.14As shown, two computing devices 1200A and 1200B are connected via a network. Specifically, the network is connected via a network interface and / or an output interface in each computing device. In this type of possible implementation, the memory 1203 in the computing device 1200A stores instructions for executing the functions of the acquisition module 211 and the recommendation module 213. At the same time, the memory 1203 in the computing device 1200B stores instructions for executing the functions of the processing module 212.

[0180] Fig.14 The connection method between the computing device clusters shown can be considered to be that the alarm root cause recommendation method provided in this application needs to analyze a large number of call chains to be tested and a large number of normal call chains to determine multiple first candidate root causes, so it is considered to hand over the functions implemented by the processing module 212 to the computing device 1200B for execution.

[0181] It should be understood that Fig.14 The functions of the computing device 1200A shown in FIG. 1200A may also be performed by multiple computing devices. Similarly, the functions of the computing device 1200B may also be performed by multiple computing devices.

[0182] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the alarm root cause recommendation method.

[0183] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions for instructing a computing device to execute an alarm root cause recommendation method.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recommending root causes of alarms, characterized in that: Applied to a microservice system including a plurality of microservices, the microservice system further including a root cause recommendation device, the method comprising: The root cause recommendation device obtains a plurality of normal call chains and a plurality of call chains to be tested, wherein the plurality of normal call chains are call chains of a plurality of normal microservice interfaces in the microservice system, and the plurality of call chains to be tested are call chains of abnormal microservice interfaces in the microservice system; The root cause recommendation device determines a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains; The root cause recommendation device collects statistics on the information of the plurality of call chains to be tested to obtain multi-dimensional statistical information corresponding to the plurality of first candidate root causes; The root cause recommendation device provides a plurality of second candidate root causes to the user, or provides the plurality of second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the plurality of second candidate root causes are obtained by sorting the plurality of first candidate root causes based on multi-dimensional statistical information corresponding to the plurality of first candidate root causes.

2. The method according to claim 1, characterized in that The multi-dimensional statistical information corresponding to each first candidate root cause among the multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

3. The method according to claim 2, characterized in that The first call event is a call event in which each of the first candidate root causes exists, and the first dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the frequency, average duration, and end time of the first call event in the call chain to be tested to which the first call event belongs; The second dimension statistical information corresponding to each first candidate root cause includes at least one of the following: the resource types called in the multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource types called by the first call event and the nodes where the resources are located.

4. The method according to any one of claims 1 to 3, characterized in that: The root cause recommendation device determines a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains, including: The root cause recommendation device compares the root microservice interface of each of the multiple call chains to be tested with the root microservice interfaces of the multiple normal call chains, compares the call relationship in each call chain to be tested with the call relationship in the multiple normal call chains, and compares the log of each call chain to be tested with the log of the multiple normal call chains to determine multiple first candidate root causes.

5. The method according to any one of claims 1 to 4, characterized in that: The root cause recommendation device obtains multiple call chains to be tested, including: The root cause recommendation device counts the number of abnormal microservice interfaces of the microservice system in multiple time periods from when the abnormality occurs to the current moment; The root cause recommendation device selects a target time period from the multiple time periods based on the number of abnormal microservice interfaces of the microservice system in the multiple time periods, and the change rate between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; The root cause recommendation device obtains the call chains of the abnormal microservice interfaces of the microservice system from the occurrence of the abnormality to the target time period as the multiple call chains to be tested.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The root cause recommendation device obtains a target candidate root cause from among the plurality of second candidate root causes selected by the user; The root cause recommendation device optimizes a plurality of weights based on the target candidate root causes, wherein the plurality of weights are weights corresponding to the multi-dimensional statistical information when the plurality of first candidate root causes are sorted.

7. An alarm root cause recommendation device, characterized in that: Applied to a microservice system including multiple microservices, the device includes: An acquisition module, used to acquire multiple normal call chains and multiple call chains to be tested, wherein the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system; A processing module, configured to determine a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains; The processing module is further used to collect statistics on the information of the multiple call chains to be tested to obtain multi-dimensional statistical information corresponding to the multiple first candidate root causes; A recommendation module is used to provide a plurality of second candidate root causes to a user, or to provide the plurality of second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the plurality of second candidate root causes are obtained by sorting the plurality of first candidate root causes based on multi-dimensional statistical information corresponding to the plurality of first candidate root causes.

8. The device according to claim 7, characterized in that The multi-dimensional statistical information corresponding to each first candidate root cause among the multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

9. The device according to claim 8, characterized in that The first call event is a call event in which each of the first candidate root causes exists, and the first dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the frequency, average duration, and end time of the first call event in the call chain to be tested to which the first call event belongs; The second dimension statistical information corresponding to each first candidate root cause includes at least one of the following: the resource types called in the multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource types called by the first call event and the nodes where the resources are located.

10. The device according to any one of claims 7 to 9, characterized in that The processing module is specifically used for: The root microservice interface of each of the multiple call chains to be tested is compared with the root microservice interface of the multiple normal call chains, the call relationship in each call chain to be tested is compared with the call relationship in the multiple normal call chains, and the log of each call chain to be tested is compared with the log of the multiple normal call chains to determine multiple first candidate root causes.

11. The device according to any one of claims 7 to 10, characterized in that The processing module is also used to count the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the abnormality to the current moment; The processing module is further configured to select a target time period from the multiple time periods based on the number of abnormal microservice interfaces of the microservice system in the multiple time periods, and the change rate between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; The acquisition module is used to acquire the call chains of the abnormal microservice interfaces of the microservice system from the occurrence of the abnormality to the target time period as the multiple call chains to be tested.

12. The device according to any one of claims 7 to 11, characterized in that The acquisition module is further used to acquire a target candidate root cause from among the plurality of second candidate root causes selected by the user; The processing module is further configured to optimize a plurality of weights based on the target candidate root causes, wherein the plurality of weights are weights corresponding to the multi-dimensional statistical information when the plurality of first candidate root causes are sorted.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device of the at least one computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.

Images