Alarm root cause recommendation method and apparatus, and computer-readable storage medium

Abstract

Provided in the present invention are an alarm root cause recommendation method and apparatus, and a computer-readable storage medium, which can be applied to a micro-service system comprising a plurality of micro-services. The method comprises: acquiring a plurality of normal call chains and a plurality of call chains under test, the plurality of normal call chains being call chains of a plurality of normal micro-service interfaces in the micro-service system, and the plurality of call chains under test being call chains of abnormal micro-service interfaces in the micro-service system; then, on the basis of the plurality of call chains under test and the plurality of normal call chains, determining a plurality of candidate root causes, and performing statistical analysis on information of the plurality of call chains under test to obtain multi-dimensional statistical information corresponding to the plurality of candidate root causes; and then, on the basis of the multi-dimensional statistical information corresponding to the plurality of candidate root causes, sorting the plurality of candidate root causes so as to provide to a user the plurality of sorted candidate root causes. This method can recommend root causes, and operation and maintenance personnel can select a target root cause from the recommended root causes for anomaly repair, thereby improving the efficiency of root cause location and anomaly repair.

Description

Alarm root cause recommendation method, device and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application with application number 202311083390.9 filed with the State Intellectual Property Office of China on August 26, 2023, and priority to the Chinese patent application with invention name “Fault root cause analysis method, device and computer-readable storage medium”, as well as priority to the Chinese patent application with application number 202311489512.4 filed with the State Intellectual Property Office of China on November 9, 2023, and priority to the Chinese patent application with invention name “Alarm root cause recommendation method, device and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of operation and maintenance technology, and in particular to a method, device, and computer-readable storage medium for recommending an alarm root cause. Background Art

[0003] With the continuous development of cloud-native technologies, the microservices architecture has become the preferred architecture for modern business systems. In a microservices architecture, each microservice is independent and can be deployed, scaled, and maintained independently. This design makes the system more flexible and can more quickly respond to changing business needs.

[0004] However, microservice systems contain a large number of microservices, each independent and heterogeneous, and typically deployed in a distributed manner across different nodes (such as servers, virtual machines, containers, and chips). Consequently, the call relationships within a microservice system are complex, and failures are not confined to a single node. They propagate along the call relationships and spread to other nodes, potentially triggering large-scale fault alerts. Currently, when a large number of anomalies occur simultaneously in a microservice system, operations and maintenance personnel are tasked with root cause identification. However, this analysis is extremely stressful, as pinpointing the root cause (e.g., an abnormally long request time for a microservice interface, a missing microservice interface, or increased network latency for a microservice interface) often takes hours or even days, resulting in missed opportunities for troubleshooting and potentially leading to network incidents.

[0005] Summary of the Invention

[0006] The present application provides an alarm root cause recommendation method, device and computer-readable storage medium, which can recommend root causes when an anomaly occurs in a microservice system. The recommended root causes can provide a good reference for operation and maintenance personnel. Operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes and repair the anomaly based on the selected root cause, thereby improving the efficiency of root cause location and anomaly repair.

[0007] In a first aspect, a method for recommending an alarm root cause is provided, which is applied to a microservice system including multiple microservices, and the microservice system also includes a root cause recommendation device, the method including the following steps: the root cause recommendation device obtains multiple normal call chains and multiple call chains to be tested, the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system; multiple first candidate root causes are determined based on the multiple call chains to be tested and the multiple normal call chains; information of the multiple call chains to be tested is counted to obtain multi-dimensional statistical information corresponding to the multiple first candidate root causes; multiple second candidate root causes are provided to the user, or multiple second candidate root causes and repair suggestions for each second candidate root cause are provided to the user, wherein the multiple second candidate root causes are obtained by sorting the multiple first candidate root causes based on the multi-dimensional statistical information corresponding to the multiple first candidate root causes.

[0008] In the above scheme, since the root cause recommendation device can obtain multiple normal call chains and multiple call chains to be tested of the microservice system, after determining multiple candidate root causes based on the multiple call chains to be tested and the multiple normal call chains, the multiple candidate root causes are sorted based on the multi-dimensional statistical information corresponding to the multiple candidate root causes, and the sorting results are provided to the operation and maintenance personnel of the microservice system for selection. The sorting results can provide a good reference for the operation and maintenance personnel. The operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes, and perform abnormal repair according to the selected root cause, thereby improving the efficiency of root cause location and abnormal repair.

[0009] In addition, since the root cause recommendation device can provide repair suggestions for each candidate root cause when providing the operation and maintenance personnel with multiple ranked candidate root causes, these repair suggestions can provide the operation and maintenance personnel with a better reference. After the operation and maintenance personnel select the root cause, they can refer to the repair suggestions corresponding to the selected root cause to perform abnormality repair, thereby further improving the efficiency of abnormality repair.

[0010] In one possible implementation, the multi-dimensional statistical information corresponding to each of the multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

[0011] By implementing the above-mentioned implementation method, since multiple first candidate root causes are sorted with reference to the call event dimension information of each first candidate root cause and the resource dimension information called by the call event, the candidate root causes with higher probability among the multiple first candidate root causes can be ranked in front, and the candidate root causes with lower probability can be ranked in the back, thereby providing users with more accurate recommendation results.

[0012] In one possible implementation, the first call event is a call event in which each first candidate root cause exists, and the first dimension statistical information corresponding to each of the above first candidate root causes includes at least one of the following: the frequency, average duration and end time of the first call event in the call chain to be tested to which the first call event belongs; the second dimension statistical information corresponding to each of the above first candidate root causes includes at least one of the following: the resource type called in multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource type called by the first call event and the nodes where the resources are located.

[0013] By implementing the above implementation, since relatively abundant information is referred to when ranking the plurality of first candidate root causes, the ranking result can be made more accurate.

[0014] In a possible implementation, the root cause recommendation device determines multiple first candidate root causes based on multiple call chains to be tested and multiple normal call chains, including:

[0015] The root cause recommendation device compares the root microservice interface of each call chain to be tested in multiple call chains to be tested with the root microservice interfaces of multiple normal call chains, compares the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and compares the log of each call chain to be tested with the log of multiple normal call chains to determine multiple first candidate root causes.

[0016] When implementing the above-mentioned implementation method, the root cause recommendation device compares the root microservice interface of each call chain to be tested in multiple call chains to be tested with the root microservice interfaces of multiple normal call chains when determining the candidate root cause, compares the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and compares the log of each call chain to be tested with the log of multiple normal call chains. Compared with some current technologies that only compare the root microservice interface of the call chain to be tested with the root microservice interface of the normal call chain when determining the candidate root cause, since the comparison granularity of this application is finer, the accuracy of the determined candidate root cause can be improved.

[0017] In one possible implementation, the root cause recommendation device obtains multiple call chains to be tested, including: the root cause recommendation device counts the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the exception to the current moment; based on the number of abnormal microservice interfaces of the microservice system in multiple time periods, a target time period is selected from multiple time periods, and the rate of change between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; the call chains of the abnormal microservice interfaces of the microservice system from the occurrence of the exception to the target time period are obtained as multiple call chains to be tested.

[0018] In one possible implementation, the method further includes: the root cause recommendation device obtaining a target candidate root cause from the plurality of second candidate root causes selected by the user; and optimizing multiple weights based on the target candidate root cause, wherein the multiple weights are weights corresponding to multi-dimensional statistical information when ranking the plurality of first candidate root causes. This improves the accuracy of root cause recommendations when subsequently using the alarm root cause recommendation method provided herein.

[0019] In a second aspect, a device for recommending an alarm root cause is provided, which is applied to a microservice system including multiple microservices, and the device includes:

[0020] An acquisition module is used to acquire multiple normal call chains and multiple call chains to be tested, wherein the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system;

[0021] a processing module, configured to determine a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains;

[0022] The processing module is further configured to collect statistics on the information of the plurality of call chains to be tested to obtain multi-dimensional statistical information corresponding to the plurality of first candidate root causes;

[0023] A recommendation module is configured to provide a plurality of second candidate root causes to a user, or to provide the plurality of second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the plurality of second candidate root causes are obtained by sorting the plurality of first candidate root causes based on multi-dimensional statistical information corresponding to the plurality of first candidate root causes.

[0024] In one possible implementation, the multi-dimensional statistical information corresponding to each first candidate root cause among the multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

[0025] In one possible implementation, the first call event is a call event in which each of the first candidate root causes exists, and the first dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the frequency, average duration and end time of the first call event in the call chain to be tested to which the first call event belongs; the second dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the resource type called in the multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource type called by the first call event and the nodes where the resources are located.

[0026] In one possible implementation, the processing module is specifically used to: compare the root microservice interface of each of the multiple call chains to be tested with the root microservice interface of the multiple normal call chains, compare the call relationship in each call chain to be tested with the call relationship in the multiple normal call chains, and compare the log of each call chain to be tested with the log of the multiple normal call chains to determine multiple first candidate root causes.

[0027] In one possible implementation, the processing module is further used to count the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the exception to the current moment; the processing module is further used to select a target time period from the multiple time periods based on the number of abnormal microservice interfaces of the microservice system in the multiple time periods, and the rate of change between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; the acquisition module is used to obtain the call chain of the abnormal microservice interface of the microservice system from the occurrence of the exception to the target time period as the multiple call chains to be tested.

[0028] In one possible implementation, the acquisition module is also used to obtain a target candidate root cause from the multiple second candidate root causes selected by the user; the processing module is also used to optimize multiple weights based on the target candidate root cause, wherein the multiple weights are the weights corresponding to the multi-dimensional statistical information when sorting the multiple first candidate root causes.

[0029] In a third aspect, a computing device cluster is provided, comprising a processor and a memory; the processor is configured to execute instructions stored in the memory, so that the computing device cluster implements the method provided in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and the instructions are used to implement the method provided in the first aspect or any possible implementation of the first aspect.

[0031] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when read and executed by a computing device, enables the computing device to perform the method provided in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of a call chain and a call chain log provided in an embodiment of the present application;

[0033] FIG2 is a schematic diagram of the structure of a microservice system provided in an embodiment of the present application;

[0034] FIG3 is a schematic diagram of the structure of another microservice system provided in an embodiment of the present application;

[0035] FIG4 is a flow chart of a method for recommending root causes of alarms according to an embodiment of the present application;

[0036] FIG5 is a diagram showing the distribution of the number of abnormal microservice interfaces in a microservice system from 21:00:00 to 21:30:00 according to an embodiment of the present application;

[0037] FIG6 is a schematic diagram of a process for comparing the i-th call chain to be tested with multiple normal call chains provided by an embodiment of the present application;

[0038] FIG7 is a schematic diagram of an embodiment of the present application showing an i-th call chain to be tested, multiple normal call chains, and call chain logs;

[0039] FIG8 is a schematic diagram of a process for obtaining first dimension statistical information and second dimension statistical information according to an embodiment of the present application;

[0040] FIG9 is a schematic diagram of multiple second sets exemplarily shown in an embodiment of the present application;

[0041] FIG10 is an exemplary diagram of first dimension statistical information and second dimension statistical information corresponding to multiple first candidate root causes in this application;

[0042] FIG11 is a schematic structural diagram of a root cause recommendation device provided in an embodiment of the present application;

[0043] FIG12 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0044] FIG13 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0045] FIG14 is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0047] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0048] In order to facilitate a clear understanding of the technical solution provided by this application, some nouns and terms involved in the technical solution provided by this application are first explained.

[0049] (1) Microservice is an architectural pattern that splits an application into a set of small, independently deployed services. Each microservice runs in its own process and can be implemented using different technology stacks. Each microservice is responsible for completing a specific business function. Each microservice can be deployed and upgraded independently without affecting the operation of other microservices. Microservices can communicate with each other through lightweight communication mechanisms, such as the Hypertext Transfer Protocol (HTTP) and message queues. Users can independently scale each microservice according to actual needs to improve the elasticity and scalability of the system.

[0050] (2) Call chain: When a request enters the system, it may need to be processed and called by multiple services (also called service interfaces) in the system before it can be responded to. The various service calls that implement the same request are called call events, and these call events form a call chain. Calling a service means calling the corresponding method in the interface that provides the corresponding service to implement the service, such as calling the MySQL method in interface A to access the MySQL database resource.

[0051] Each call chain has a unique identifier. When a request enters the system, a globally unique trace ID is generated to identify the request. All call log information of the request will record the ID, so that the call records of the request can be associated.

[0052] Each call has a unique identifier: After receiving a request, the server generates a unique identifier to mark the call. On the one hand, this identifier can be used to distinguish different calls in the call chain (it is possible that the same interface is called multiple times). On the other hand, this identifier is also passed to the sub-call (the next service) that needs to be executed during this service process and serves as the parent call identifier of the sub-call, thus forming a hierarchical relationship between adjacent service calls (referred to as a call relationship). Each call can be called a time span (span). The unique identifier of the call is called the time span identifier (span_id), which is generated by the server and passed to the next call for use.

[0053] When the server receives a request, it records the trace ID, span_id, and call start time (start timestamp) in the call log. When the call ends, it records the end time (finish timestamp) in the call log. This allows the order of different service calls to be sorted and the time consumption of service calls to be analyzed.

[0054] Through the above information, the entire call chain of a request can be associated: all call logs are found through trace_id, the call relationship is formed through span_id and the parent span_id of each call, and the order of calls is formed through timestamp, thus forming a directed acyclic graph. This directed acyclic graph is the call chain. See Figure 1 for an example of a call chain (indicated by a solid arrow) and the call chain log.

[0055] In Figure 1, the microservice interface details corresponds to two call logs, which means that the microservice interface details is called twice by the microservice interface productpage.

[0056] The following introduces the application scenarios of the embodiments of the present application.

[0057] With the continuous development of cloud-native technologies, microservices architecture has become the preferred architecture for modern business systems. Compared to traditional monolithic application architectures, microservices architectures offer greater scalability, maintainability, and reliability. In a microservices architecture, each microservice is independent and can be deployed, scaled, and maintained independently. This design makes the system more flexible and allows for faster response to changing business needs.

[0058] Referring to the microservice system 200 shown in FIG2 , the system 200 includes microservices such as product microservices, order microservices, and user microservices. These microservices are independently deployed on different computing devices (also referred to as computing nodes).

[0059] A user can operate a local device 100 and send a request to the microservice system 200 via a communication network, requesting the use of microservices such as product microservices, order microservices, and user microservices provided by the microservice system 200. The local device 100, also referred to as a terminal device 100, can be any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a virtual reality (VR) device, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0060] The computing devices used to provide microservices in microservice system 200 may include, but are not limited to, physical servers, virtual machines, containers, chips, etc., and are not specifically limited here. Physical servers include, but are not limited to, ARM servers and X86 servers. A virtual machine refers to a complete computer system implemented using network function virtualization (NFV) technology, emulated through software with complete hardware system functions, and running in a completely isolated environment. A container refers to a group of processes that are resource-constrained and isolated from each other.

[0061] However, the number of microservices in microservice system 200 is large, each microservice is independent and heterogeneous, and each microservice is typically deployed in a distributed manner on different nodes (such as servers, virtual machines, containers, chips, etc.). Therefore, the call relationships in microservice system 200 are very complex, and the occurrence of failures is not limited to a single node. It will propagate along the call relationships and spread to other nodes. As a result, a single point of failure may trigger a large-scale fault alarm. Currently, when a large number of anomalies occur simultaneously in microservice system 200, operations and maintenance personnel are responsible for root cause identification. However, the analysis pressure on operations and maintenance personnel is very high. It often takes hours or even days to locate the actual root cause, resulting in missing the optimal time for fault handling and causing network accidents.

[0062] In order to solve the above problems, the embodiments of the present application provide an alarm root cause recommendation method and device, etc., which can recommend root causes when an abnormality occurs in the microservice system 200. The recommended root causes can provide a good reference for the operation and maintenance personnel of the microservice system 200. The operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes, and repair the abnormality according to the selected root cause, thereby improving the efficiency of root cause location and abnormality repair.

[0063] First, please refer to Figure 3, which is a structural diagram of a microservice system 200 provided in an embodiment of the present application. As shown in Figure 3, in addition to including computing devices that provide various microservices, the system 200 also includes a root cause recommendation device 210.

[0064] Root cause recommendation device 210, also known as alarm root cause recommendation device, can be a software module deployed on a single computing device. Alternatively, root cause recommendation device 210 can be implemented as hardware within the computing device, for example, by a processor within the computing device. Alternatively, root cause recommendation device 210 can be a software module deployed within the computing device providing each microservice, or deployed on a single or multiple computing devices independent of the computing devices providing each microservice (this deployment is used as an example in FIG3 ). Root cause recommendation device 210 can be connected to the computing devices providing each microservice via a communication network using any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0065] The computing devices in Figure 3 may include, but are not limited to, physical servers, virtual machines, containers, chips, and the like, and are not specifically limited here. Physical servers may include, but are not limited to, ARM servers and X86 servers. The root cause recommendation device 210 is responsible for recommending root causes when an anomaly occurs in the microservice system 200 (e.g., an alarm is generated or the response time of a microservice interface exceeds the normal response time).

[0066] In order to facilitate a clearer understanding of the process of performing root cause recommendation by the root cause recommendation device 210 in the microservice system 200 shown in FIG3 , a detailed introduction is given below in conjunction with a flow chart of an alarm root cause recommendation method provided in an embodiment of the present application shown in FIG4 .

[0067] As shown in FIG4 , the method may include the following steps:

[0068] S401 : The root cause recommendation device 210 obtains a plurality of normal call chains, where the plurality of normal call chains are call chains of a plurality of normal microservice interfaces in the microservice system 200 .

[0069] S402 : The root cause recommendation device 210 obtains a plurality of call chains to be tested, where the plurality of call chains to be tested are call chains of abnormal microservice interfaces in the microservice system 200 .

[0070] S403: The root cause recommendation device 210 determines a plurality of first candidate root causes based on a plurality of call chains to be tested and a plurality of normal call chains.

[0071] S404: The root cause recommendation device 210 collects statistics on information of multiple call chains to be tested, and obtains multi-dimensional statistical information corresponding to multiple first candidate root causes.

[0072] S405: The root cause recommendation device 210 sorts the plurality of first candidate root causes based on the multi-dimensional statistical information corresponding to the plurality of first candidate root causes to obtain sorted plurality of first candidate root causes.

[0073] S406: The root cause recommendation device 210 provides the user with a plurality of sorted first candidate root causes, or provides the user with a plurality of sorted first candidate root causes and a repair suggestion for each first candidate root cause.

[0074] In order to facilitate understanding of the embodiments of the present application, the microservice interfaces (including normal microservice interfaces and abnormal microservice interfaces) involved in the embodiments of the present application and the call events with candidate root causes are first introduced.

[0075] Microservice interfaces are interfaces that provide users with various microservices, such as interfaces for accessing the product microservice / order microservice / user microservice shown in Figure 2, interfaces for accessing database resources, interfaces for accessing network resources, etc.

[0076] The embodiments of the present application relate to normal microservice interfaces and abnormal microservice interfaces. A normal microservice interface refers to a microservice interface with normal response time, and an abnormal microservice interface refers to a microservice interface with abnormal response time. The microservice interface response time refers to the period from when the microservice interface receives a call request to when the call ends. If the microservice interface response time is less than or equal to the normal response time, the microservice interface response time is determined to be normal, thereby determining that the microservice interface is normal. Otherwise, the microservice interface response time is determined to be abnormal, thereby determining that the microservice interface is abnormal. The normal response time corresponding to each microservice interface can be set according to the actual scenario, and this application does not make specific limitations. In this application, algorithms such as box plots, histograms, and scatter plots can be used to detect whether the microservice interface response time is normal. For the sake of brevity of the specification, they will not be elaborated here.

[0077] For call events with candidate root causes, for example, assuming that call event C is a call to the MySQL method in microservice interface B on node N1, subsequent analysis shows that call event C has candidate root cause RC1 "Increased time consumption for calling the MySQL method in microservice interface B on node N1". In this case, call event C "Calling the MySQL method in microservice interface B on node N1" is called a call event with candidate root cause RC1 "Increased time consumption for calling the MySQL method in microservice interface B on node N1".

[0078] The following is a detailed introduction to each step shown in FIG4 .

[0079] In S401, the multiple normal call chains corresponding to the multiple normal microservice interfaces in the microservice system 200 are the call chains of all the microservice interfaces in the microservice system 200 when the microservice system 200 is normal. A normal microservice interface may correspond to at least one normal call chain in the multiple normal call chains, and any two normal microservice interfaces may correspond to different normal call chains.

[0080] The root cause recommendation device 210 may obtain multiple normal call chains in the following manner: the root cause recommendation device 210 receives logs of multiple normal call chains sent by the microservice system 200 , and obtains multiple normal call chains based on the logs of the multiple normal call chains.

[0081] In the embodiment of the present application, the multiple normal call chains acquired by the root cause recommendation device 210 include not only the structure of the normal call chain (similar to that shown by the solid arrow in FIG. 1 ), but also the logs of the normal call chain.

[0082] In S402 , the root cause recommendation device 210 may obtain multiple call chains to be tested by receiving logs of the multiple call chains to be tested sent by the microservice system 200 and obtaining the multiple call chains to be tested based on the logs of the multiple call chains to be tested.

[0083] In the embodiment of the present application, the multiple call chains to be tested obtained by the root cause recommendation device 210 include not only the structure of the call chains to be tested (similar to that shown by the solid arrows in FIG. 1 ), but also the logs of the call chains to be tested.

[0084] It can be understood that in actual scenarios, there may be one or more abnormal microservice interfaces in the microservice system 200, and each abnormal microservice interface may generate one or more call chains during the abnormal period. In a possible embodiment, the root cause recommendation device 210 can obtain all call chains of all abnormal microservice interfaces in the microservice system 200 from the time the abnormality occurs in the microservice system 200 to the current moment, as multiple call chains to be tested. Optionally, the root cause recommendation device 210 can also select some call chains from all call chains as multiple call chains to be tested. Among them, when an abnormality occurs in the microservice system 200, an alarm message can be generated for the microservice system 200 to indicate that an abnormality has occurred in the microservice system 200, or it can be determined that an abnormality has occurred in the microservice system 200 when an abnormal microservice interface is detected in the microservice system 200.

[0085] Next, the implementation process of the root cause recommendation device 210 selecting some call chains from all the above call chains as multiple call chains to be tested is described:

[0086] S1: Count the number of abnormal microservice interfaces of the microservice system 200 in multiple time periods from the occurrence of the abnormality to the current moment.

[0087] The multiple time periods may be time periods of equal length. For example, assuming that the microservice system 200 experiences an exception at 21:00:00 and the current time is 21:30:00, the multiple time periods may be every minute between 21:00:00 and 21:30:00, or every two minutes, etc., which is not specifically limited in this application. Preferably, the length of the multiple time periods is one minute.

[0088] S2: Based on the number of abnormal microservice interfaces of the microservice system 200 in multiple time periods, a target time period is selected from the multiple time periods, wherein a rate of change between the number of abnormal microservice interfaces of the microservice system 200 in the target time period and the number of abnormal microservice interfaces of the microservice system 200 in a time period before the target time period exceeds a threshold.

[0089] That is, the target time period is the time period in which the number of abnormal microservice interfaces corresponding to multiple time periods changes suddenly compared with the number of abnormal microservice interfaces corresponding to the previous time period.

[0090] The above-mentioned change rate can be a growth rate or a reduction rate, and the threshold value can be customized according to the actual scenario, and this application does not make any specific limitations.

[0091] In a possible embodiment, when the change rate is a growth rate, the threshold value may be set to a maximum growth rate / 2, where the maximum growth rate is the maximum value of the number of abnormal microservice interfaces of the microservice system 200 in each time period and the growth rate of the number of abnormal microservice interfaces of the microservice system 200 in the previous time period of each time period; when the change rate is a reduction rate, the threshold value may be set to a maximum reduction rate / 3, where the maximum reduction rate is the maximum value of the number of abnormal microservice interfaces of the microservice system 200 in each time period and the reduction rate of the number of abnormal microservice interfaces of the microservice system 200 in the previous time period of each time period.

[0092] It can be understood that if the growth rate corresponding to the target time period exceeds the threshold, it indicates that the number of abnormal microservice interfaces in microservice system 200 increased suddenly during the target time period. This usually means that microservice system 200 experienced a failure before the target time period. Root cause analysis is primarily used to detect the cause of a failure at the onset of the failure. Therefore, the root cause should be located before the target time period. If the decline rate corresponding to the target time period exceeds the threshold, it indicates that the number of abnormal microservice interfaces in microservice system 200 decreased suddenly during the target time period. This usually means that the failure of microservice system 200 was recovered during the target time period, and the root cause should be located before the target time period.

[0093] S3: Acquire the call chains of the abnormal microservice interfaces of the microservice system 200 from the occurrence of the abnormality to the target time period as multiple call chains to be tested.

[0094] In a specific implementation, if each abnormal microservice interface generates a large number of call chains from the time when the microservice system 200 becomes abnormal to the target time period, then some call chains can be selected from the large number of call chains generated by the abnormal microservice interface as the call chains to be tested corresponding to the abnormal microservice interface. The selected partial call chains can be call chains with abnormal response times in the call chains. Further, call chains with earlier start times and longer response times can be selected from more call chains. This is because call chains with earlier start times and longer response times are usually call chains with root causes (such as call events with time-consuming abnormalities in the call chains). Among them, the response time of the call chain can be the period from the start call time to the end call time of the call chain. The abnormal response time of the call chain can be comparing the response time of the call chain with the normal response time of the call chain. When the response time of the call chain is less than or equal to the normal response time of the call chain, it is determined that the response time of the call chain is normal. Otherwise, it is determined that the response time of the call chain is abnormal. Among them, the normal response time of the call chain can be customized according to the actual scenario, and this application does not make specific restrictions on this.

[0095] In a possible embodiment, if the time period from the occurrence of an exception in the microservice system 200 to the target time period is relatively short, such as a few minutes, the call chain of the abnormal microservice interface of the microservice system 200 within these few minutes can be directly obtained as multiple call chains to be tested; if the time period from the occurrence of an exception in the microservice system 200 to the target time period is a very long period of time, such as more than ten minutes or even more than half an hour, a continuous time period (such as a few minutes) including the target time period can be intercepted from the longer time period, and the call chain of the abnormal microservice interface of the microservice system 200 within the intercepted time period can be obtained as multiple call chains to be tested. This application does not specifically limit the intercepted time period.

[0096] For example, refer to Figure 5, which is an illustrative distribution diagram of the number of abnormal microservice interfaces of the microservice system 200 from 21:00:00 to 21:30:00 in an embodiment of the present application. It can be seen from Figure 5 that the number of abnormal microservice interfaces in the microservice system 200 began to increase sharply at 21:10:00. Therefore, it can be inferred that the resources of the microservice system 200 reached the upper limit at 21:10:00, resulting in a substantial increase in the number of abnormal microservice interfaces. Since root cause analysis is mainly used to detect the cause of the failure at the beginning of the failure, the call chain of the abnormal microservice interface before 21:10:00 can be obtained for analysis, such as obtaining the call chain of the abnormal microservice interface between 21:03:00-21:10:00 as the call chain to be tested for analysis. It can also be seen from Figure 5 that the number of abnormal microservice interfaces in microservice system 200 began to drop suddenly around 21:22:00. Therefore, it can be inferred that the failure of microservice system 200 ended at 21:22:00. Optionally, the call chain of the abnormal microservice interface between 21:00:00 and 21:22:00 can also be obtained as the call chain to be tested for analysis.

[0097] It should be understood that the above S1 to S3 are merely exemplary implementations of the root cause recommendation device 210 obtaining multiple call chains to be tested, and this application does not specifically limit them. For example, in a specific implementation, S1 can be the ratio of the number of abnormal microservice interfaces of the microservice system 200 in multiple time periods from the occurrence of the exception to the current moment to the total number of microservice interfaces of the microservice system 200, and S2 can be based on the ratio of the number of abnormal microservice interfaces of the microservice system 200 in multiple time periods to the total number of microservice interfaces, and a target time period is selected from multiple time periods, wherein the ratio of the number of abnormal microservice interfaces of the microservice system 200 in the target time period to the total number of microservice interfaces, and the ratio of the number of abnormal microservice interfaces of the microservice system 200 in the time period before the target time period to the total number of microservice interfaces, exceeds a threshold, and S3 remains unchanged.

[0098] In S403 , the root cause recommendation device 210 may determine a plurality of first candidate root causes by comparing a plurality of call chains to be tested with a plurality of normal call chains.

[0099] Specifically, multiple first candidate root causes can be determined by comparing the root microservice interface of each call chain to be tested (referring to the microservice interface that is called earliest in the call chain) with the root microservice interfaces of multiple normal call chains, comparing the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and comparing the log of each call chain to be tested with the log of multiple normal call chains.

[0100] Taking the i-th call chain to be tested as an example, refer to FIG6 as a schematic flow chart of comparing the i-th call chain to be tested with multiple normal call chains.

[0101] S601: Determine whether there is a root microservice interface in the root microservice interfaces of multiple normal call chains that is the same as the root microservice interface of the i-th call chain to be tested. If it is determined that the same root microservice interface exists, execute S602 and S603. If it is determined that the same root microservice interface does not exist, execute S607, that is, analyze the root microservice interface of the i-th call chain to be tested to obtain the first candidate root cause.

[0102] The aforementioned same root microservice interface means that the root microservice interface identifier and the root microservice interface calling method are the same.

[0103] Taking the i-th call chain to be tested and multiple normal call chains shown in Figure 7 as an example, in Figure 7, the root microservice interface of the i-th call chain to be tested is the microservice interface productpage, and the calling method is GET. The multiple normal call chains include normal call chain 1, normal call chain 2..., among which the root microservice interface of normal call chain 1 is the microservice interface productpage, and the calling method is GET, and the root microservice interface of normal call chain 2 is the microservice interface bluewhare, and the calling method is POST. Therefore, it can be determined that the root microservice interface of normal call chain 1 is the same as the root microservice interface of the i-th call chain to be tested, and the root microservice interface of normal call chain 2 is different from the root microservice interface of the i-th call chain to be tested.

[0104] There is no normal call chain in multiple normal call chains that is the same as the root microservice interface of the i-th call chain to be tested. This may include the following situations:

[0105] Case 1: There is no normal call chain with the same root microservice interface identifier X (productpage as shown in Figure 7) as the root microservice interface identifier X of the i-th call chain to be tested among multiple normal call chains. In this case, the first candidate root cause may be: microservice interface X does not exist, or the microservice interface X requested by the client does not exist, or the microservice interface X does not exist in the microservice system 200, etc.

[0106] Case 2: Among multiple normal call chains, there is a normal call chain with the same root microservice interface identifier X as the i-th call chain to be tested, but with a different root microservice interface calling method. In this case, the first candidate root cause may be: microservice interface X does not have calling method Y (referring to the calling method of the root microservice interface X of the i-th call chain to be tested, such as the calling method GET of the root microservice interface productpage of the i-th call chain to be tested as shown in Figure 7), or the client calls the microservice interface productpage using an incorrect method, or the client requests an incorrect HTTP method, etc.

[0107] S602: Determine, from the calling relationships of multiple normal calling chains, a calling relationship that best matches the calling relationship in the i-th calling chain to be tested.

[0108] Specifically, the similarity between the call relationship in the i-th call chain to be tested and the call relationships in multiple normal call chains can be calculated, and the call relationship with the greatest similarity between the call relationship in the multiple normal call chains and the call relationship in the i-th call chain to be tested is determined as the most matching call relationship. The similarity can be calculated using Euclidean distance, cosine similarity, Pearson similarity, or Jaccard similarity, etc., which is not specifically limited in this application.

[0109] S603: Determine whether the best-matching call relationship is consistent with the call relationship in the i-th call chain to be tested. If the call relationships are consistent, execute S604 and S605. If the call relationships are inconsistent, execute S608, that is, perform a difference analysis on the best-matching call relationship and the call relationship in the i-th call chain to be tested to obtain the first candidate root cause.

[0110] Continuing with FIG. 7 as an example, it can be seen from FIG. 7 that the calling relationship of the normal call chain 1 is the same as the calling relationship in the i-th call chain to be tested.

[0111] Assume that the call relationship of normal call chain 1 is the call relationship that best matches the call relationship in the i-th call chain to be tested. The difference between the call relationship of normal call chain 1 and the i-th call chain to be tested is that: the call relationship of normal call chain 1 does not contain the microservice interface reviewsv2 calling the microservice interface ratings, while the call relationship in the i-th call chain to be tested contains the microservice interface reviewsv2 calling the microservice interface ratings. At this time, the first candidate root cause can be: the microservice interface ratings is missing, or the microservice interface ratings is incorrect, etc.

[0112] S604: Determine, from the logs of multiple normal call chains, a log that best matches the log of the i-th call chain to be tested.

[0113] Specifically, the similarity between the log of the i-th call chain to be tested and the logs of multiple normal call chains can be calculated. The log with the greatest similarity between the multiple normal call chain logs and the log of the i-th call chain to be tested is determined as the best match. The similarity can be calculated using Euclidean distance, cosine similarity, Pearson similarity, or Jaccard similarity, which is not specifically limited in this application.

[0114] S605: Determine whether the best-matching log is consistent with the log of the i-th call chain to be tested. If the logs are consistent, execute S606, that is, determine that the first candidate root cause does not exist in the i-th call chain to be tested. If the logs are inconsistent, execute S609, that is, perform a difference analysis on the best-matching log and the log of the i-th call chain to be tested to obtain the first candidate root cause.

[0115] The above-mentioned log consistency refers to the consistency of attributes of the same call event recorded in the log, wherein the event attribute may be event duration and / or event frequency, etc.

[0116] Taking the event attribute of event duration as an example, if the difference between the duration of all call events recorded in the most matching log and the duration of the same call event recorded in the log of the i-th call chain to be tested is less than the duration threshold, then the two logs are determined to be consistent. Otherwise, the two logs are determined to be inconsistent. Assume that the duration of call event C recorded in the most matching log is much greater than the duration of call event C recorded in the log of the i-th call chain to be tested, that is, the difference between the duration of call event C recorded in the most matching log and the duration of call event C recorded in the log of the i-th call chain to be tested is greater than the duration threshold. Assuming that call event C is a Mysql method that calls microservice interface B, the first candidate root cause can be: the Mysql method of microservice interface B has increased in duration.

[0117] Taking the event attribute as event frequency, if the difference between the frequency of all call events recorded in the most matching log and the frequency of the same call events recorded in the log of the i-th call chain to be tested is less than the frequency threshold, then the two logs are determined to be consistent; otherwise, the two logs are determined to be inconsistent. Assume that the frequency of call event C recorded in the most matching log is much greater than the frequency of call event C recorded in the log of the i-th call chain to be tested, that is, the difference between the frequency of call event C recorded in the most matching log and the frequency of call event C recorded in the log of the i-th call chain to be tested is greater than the frequency threshold. Assuming that call event C is a Mysql method call of microservice interface B, the first candidate root cause can be: the Mysql method call of microservice interface B is too frequent.

[0118] It should be understood that the description information of the first candidate root cause listed above is only an example, and this application does not specifically limit the description information of the first candidate root cause.

[0119] In S404 , based on the above content, after obtaining multiple first candidate root causes, the root cause recommendation device 210 may collect statistics on information of multiple call chains to be tested based on the multiple first candidate root causes to obtain multi-dimensional statistical information corresponding to each of the multiple first candidate root causes.

[0120] The multi-dimensional statistical information corresponding to each first candidate root cause includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the call event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the call event in which each first candidate root cause exists.

[0121] For the sake of convenience of description, taking the i-th first candidate root cause as an example, call event i represents the call event with the i-th first candidate root cause, and the first dimension statistical information corresponding to the i-th first candidate root cause includes at least one of the following: the frequency, average duration and end time of call event i in the call chain to be tested to which call event i belongs; the second dimension statistical information corresponding to the i-th first candidate root cause includes at least one of the following: the frequency, average duration and earliest start time of call events whose resource types called in multiple call chains to be tested and the resource nodes where the resources are located are the same as the resource types called by call event i and the resource nodes where the resources are located. Among them, resource types can include Mysql, message queues, input / output (IO) and network bandwidth. It should be understood that the specific division method of resource types can be set according to the actual scenario, and this application does not make specific limitations on this.

[0122] It should be understood that the information types included in the first dimension statistical information corresponding to the i-th first candidate root cause and the information types included in the second dimension statistical information are merely examples and should not be considered as specific limitations. For example, in one possible embodiment, the first dimension statistical information also includes the start time of call event i in the call chain to be tested to which call event i belongs, and the second dimension statistical information also includes the resource types and resource nodes called in multiple call chains to be tested, and the latest end time of the call event that has the same resource type and resource node as the call event i.

[0123] The following describes in detail the process of obtaining the statistical information of the first dimension and the statistical information of the second dimension with reference to a specific example, as shown in the flowchart of FIG8 .

[0124] S801: Obtain the frequency, average duration, and end time of the call event i in the call chain to be tested to which the call event i belongs.

[0125] The call event i in the call chain to be tested to which the above-mentioned call event i belongs may occur once or multiple times. When the call event i occurs once, the end time is the end time of the call event i. When the call event i occurs multiple times, the end time can be the end time of the call event i that starts first among the multiple call events i. Optionally, it can also be the end time of any call event i.

[0126] The end time of call event i is obtained instead of its start time. This is because for a call chain to be tested, the exception that is triggered first is likely to cause the occurrence of subsequent exceptions (for example, the failure of the database read and write event operation during the request processing leads to the failure of the data synchronization event). Usually, there is no case where a later exception causes the occurrence of an earlier exception. In other words, the candidate root cause corresponding to the call event with the earlier end time is more likely to be the real root cause. Therefore, in this application, the end time of the call event is obtained as a reference factor for sorting multiple first candidate root causes.

[0127] S802: Divide the call events in the multiple call chains to be tested according to the occurrence time and the end time to obtain multiple first sets, where any two of the multiple first sets correspond to non-overlapping time periods.

[0128] Taking 1143 call events in multiple call chains to be tested as an example, assuming that 57 of them occur between 21:09:59 and 21:10:00, 86 of them occur between 21:10:31 and 21:10:32, and the remaining 1000 occur between 21:10:35 and 21:13:31, we can obtain the three first sets shown in Table 1.

[0129] Table 1 Multiple first sets

[0130] It should be understood that Table 1 is merely an example of multiple first sets and should not be considered as a specific limitation.

[0131] S803: Divide the call events in each first set according to the called resource type and the node where the resource is located, and obtain multiple second sets corresponding to each first set. Any two second sets in the multiple second sets correspond to different resource types and different nodes.

[0132] Taking resource types including Mysql, message queue, IO and network bandwidth as an example, assuming that among the 1000 call events contained in the first set 3 of Table 1, 300 of the call events call the resource type of Mysql type, 55 of the call events call the resource type of message queue type, 600 of the call events call the resource type of IO type, and 45 of the call events call the resource type of network bandwidth type. Among the 300 call events where the resource type is Mysql type, 282 of the call events are located on the node N1, and the remaining 18 of the call events are located on the node N2. Among the 55 call events, 43 of the calls were to node N1, and the remaining 12 were to node N2. Among the 600 call events where the resource type was IO, 545 were to node N1, and the remaining 55 were to node N2. Among the 45 call events where the resource type was network bandwidth, 38 were to node N1, and the remaining 7 were to node N2. The 908 call events in the first set 3 are divided according to the resource type called and the node where the resource is located, and the resulting multiple second sets are shown in FIG9 .

[0133] S804: Counting the average time consumption of all call events in each second set (see the average time consumption in each second set shown in FIG9 ) and the earliest start time (not shown in FIG9 ).

[0134] S805: Map each second set to a corresponding first candidate root cause among the multiple first candidate root causes according to the resource type called by the call event and the node where the resource is located.

[0135] Taking the i-th first candidate root cause as an example, assuming that there is a call event i of the i-th first candidate root cause, which is a Mysql method that calls the microservice interface B on the node N1. Since the second set 1 corresponds to the node N1 and the Mysql resource type among the multiple second sets shown in Figure 9, the second set 1 can be mapped to the i-th first candidate root cause.

[0136] It can be seen that based on the example shown in Figure 8, the first dimension statistical information and the second dimension statistical information corresponding to the i-th first candidate root cause are obtained. It can be understood that the method of obtaining the first dimension statistical information and the second dimension statistical information corresponding to the i-th first candidate root cause as shown in Figure 8 can be used to obtain more first dimension statistical information and the second dimension statistical information corresponding to the first candidate root cause. For the sake of brevity, this will not be elaborated here.

[0137] Please refer to Figure 10. Figure 10 is the first dimension statistical information and second dimension statistical information corresponding to multiple first candidate root causes exemplified in this application. It should be noted that in Figure 10, interface is the abbreviation of microservice interface. Taking the first candidate root cause "the Mysql method time consumption of interface B has increased" in the row with sequence number 1 in Figure 10 and the first candidate root cause "the Mysql method time consumption of interface B has increased" in the row with sequence number 2 as examples, it should be noted that these two first candidate root causes are obtained based on the analysis of different call chains to be tested. For example, the first candidate root cause "the Mysql method time consumption of interface B has increased" in the row with sequence number 1 is obtained based on the analysis of the call chain to be tested with trace id f1041, and the first candidate root cause "the Mysql method time consumption of interface B" in the row with sequence number 2 is obtained based on the analysis of the call chain to be tested with trace id f1042.

[0138] In Figure 10, "Frequency of exceptions of the same type of calls" indicates the frequency of calling event i in the call chain to be tested to which calling event i belongs, and "Number of resource calls of the same type" indicates the frequency of calling events whose resource types and resource nodes called in multiple call chains to be tested are the same as those of calling event i, whose resource types and resource nodes are the same.

[0139] Continuing with the first candidate root cause "Increased time consumption of the Mysql method of interface B" in the row numbered 1 in Figure 10 as an example, it can be seen from Figure 10 that the first dimension statistical information corresponding to the first candidate root cause includes: the frequency of the call event (i.e., calling the Mysql method of interface B) in the call chain to be tested to which the call event with the first candidate root cause (i.e., calling the Mysql method of interface B) belongs is 3, the average time consumption is 40666.67 (in milliseconds), and the end time is 21:11:06 on August 14, 2023; the second dimension statistical information includes: the frequency of the call event in multiple call chains to be tested, the resource type called, the node where the resource is located, the resource type (i.e., Mysql type) called by the call event (i.e., calling the Mysql method of interface B), and the node where the resource is located (i.e., node N1) are the same, the average time consumption is 282, the earliest start time is 21:11:02 on August 14, 2023.

[0140] It should be understood that the contents in Figures 9 and 10 are merely examples provided in this application and should not be regarded as specific limitations.

[0141] In S405, the root cause recommendation device 210 may apply a weight to each dimension of the multidimensional statistical information, then encode the multidimensional statistical information corresponding to each first candidate root cause into a multidimensional vector, multiply each element in the multidimensional vector by the corresponding weight, and then add the results to obtain a weighted sum (also referred to as a score). After calculating the multiple weighted sums corresponding to the multiple first candidate root causes, the multiple first candidate root causes may be ranked based on the multiple weighted sums, as shown in the last column of Table 2 for the ranking scores corresponding to the multiple first candidate root causes.

[0142] In S406, taking the first candidate root cause "the Mysql method of interface B consumes more time" as an example, the root cause recommendation device 210 may provide the following repair suggestions for the candidate root cause:

[0143] Repair suggestion (1), troubleshooting: a. Use monitoring tools (such as Data Dog) or MySQL's slow query log tool to find out the query statements that take a long time to execute; b. Analyze the query statements to determine whether there are potential performance issues, such as not using indexes, table scans, complex join operations, etc.

[0144] Repair suggestion (2), optimize query statements: a. Review query statements to ensure that correct table joins, group by, and order by conditions are used, and avoid using select *; b. Use the explain command to analyze query statements and understand their execution plans so that corresponding optimizations can be performed.

[0145] It can be understood that when the root cause recommendation device 210 provides the user with a plurality of sorted first candidate root causes, if it also provides the user with repair suggestions for each first candidate root cause, compared to only providing the plurality of sorted first candidate root causes, it can provide the user with a better reference, making it easier for the user to refer to the repair suggestions to repair the anomaly.

[0146] In a possible embodiment, when providing the ranked multiple first candidate root causes to the user, the root cause recommendation device 210 may also provide the ranking scores corresponding to the multiple first candidate root causes for the user's reference.

[0147] In a possible embodiment, after providing the user with the sorted multiple first candidate root causes, the root cause recommendation device 210 can also obtain the target candidate root cause selected by the user from the sorted multiple first candidate root causes, and then optimize the weights corresponding to the multi-dimensional statistical information when sorting the multiple first candidate root causes based on the target candidate root cause, so as to improve the accuracy of the root cause recommendation when the alarm root cause recommendation method provided in the embodiment of the present application is used to recommend the root cause in the future. The algorithm for optimizing the weight can be Bayesian optimization (such as tree-structured Bayesian optimization (tree-structured parzen estimator, TPE) algorithm), random search, genetic algorithm, simulated annealing algorithm, reinforcement learning algorithm, etc., and this application does not make specific restrictions on this.

[0148] In summary, in the alarm root cause recommendation method provided by the present application, the root cause recommendation device 210 can obtain multiple normal call chains and multiple call chains to be tested of the microservice system 200. After determining multiple candidate root causes based on the multiple call chains to be tested and the multiple normal call chains, the multiple candidate root causes are sorted based on the multi-dimensional statistical information corresponding to the multiple candidate root causes, and the sorting results are provided to the operation and maintenance personnel of the microservice system 200. The sorting results can provide a good reference for the operation and maintenance personnel. The operation and maintenance personnel can select the most likely root cause with the highest ranking from the recommended root causes, and perform abnormal repair according to the selected root cause, thereby improving the root cause location efficiency and abnormal repair efficiency.

[0149] When providing the sorted candidate root causes to the operation and maintenance personnel, the root cause recommendation device 210 can provide repair suggestions for each candidate root cause. These repair suggestions can provide a better reference for the operation and maintenance personnel. After the operation and maintenance personnel select the root cause, they can refer to the repair suggestions corresponding to the selected root cause to perform abnormality repair, thereby further improving the efficiency of abnormality repair.

[0150] The above describes in detail the method of the embodiment of the present application. In order to facilitate the better implementation of the above scheme of the embodiment of the present application, correspondingly, the following also provides relevant devices and equipment for cooperating in implementing the above scheme.

[0151] 11 is a schematic diagram of a root cause recommendation device 210 according to an embodiment of the present invention. As shown in FIG11 , the device 210 includes an acquisition module 211 , a processing module 212 , and a recommendation module 213 .

[0152] An acquisition module 211 is configured to acquire multiple normal call chains and multiple call chains to be tested, wherein the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system;

[0153] A processing module 212 is configured to determine a plurality of first candidate root causes based on a plurality of call chains to be tested and a plurality of normal call chains;

[0154] The processing module 212 is further configured to collect statistics on the information of the multiple call chains to be tested, and obtain multi-dimensional statistical information corresponding to the multiple first candidate root causes;

[0155] The recommendation module 213 is configured to provide a plurality of second candidate root causes to the user, or to provide a plurality of second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the plurality of second candidate root causes are obtained by sorting the plurality of first candidate root causes based on multi-dimensional statistical information corresponding to the plurality of first candidate root causes.

[0156] In some possible implementations, the multi-dimensional statistical information corresponding to each of the multiple first candidate root causes mentioned above includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

[0157] In some possible implementations, the first call event is a call event in which each first candidate root cause exists, and the first dimension statistical information corresponding to each of the above-mentioned first candidate root causes includes at least one of the following: the frequency, average duration and end time of the first call event in the call chain to be tested to which the first call event belongs; the second dimension statistical information corresponding to each of the above-mentioned first candidate root causes includes at least one of the following: the resource type called in multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource type called by the first call event and the nodes where the resources are located.

[0158] In some possible implementations, the processing module 212 is specifically used to: compare the root microservice interface of each call chain to be tested in multiple call chains to be tested with the root microservice interfaces of multiple normal call chains, compare the call relationship in each call chain to be tested with the call relationship in multiple normal call chains, and compare the log of each call chain to be tested with the log of multiple normal call chains to determine multiple first candidate root causes.

[0159] In some possible implementations, the processing module 212 is also used to count the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the exception to the current moment; the processing module 212 is also used to select a target time period from multiple time periods based on the number of abnormal microservice interfaces of the microservice system in multiple time periods, and the rate of change between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; the acquisition module 211 is used to obtain the call chains of the abnormal microservice interfaces of the microservice system from the occurrence of the exception to the target time period as multiple call chains to be tested.

[0160] In some possible implementations, the acquisition module 211 is further used to obtain a target candidate root cause from a plurality of second candidate root causes selected by the user; the processing module 212 is further used to optimize a plurality of weights based on the target candidate root cause, wherein the plurality of weights are weights corresponding to the multi-dimensional statistical information when sorting the plurality of first candidate root causes.

[0161] The acquisition module 211, processing module 212, and recommendation module 213 in all possible implementations described above can be implemented via software or hardware. For example, the implementation of acquisition module 211 will be described below using acquisition module 211 as an example. Similarly, the implementations of processing module 212 and recommendation module 213 can refer to the implementation of acquisition module 211.

[0162] As an example of a software functional unit, the acquisition module 211 may include a program running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module 211 may include a program running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the program may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the program may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0163] Similarly, multiple hosts / virtual machines / containers used to run the program can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0164] As an example of a hardware functional unit, the acquisition module 211 may include at least one computing device, such as a server. Alternatively, the acquisition module 211 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0165] The multiple computing devices included in acquisition module 211 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 211 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 211 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0166] It should be noted that the steps that the acquisition module 211, processing module 212, and recommendation module 213 are responsible for implementing can be specified as needed. The acquisition module 211, processing module 212, and recommendation module 213 respectively implement different steps in the alarm root cause recommendation method to realize all the functions of the device 210.

[0167] 12 is a schematic diagram of a computing device according to an embodiment of the present application. The computing device can be used to deploy the root cause recommendation apparatus 210 shown in FIG11 to implement the alarm root cause recommendation method described in the above method embodiment.

[0168] As shown in Figure 12, computing device 1200 includes: a bus 1201, a processor 1202, a memory 1203, a network interface 1204, and an output interface 1205. The processor 1202, the memory 1203, the network interface 1204, and the output interface 1205 communicate with each other via bus 1201. Computing device 1200 can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, computing device 1200 can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone. It should be understood that this application does not limit the number of processors and memories in computing device 1200.

[0169] Bus 1201 may be a standard peripheral component interconnect bus or an extended industry standard architecture bus. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG12 shows a single line, but this does not imply a single bus or a single type of bus. Bus 1201 may include a path for transmitting information between various components of computing device 1200 (e.g., processor 1202, memory 1203, network interface 1204, and output interface 1205).

[0170] Processor 1202 is comprised of one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an ASIC, a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), an FPGA, a GAL, a data processing unit (DPU), a system on a chip (SoC), or any combination thereof. Processor 1202 executes various types of digitally stored instructions, such as software or firmware programs stored in memory 1203, enabling computing device 1200 to provide a wide variety of services.

[0171] The memory 1203 may include a volatile memory, such as a random access memory, or a non-volatile memory, such as a read-only memory, a flash memory, a mechanical hard disk, or a solid-state drive.

[0172] Memory 1203 stores an executable program. Processor 1202 executes the executable program to implement the functions of acquisition module 211, processing module 212, and recommendation module 213, respectively, thereby implementing the alarm root cause recommendation method described in the above method embodiment. Specifically, memory 1203 stores instructions for executing the alarm root cause recommendation method described in the above method embodiment.

[0173] The network interface 1204 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to obtain files or network data.

[0174] The output interface 1205 uses, for example but not limited to, a wired interface to enable communication between the computing device 1200 and other devices or a communication network.

[0175] Referring to Figure 13, Figure 13 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. The computing device cluster provided in the present application includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0176] As shown in Figure 13, the computing device cluster includes at least one computing device 1200. The memory 1203 in one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the alarm root cause recommendation method.

[0177] In some possible implementations, the memory 1203 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the alarm root cause recommendation method. In other words, the combination of one or more computing devices 1200 can jointly execute the instructions for executing the alarm root cause recommendation method.

[0178] It should be noted that the memory 1203 in different computing devices 1200 in the computing device cluster can store different instructions, each for executing a portion of the functions of the root cause recommendation apparatus 210. In other words, the instructions stored in the memory 1203 in different computing devices 1200 can implement the functions of one or more of the aforementioned acquisition module 211, processing module 212, and recommendation module 213.

[0179] Referring to Figure 14, Figure 14 is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. In some possible implementations, one or more computing devices in the computing device cluster can be connected via a network. The network can be a wide area network or a local area network, etc. As shown in Figure 14, two computing devices 1200A and 1200B are connected via a network. Specifically, the network is connected to the network through the network interface and / or output interface in each computing device. In this type of possible implementation, the memory 1203 in the computing device 1200A stores instructions for executing the functions of the acquisition module 211 and the recommendation module 213. At the same time, the memory 1203 in the computing device 1200B stores instructions for executing the functions of the processing module 212.

[0180] The connection method between the computing device clusters shown in Figure 14 can be considered to be that the alarm root cause recommendation method provided in this application needs to analyze a large number of call chains to be tested and a large number of normal call chains to determine multiple first candidate root causes, so it is considered to hand over the functions implemented by the processing module 212 to the computing device 1200B for execution.

[0181] It should be understood that the functions of the computing device 1200A shown in Figure 14 may also be completed by multiple computing devices. Similarly, the functions of the computing device 1200B may also be completed by multiple computing devices.

[0182] Embodiments of the present application also provide a computer program product containing instructions. This computer program product can be software or a program product containing instructions that can be executed on a computing device or stored on any available medium. When executed on at least one computing device, the computer program product causes the at least one computing device to execute a method for recommending a root cause of an alarm.

[0183] Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions for instructing the computing device to execute the alarm root cause recommendation method.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for recommending root causes of alarms, characterized in that: Applied to a microservice system including a plurality of microservices, the microservice system further including a root cause recommendation device, the method comprising: The root cause recommendation device obtains a plurality of normal call chains and a plurality of call chains to be tested, wherein the plurality of normal call chains are call chains of a plurality of normal microservice interfaces in the microservice system, and the plurality of call chains to be tested are call chains of abnormal microservice interfaces in the microservice system; The root cause recommendation device determines a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains; The root cause recommendation device collects statistics on the information of the plurality of call chains to be tested to obtain multi-dimensional statistical information corresponding to the plurality of first candidate root causes; The root cause recommendation device provides a plurality of second candidate root causes to the user, or provides the plurality of second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the plurality of second candidate root causes are obtained by sorting the plurality of first candidate root causes based on multi-dimensional statistical information corresponding to the plurality of first candidate root causes.

2. The method according to claim 1, characterized in that The multi-dimensional statistical information corresponding to each first candidate root cause among the multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

3. The method according to claim 2, characterized in that The first call event is a call event in which each of the first candidate root causes exists, and the first dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the frequency, average duration, and end time of the first call event in the call chain to be tested to which the first call event belongs; The second dimension statistical information corresponding to each first candidate root cause includes at least one of the following: the resource types called in the multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource types called by the first call event and the nodes where the resources are located.

4. The method according to any one of claims 1 to 3, characterized in that: The root cause recommendation device determines a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains, including: The root cause recommendation device compares the root microservice interface of each of the multiple call chains to be tested with the root microservice interfaces of the multiple normal call chains, compares the call relationship in each call chain to be tested with the call relationship in the multiple normal call chains, and compares the log of each call chain to be tested with the log of the multiple normal call chains to determine multiple first candidate root causes.

5. The method according to any one of claims 1 to 4, characterized in that: The root cause recommendation device obtains multiple call chains to be tested, including: The root cause recommendation device counts the number of abnormal microservice interfaces of the microservice system in multiple time periods from when the abnormality occurs to the current moment; The root cause recommendation device selects a target time period from the multiple time periods based on the number of abnormal microservice interfaces of the microservice system in the multiple time periods, and the change rate between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; The root cause recommendation device obtains the call chains of the abnormal microservice interfaces of the microservice system from the occurrence of the abnormality to the target time period as the multiple call chains to be tested.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The root cause recommendation device obtains a target candidate root cause from among the plurality of second candidate root causes selected by the user; The root cause recommendation device optimizes a plurality of weights based on the target candidate root causes, wherein the plurality of weights are weights corresponding to the multi-dimensional statistical information when the plurality of first candidate root causes are sorted.

7. An alarm root cause recommendation device, characterized in that: Applied to a microservice system including multiple microservices, the device includes: An acquisition module, used to acquire multiple normal call chains and multiple call chains to be tested, wherein the multiple normal call chains are call chains of multiple normal microservice interfaces in the microservice system, and the multiple call chains to be tested are call chains of abnormal microservice interfaces in the microservice system; A processing module, configured to determine a plurality of first candidate root causes based on the plurality of call chains to be tested and the plurality of normal call chains; The processing module is further used to collect statistics on the information of the plurality of call chains to be tested, and obtain a plurality of first candidate root causes corresponding to the plurality of first candidate root causes. Dimension statistics; A recommendation module is used to provide a plurality of second candidate root causes to a user, or to provide the plurality of second candidate root causes and repair suggestions for each second candidate root cause to the user, wherein the plurality of second candidate root causes are obtained by sorting the plurality of first candidate root causes based on multi-dimensional statistical information corresponding to the plurality of first candidate root causes.

8. The device according to claim 7, characterized in that The multi-dimensional statistical information corresponding to each first candidate root cause among the multiple first candidate root causes includes first dimension statistical information and second dimension statistical information, wherein the first dimension is the calling event dimension in which each first candidate root cause exists, and the second dimension is the resource dimension called by the calling event in which each first candidate root cause exists.

9. The device according to claim 8, characterized in that The first call event is a call event in which each of the first candidate root causes exists, and the first dimension statistical information corresponding to each of the first candidate root causes includes at least one of the following: the frequency, average duration, and end time of the first call event in the call chain to be tested to which the first call event belongs; The second dimension statistical information corresponding to each first candidate root cause includes at least one of the following: the resource types called in the multiple call chains to be tested and the nodes where the resources are located, and the frequency, average duration and earliest start time of call events that are the same as the resource types called by the first call event and the nodes where the resources are located.

10. The device according to any one of claims 7 to 9, characterized in that The processing module is specifically used for: The root microservice interface of each of the multiple call chains to be tested is compared with the root microservice interface of the multiple normal call chains, the call relationship in each call chain to be tested is compared with the call relationship in the multiple normal call chains, and the log of each call chain to be tested is compared with the log of the multiple normal call chains to determine multiple first candidate root causes.

11. The device according to any one of claims 7 to 10, characterized in that The processing module is also used to count the number of abnormal microservice interfaces of the microservice system in multiple time periods from the occurrence of the abnormality to the current moment; The processing module is further configured to select a target time period from the multiple time periods based on the number of abnormal microservice interfaces of the microservice system in the multiple time periods, and the change rate between the number of abnormal microservice interfaces of the microservice system in the target time period and the number of abnormal microservice interfaces of the microservice system in the time period before the target time period exceeds a threshold; The acquisition module is used to acquire the call chains of the abnormal microservice interfaces of the microservice system from the occurrence of the abnormality to the target time period as the multiple call chains to be tested.

12. The device according to any one of claims 7 to 11, characterized in that The acquisition module is further used to acquire a target candidate root cause from among the plurality of second candidate root causes selected by the user; The processing module is further configured to optimize a plurality of weights based on the target candidate root causes, wherein the plurality of weights are weights corresponding to the multi-dimensional statistical information when the plurality of first candidate root causes are sorted.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device of the at least one computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.

Images