Fault root cause positioning method and device

Abstract

The invention provides a fault root cause positioning method, the method is applied to a distributed system, and the method comprises the steps: responding to a fault alarm signal, and obtaining a target module and module data of the target module; obtaining a failure interface request and operation and maintenance change data of the target module based on the module data, and determining a calling link and link data of the calling link corresponding to the failure interface request; inputting the fault alarm signal, the calling link, the link data and the operation and maintenance change information to a preset target model group for multi-model cross validation to generate a predicted fault root cause; and analyzing the predicted fault root cause according to a preset quantitative evaluation rule to determine a target fault root cause.

Description

Technical Field

[0001] This application relates to the field of computer system fault diagnosis technology, and in particular to a method and device for locating the root cause of a fault. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this application as set forth in the claims. The description herein is not to be construed as prior art simply because it is included in this section.

[0003] In traditional microservice operations and maintenance (O&M) models, troubleshooting interface failures is typically complex and relies heavily on manual intervention. When an interface in a microservice system experiences a drop in success rate or a timeout, triggering an alarm, O&M or R&D personnel must first manually identify the specific microservice module experiencing the failure, its business line, and the approximate timeframe of the failure. Subsequently, based on this information, technical staff need to log into the distributed tracing system, locate failed request records within the corresponding time period, and analyze each record's complete call chain. Furthermore, they must collect log information and monitoring data from multiple related systems, including but not limited to system logs, POD-level monitoring metrics, storage system performance data, and host machine resource usage. Finally, through manual analysis of this scattered data, they attempt to pinpoint the root cause of the failure.

[0004] However, this manual troubleshooting method has many limitations. First, because it involves data collection and cross-comparison across multiple systems, the entire process is inefficient, especially in cases of complex system architecture and long call chains. Maintenance personnel often need to frequently switch between different tools and platforms, consuming a significant amount of time for data searching and processing. Second, fault analysis is highly dependent on the experience and technical skills of the operators; subjective judgment may lead to misjudgments or omissions, thus affecting the accuracy and timeliness of fault repair. Furthermore, as the system scale continues to expand and business logic becomes increasingly complex, the frequency and types of interface faults are also on the rise, making traditional manual maintenance methods insufficient to meet the current system's demands for rapid fault response and accurate fault location.

[0005] In summary, existing microservice operation and maintenance mechanisms have significant shortcomings in terms of automation, data analysis efficiency, and accuracy of fault root cause identification. There is an urgent need for a more intelligent and systematic fault root cause localization method to improve operation and maintenance efficiency and ensure the stable operation of the system.

[0006] Therefore, this application urgently needs to propose a fault root cause localization method and device that can solve the above-mentioned technical problems. Summary of the Invention

[0007] This application provides a method and apparatus for locating the root cause of a fault, which can accurately and efficiently locate the root cause of the fault.

[0008] One aspect of this application provides a method for locating the root cause of a fault, the method being applied to a distributed system, the method comprising: In response to a fault alarm signal, acquire the target module and its module data; Based on the module data, obtain the failed interface requests and operation and maintenance change data of the target module, and determine the call chain and the link data of the call chain corresponding to the failed interface requests; Input the fault alarm signal, the call link, the link data, and the operation and maintenance change information into a preset target model group to perform multi-model cross-validation and generate a predicted fault root cause; The predicted root cause of the failure is determined by analyzing the predicted root cause based on the preset quantitative evaluation rules.

[0009] Furthermore, the step of acquiring the target module and its module data in response to a fault alarm signal includes: in response to a fault alarm signal, determining the target module corresponding to the fault alarm signal; determining the fault occurrence time, fault level, and target service line based on the target module; and determining the module data based on the fault occurrence time, the fault level, and the target service line.

[0010] Further, the step of obtaining the failed interface requests and operation and maintenance change data of the target module based on the module data includes: determining the time period of the failure based on the time of the failure; obtaining the operation and maintenance change data of the distributed system based on the time period of the failure; obtaining the failed interface requests of the target business line of the target module within the time period of the failure; wherein, determining the time period of the failure based on the time of the failure includes: determining a first time period and a second time period based on the time of the failure, wherein the first time period is the time interval immediately before the time of the failure, and the second time period is the time interval immediately after the time of the failure; and determining the time period of the failure based on the first time period and the second time period.

[0011] Further, determining the call chain and the link data of the call chain corresponding to the failed interface request includes: obtaining the call chain of the failed request interface from the distributed system; determining the transfer node of the call chain, wherein the transfer node includes at least one of system, POD, service and host machine; obtaining the node data of the transfer node corresponding to the transfer node, and generating the link data based on the node data.

[0012] Further, the step of inputting the fault alarm signal, the call link, the link data, and the operation and maintenance change information into a preset target model group for multi-model cross-validation to generate a predicted fault root cause includes: inputting the fault alarm signal, the call link, the link data, and the operation and maintenance change information into the target model of the preset target model group respectively, and obtaining the initial fault root cause and prediction logic of the target model; performing cross-validation of the initial fault root cause and the prediction logic into other models in the preset target model group other than the target model, and obtaining the cross-validation result; when the cross-validation result is passed, the initial fault root cause is determined to be the predicted fault root cause.

[0013] Further, the step of analyzing the predicted root causes of failures and determining the target root cause based on preset quantitative evaluation rules includes: dividing the predicted root causes of failures based on the failed interface requests, clustering the predicted root causes of failures, determining the number of predicted root causes of failures in each category as a first value, and determining the total number of predicted root causes of failures as a second value; determining the proportion value based on the first value and the second value; and determining the predicted root cause of failures corresponding to the largest proportion value as the target root cause of failure.

[0014] Further, obtaining node data for the corresponding transfer node and generating the link data based on the node data includes: obtaining log records for the system; obtaining POD logs for the POD, the POD logs including at least one of CPU utilization, memory usage, and network traffic; obtaining storage logs for the service, the storage logs including at least one of disk I / O and storage space usage status; obtaining host logs for the host machine, the host logs including at least one of resource utilization and system load; and generating the link data based on at least one of the log records, the POD logs, the storage logs, and the host logs.

[0015] In another aspect, this application provides a fault root cause localization device, the device being applied to a distributed system, the device comprising: A response unit is used to acquire the target module and the module data of the target module in response to a fault alarm signal; The acquisition unit is used to acquire the failed interface requests and operation and maintenance change data of the target module based on the module data, and to determine the call chain and the link data of the call chain corresponding to the failed interface requests; The generation unit is used to input the fault alarm signal, the call link, the link data and the operation and maintenance change information into a preset target model group to perform multi-model cross-validation to generate a predicted fault root cause. The determination unit is used to analyze the predicted root causes of failures according to preset quantitative evaluation rules to determine the target root causes of failures.

[0016] In another aspect, this application also provides an electronic device, the electronic device including at least one processor; and

[0017] A memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0018] In another aspect, this application provides a computer-readable medium having computer program instructions stored thereon, characterized in that the computer program instructions can be executed by a processor to implement the method described above.

[0019] The fault root cause localization method proposed in this application includes: in response to a fault alarm signal, acquiring a target module and its module data; based on the module data, acquiring the failed interface requests and maintenance change data of the target module, and determining the call chain and its link data corresponding to the failed interface requests; inputting the fault alarm signal, the call chain, the link data, and the maintenance change information into a preset target model group for multi-model cross-validation to generate a predicted fault root cause; and analyzing the predicted fault root cause according to preset quantitative evaluation rules to determine the target fault root cause. The fault root cause localization method proposed in this application significantly improves the intelligent operation and maintenance capabilities of the system, helping to achieve rapid fault recovery and stable system operation, specifically including the following beneficial effects: First, it improves the automation level and efficiency of fault location. Compared with the traditional method of relying on manual log collection, link analysis and cause judgment, this invention realizes rapid collection and structured processing of fault-related information by systematically responding to alarm signals and automatically acquiring relevant module data. This effectively reduces the manual intervention links and significantly improves the speed and efficiency of fault location.

[0020] Secondly, it enhances the comprehensiveness and accuracy of fault analysis. This invention not only conducts in-depth analysis of the call relationships between services based on call chain data, but also introduces operation and maintenance change data as an auxiliary basis for judgment, making the dimensions of fault analysis richer. At the same time, it uses a preset target model group to cross-validate multiple potential fault causes, avoiding the misjudgment problems that may be caused by a single model, thereby improving the accuracy of root cause identification.

[0021] Then, a scientific quantitative assessment and optimization of the root cause of the failure was achieved. Based on the output of multiple models, this invention further introduces preset quantitative assessment rules to perform weighted analysis and ranking optimization of each prediction result, thereby determining the final root cause of the failure more objectively and scientifically. This mechanism effectively reduces subjective errors caused by differences in human experience and improves the credibility and consistency of diagnostic results.

[0022] Finally, the root cause localization method proposed in this application has good applicability and scalability; the method of this invention has a clear process and rigorous logic, and is applicable to fault diagnosis scenarios under various complex microservice architectures. Through the design of the model group and the flexible configuration of the evaluation rules, it can adapt to the root cause analysis needs of different business types and technical environments, and has strong versatility and scalability. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating a fault root cause localization method provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of a fault root cause localization device provided in yet another embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing the solutions in the embodiments of this application; The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0027] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0028] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0029] This application provides a method for locating the root cause of a fault, which is applied to a distributed system and includes at least the following processing steps: Step S101: In response to the fault alarm signal, acquire the target module and the module data of the target module; Step S102: Based on the module data, obtain the failed interface request and operation and maintenance change data of the target module, and determine the call chain and the link data of the call chain corresponding to the failed interface request; Step S103: Input the fault alarm signal, the call link, the link data, and the operation and maintenance change information into the preset target model group to perform multi-model cross-validation to generate a predicted fault root cause; Step S104: Analyze the predicted root causes of failures according to the preset quantitative evaluation rules to determine the target root causes of failures.

[0030] In practical scenarios, the execution entity of this method can be a user device, or a device formed by integrating a user device and a network device through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0031] Example 1

[0032] Figure 1 This application illustrates a fault root cause localization method provided by an embodiment of the present application. The method is applied to a distributed system and includes at least the following processing steps: Step S101: In response to the fault alarm signal, acquire the target module and the module data of the target module; Step S102: Based on the module data, obtain the failed interface request and operation and maintenance change data of the target module, and determine the call chain and the link data of the call chain corresponding to the failed interface request; Step S103: Input the fault alarm signal, the call link, the link data, and the operation and maintenance change information into the preset target model group to perform multi-model cross-validation to generate a predicted fault root cause; Step S104: Analyze the predicted root causes of failures according to the preset quantitative evaluation rules to determine the target root causes of failures.

[0033] The fault root cause localization method proposed in this application integrates key technologies such as automated data acquisition, multi-dimensional data analysis, multi-model collaborative reasoning, and quantitative evaluation, which solves the problems of excessive manual intervention, low analysis efficiency, and poor judgment accuracy in existing technologies.

[0034] In one embodiment, the step of acquiring the target module and its module data in response to a fault alarm signal includes: determining the target module corresponding to the fault alarm signal in response to the fault alarm signal; determining the fault occurrence time, fault level, and target service line based on the target module; and determining the module data based on the fault occurrence time, the fault level, and the target service line.

[0035] Specifically, when an alarm is received indicating a decrease in the success rate of the service interface or a timeout in the response, the distributed system will automatically receive a fault alarm signal. Upon receiving the fault alarm signal, the distributed system will determine the target business line to which the target module corresponding to the fault alarm signal belongs, the time of the fault occurrence (e.g., 11:30:21 on June 21, 2025), and the fault level.

[0036] Specifically, the fault level may include a service interface success rate not exceeding a preset value, or a service interface success rate of level X. It should be understood that this application does not limit the magnitude of the preset value; those skilled in the art can rationally select and limit the preset value according to the actual application scenario. It should also be understood that in actual application scenarios, the target module may correspond to multiple business lines or only one business line. When the target module corresponds to only one business line, that business line can be directly identified as the target business line. When the target module corresponds to multiple business lines, it is necessary to determine which business line corresponding to the target module issued the fault alarm based on the fault alarm signal and other necessary information.

[0037] In one embodiment, the method may further include automatically triggering a fault alarm signal when the fault level reaches a preset alarm threshold.

[0038] In one implementation, obtaining the failed interface requests and operation and maintenance change data of the target module based on the module data includes: determining the time period of the failure based on the time of the failure; obtaining the operation and maintenance change data of the distributed system based on the time period of the failure; obtaining the failed interface requests of the target business line of the target module within the time period of the failure; wherein, determining the time period of the failure based on the time of the failure includes: determining a first time period and a second time period based on the time of the failure, wherein the first time period is the time interval immediately before the time of the failure, and the second time period is the time interval immediately after the time of the failure; and determining the time period of the failure based on the first time period and the second time period.

[0039] In one implementation, when the fault occurs at 11:31:05 on June 21, 2025, the fault occurrence time period can be from 11:31:00 to 11:32:00 on June 21, 2025. That is, the lengths of the first and second time periods in this application can be different; similarly, the lengths of the first and second time periods can also be the same. Those skilled in the art can make reasonable choices based on actual application scenarios. However, it should be noted that if the lengths of the first and second time periods are too long, it may consume a large amount of resources and time during the process of obtaining the failed request interface.

[0040] In one implementation, determining the call chain and the call chain data corresponding to the failed interface request includes: obtaining the call chain of the failed request interface from the distributed system; determining the transfer node of the call chain, wherein the transfer node includes at least one of system, POD, service and host; obtaining the node data of the transfer node corresponding to the transfer node, and generating the link data based on the node data.

[0041] In one implementation, obtaining the call chain of the failed request interface from the distributed system for that failed request interface includes: obtaining an interface identifier for the failed request interface; and obtaining the call chain of the failed request interface based on the interface identifier. Specifically, assuming there are 10 failed request interfaces, the interface identifiers of these 10 failed request interfaces are obtained, and then the corresponding call chain is obtained based on each interface identifier; wherein the interface identifier is unique, and the interface identifier and its corresponding call chain can also be unique.

[0042] Furthermore, the flow nodes include systems, PODs, services, and hosts. There are relationships between systems, services, PODs, and hosts, with the system encompassing all the aforementioned components. It utilizes concepts such as services, PODs, and hosts to build complex distributed applications. The system's goal is to ensure the efficient operation of each service through effective resource management and scheduling. POD and Service: A service provides a unified entry point for a group of functionally similar PODs. Clients do not need to know which specific POD is handling a request; they only need to send a request to the service. The service automatically distributes the request to the appropriate POD. POD and Host: PODs are assigned to specific hosts to run on. The host provides the computing resources required to run the POD. Service and Host: Although a service is not directly associated with a specific host, it indirectly depends on the resources and environment provided by the host because the POD it points to actually runs on some host.

[0043] In one implementation, the step of inputting the fault alarm signal, the call link, the link data, and the operation and maintenance change information into a preset target model group for multi-model cross-validation to generate a predicted fault root cause includes: inputting the fault alarm signal, the call link, the link data, and the operation and maintenance change information into a target model of the preset target model group respectively, and obtaining the initial fault root cause and prediction logic of the target model; performing cross-validation of the initial fault root cause and the prediction logic into other models in the preset target model group other than the target model, and obtaining the cross-validation result; when the cross-validation result is passed, the initial fault root cause is determined to be a predicted fault root cause.

[0044] Specifically, the target model group includes a first model and a second model. The first and second models, as target models, receive fault alarm signals, call links, link data, and operation and maintenance change information, respectively. The first and second models output initial fault root causes. The fault alarm signals, call links, link data, and operation and maintenance change information are summarized to generate summary data. The summary data is then input into the first and second models of the target model group, and the first and second models output initial fault root causes, respectively. After the first model outputs the initial fault root cause, the initial fault root cause of the first model, its prediction logic, and the summary data are output to the second model. The second model cross-validates the fault root cause prediction results of the first model.

[0045] In one implementation, the target model group includes the QWEN model and the DS model.

[0046] In one embodiment, the step of analyzing the predicted root causes of failures and determining the target root cause based on preset quantitative evaluation rules includes: dividing the predicted root causes of failures based on the failed interface requests, clustering the predicted root causes of failures, determining the number of predicted root causes of failures in each category as a first value, and determining the total number of predicted root causes of failures as a second value; determining a percentage value based on the first value and the second value; and determining the predicted root cause of failures corresponding to the largest percentage value as the target root cause of failure.

[0047] Specifically, assuming the predicted root causes of a fault include a first predicted root cause, a second predicted root cause, and a third predicted root cause, and setting the number of first predicted root causes to be 2, the number of second predicted root causes to be 3, and the number of third predicted root causes to be 5, that is, the total number of predicted root causes is 10; based on the total number of predicted root causes, the number of second predicted root causes, and the number of third predicted root causes, the proportion of the first predicted root cause is determined to be 2 / 10 = 0.2, the proportion of the second predicted root cause is determined to be 3 / 10 = 0.3, and the proportion of the third predicted root cause is determined to be 5 / 10 = 0.5; based on the proportions of the first, second, and third predicted root causes, the third predicted root cause is determined to be the target root cause.

[0048] In one implementation, obtaining node data for the corresponding transfer node and generating link data based on the node data includes: obtaining log records for the system; obtaining POD logs for the POD, the POD logs including at least one of CPU utilization, memory usage, and network traffic; obtaining storage logs for the service, the storage logs including at least one of disk I / O and storage space usage status; obtaining host logs for the host machine, the host logs including at least one of resource utilization and system load; and generating the link data based on at least one of the log records, the POD logs, the storage logs, and the host logs.

[0049] It's important to understand that a Pod (Portable Operating Domain) is the smallest deployable unit in Kubernetes. It can be understood as a group of one or more containers sharing resources. These containers are always scheduled together and run on the same host (Node). They share network namespaces (i.e., shared IP addresses and port spaces), storage volumes, and other resources. Therefore, from a semantic perspective, a Pod can be understood as a "container group" or, more figuratively, "a group of containers working together." Kubernetes is an open-source platform designed for automating the deployment, scaling, and management of containerized applications, supporting and facilitating the construction and operation of modern distributed systems.

[0050] The process of obtaining storage logs for the corresponding service includes: obtaining the service name for the corresponding service, and retrieving the monitoring data of the storage system associated with the service from the monitoring system of the distributed system, so as to analyze whether there is information related to the root cause of the failure at the storage layer of the distributed system.

[0051] The process of obtaining log records for the system includes: obtaining the global identifier of the call chain; obtaining the log records of the flow nodes included in the system associated with the global identifier, in order to find abnormal information related to the root cause of the fault.

[0052] The process of obtaining host machine logs for the host machine includes: determining the POD ID of the POD, obtaining the host machine and its host machine identifier based on the POD ID; retrieving the host machine logs from the monitoring system based on the host machine identifier, and checking whether the host machine failure is related to the root cause of the failure.

[0053] The process of obtaining POD logs for a given POD includes: obtaining the POD ID for the given POD, and retrieving relevant monitoring data for that POD from the monitoring system based on the POD ID to determine whether the POD has performance issues.

[0054] The fault root cause localization method proposed in this embodiment first automatically collects and integrates data information from multiple dimensions, then predicts and determines the fault root cause based on the target model group, and finally achieves accurate localization of the fault root cause.

[0055] Specifically, upon receiving an alarm about a microservice interface anomaly, the system automatically collects various key data related to the fault, including call chain information, system logs, POD monitoring metrics, storage system operating status, host machine resource usage, and recent operation and maintenance change records. By uniformly cleaning, correlating, and structurally integrating the above multi-source heterogeneous data, a complete and accurate fault analysis data foundation is built, providing comprehensive support for subsequent fault root cause identification.

[0056] Furthermore, this application utilizes multiple large-scale models with extensive knowledge reserves and powerful reasoning capabilities to conduct in-depth analysis of the integrated fault data. Each model mines potential fault causes from different perspectives, such as performance bottlenecks, configuration errors, and abnormal dependent services, thereby achieving intelligent identification and judgment of complex fault scenarios.

[0057] Furthermore, to improve the accuracy and reliability of root cause identification, the system cross-validates the analysis results from multiple large models, eliminating biased or inconsistent conclusions. Based on this, the system further statistically summarizes the analysis results of all failed requests, calculates the frequency of occurrence of various fault causes and their proportion in the overall picture, and ultimately determines the most likely core cause of the fault, significantly improving the accuracy of fault location and decision-making efficiency.

[0058] Example 2

[0059] Figure 2 This application illustrates a fault root cause localization device provided in an embodiment of the present application. The device is applied to a distributed system and includes at least the following units: A response unit is used to acquire the target module and the module data of the target module in response to a fault alarm signal; The acquisition unit is used to acquire the failed interface requests and operation and maintenance change data of the target module based on the module data, and to determine the call chain and the link data of the call chain corresponding to the failed interface requests; The generation unit is used to input the fault alarm signal, the call link, the link data and the operation and maintenance change information into a preset target model group to perform multi-model cross-validation to generate a predicted fault root cause. The determination unit is used to analyze the predicted root causes of failures according to preset quantitative evaluation rules to determine the target root causes of failures.

[0060] Example 3

[0061] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the fault root cause localization method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0062] The electronic device can be a user device, or a device formed by integrating user devices and network devices through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0063] Figure 3 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The device 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on a program stored in a Read-Only Memory (ROM) 502 or a program loaded from a storage portion 508 into a Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0064] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 508 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet.

[0065] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, it performs the functions defined in the methods of this application.

[0066] Example 4

[0067] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0068] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0069] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, device, or apparatus.

[0070] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0071] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0072] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0073] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces, or indirect couplings or communication connections between devices or units, and may be electrical, mechanical, or other forms.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0077] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0079] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for locating the root cause of a fault, characterized in that, The method is applied to a distributed system, and the method includes: In response to a fault alarm signal, acquire the target module and its module data; Based on the module data, obtain the failed interface requests and operation and maintenance change data of the target module, and determine the call chain and the link data of the call chain corresponding to the failed interface requests; Input the fault alarm signal, the call link, the link data, and the operation and maintenance change information into a preset target model group to perform multi-model cross-validation and generate a predicted fault root cause; The predicted root cause of the failure is determined by analyzing the predicted root cause based on the preset quantitative evaluation rules.

2. The fault root cause localization method according to claim 1, characterized in that, The step of acquiring the target module and its module data in response to a fault alarm signal includes: In response to a fault alarm signal, the target module corresponding to the fault alarm signal is determined; Based on the target module, the time of fault occurrence, fault level, and target service line are determined; The module data is determined based on the time of the fault occurrence, the fault level, and the target service line.

3. The fault root cause localization method according to claim 2, characterized in that, The step of obtaining the failed interface requests and operation and maintenance change data of the target module based on the module data includes: The time period of the fault occurrence is determined based on the time of the fault occurrence. Obtain the operation and maintenance change data of the distributed system based on the time period in which the fault occurred; Obtain the failure interface request of the target business line of the target module within the time of the failure occurrence; The step of determining the fault occurrence time period based on the fault occurrence time includes: A first time period and a second time period are determined based on the time of the fault occurrence. The first time period is the time interval immediately before the time of the fault occurrence, and the second time period is the time interval immediately after the time of the fault occurrence. The time period in which the fault occurred is determined based on the first time period and the second time period.

4. The fault root cause localization method according to claim 3, characterized in that, The determination of the call chain and the chain data corresponding to the failed interface request includes: The call chain of the failed request interface is obtained from the distributed system corresponding to the failed request interface; Determine the flow nodes of the call chain, wherein the flow nodes include at least one of the system, POD, service, and host machine; The node data of the corresponding transfer node is obtained, and the link data is generated based on the node data.

5. The fault root cause localization method according to claim 4, characterized in that, The process of inputting the fault alarm signal, the call link, the link data, and the operation and maintenance change information into a preset target model group for multi-model cross-validation to generate a predicted fault root cause includes: The fault alarm signal, the call link, the link data, and the operation and maintenance change information are respectively input into the target model of the preset target model group to obtain the initial fault root cause and prediction logic of the target model; The initial root cause of the fault and the prediction logic are cross-validated with other models in the preset target model group other than the target model to obtain the cross-validation results. If the cross-validation result is passed, then the initial root cause of the fault is determined to be the predicted root cause of the fault.

6. The fault root cause localization method according to claim 5, characterized in that, The step of analyzing the predicted root causes of failures and determining the target root causes according to preset quantitative evaluation rules includes: After dividing the predicted root causes of failure based on the failed interface requests, the predicted root causes of failure are clustered, and the number of predicted root causes in each category is determined as a first value, and the total number of predicted root causes is determined as a second value. The percentage value is determined based on the first value and the second value; Identify the predicted root cause of the fault corresponding to the largest percentage value as the target root cause of the fault.

7. The fault root cause localization method according to claim 6, characterized in that, Obtaining node data for the corresponding transfer node, and generating link data based on the node data, including: Log records are obtained corresponding to the system described above; The POD logs are obtained for the POD, and the POD logs include at least one of CPU utilization, memory usage, and network traffic. The storage logs are obtained in accordance with the service, and the storage logs include at least one of disk I / O and storage space usage status; The host machine logs are obtained corresponding to the host machine, and the host machine logs include at least one of resource utilization and system load; The link data is generated based on at least one of the log records, the POD logs, the storage logs, and the host logs.

8. A fault root cause location device, characterized in that, The device is used in a distributed system, and the device includes: A response unit is used to acquire the target module and the module data of the target module in response to a fault alarm signal; The acquisition unit is used to acquire the failed interface requests and operation and maintenance change data of the target module based on the module data, and to determine the call chain and the link data of the call chain corresponding to the failed interface requests; The generation unit is used to input the fault alarm signal, the call link, the link data and the operation and maintenance change information into a preset target model group to perform multi-model cross-validation to generate a predicted fault root cause. The determination unit is used to analyze the predicted root causes of failures according to preset quantitative evaluation rules to determine the target root causes of failures.

9. An electronic device, the electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; characterized in that, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A computer-readable medium having computer program instructions stored thereon, characterized in that, The computer program instructions can be executed by a processor to implement the method as described in any one of claims 1 to 7.

Images