A method, device, equipment and storage medium for locating business faults

Abstract

The present invention discloses a method, device, equipment and storage medium for locating a business fault, including: after detecting that a target business channel has a fault, triggering a fault locating task matching the target business channel; obtaining call chain data corresponding to the target business channel, establishing matching tags for the call chain data to obtain target data to be analyzed; determining the earliest business node with the earliest fault occurrence time according to the fault occurrence time corresponding to the target data, aggregating the information of the business node, and determining the service or application instance with the largest abnormal proportion according to the aggregation result; determining the fault root cause analysis result corresponding to the target business channel according to the abnormal proportion of the service or application instance. The technical solution of the embodiment of the present invention can improve the efficiency of locating business faults, improve the accuracy of locating results, and reduce resource consumption caused by the fault locating process.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a service fault locating method, apparatus, device and storage medium. Background Art

[0002] With the development of microservice architecture, the scalability and flexibility of enterprise business have been further improved. In order to quickly locate business failures, enterprises generally build root cause location or fault demarcation tools to assist in problem location.

[0003] In the existing technology, the following methods are usually used to locate business faults: the first is to aggregate traditional basic alarms, sort them in chronological order after the fault occurs, and locate the earliest and lowest-level alarm as the root cause of the problem; the second is to build a basic call topology based on the call chain, find the deepest fault point according to the topology, and thus locate the root cause of the fault.

[0004] However, the first method has inaccurate positioning and is easily interfered by false alarms or irrelevant resource alarms. In the second method, there are too many fault trigger points, resulting in too many root cause results and too much interference. The fault demarcation task is frequently triggered in a continuous time period, which requires a lot of computing resources. The large amount of data analyzed in a single time causes the analysis to take too long, and the analysis results are not aggregated. The end node of the analysis process is not rigorous enough, resulting in inaccurate analysis results. Summary of the Invention

[0005] The present invention provides a service fault locating method, apparatus, device and storage medium, which can improve the efficiency of locating service faults, improve the accuracy of locating results and reduce resource consumption caused by the fault locating process.

[0006] According to one aspect of the present invention, a service fault locating method is provided, the method comprising:

[0007] After detecting a fault in a target service channel, triggering a fault location task matching the target service channel; the target service channel includes multiple system services;

[0008] Acquire call chain data corresponding to the target business channel, and establish matching tags for the call chain data to obtain target data to be analyzed corresponding to the target business channel;

[0009] Determine the service node with the earliest fault occurrence time based on the fault occurrence time corresponding to the target data, aggregate the information of the service node, and determine the service or application instance with the largest abnormality proportion based on the aggregation result;

[0010] Determine the root cause analysis result of the fault corresponding to the target business channel based on the abnormal proportion of the service or application instance.

[0011] Optionally, after a fault is detected on a target service channel, a fault location task matching the target service channel is triggered, including:

[0012] After detecting that a fault occurs in a target business channel, obtain the fault type corresponding to the target business channel;

[0013] According to the fault type, a fault location task matching the target service channel is triggered.

[0014] Optionally, after a fault is detected on a target service channel, a fault location task matching the target service channel is triggered, including:

[0015] Collect the number of business timeouts and business exceptions in the target business channel within the preset time period;

[0016] Determining whether the number of service timeouts is greater than a preset first threshold;

[0017] If so, determining whether the target service channel triggered a service timeout analysis process in the previous collection cycle;

[0018] If not, trigger the timeout analysis service that matches the target service channel.

[0019] Optionally, after determining whether the number of service timeouts is greater than a preset first threshold, the method further includes:

[0020] If not, determining whether the number of business anomalies is greater than a preset second threshold;

[0021] If so, determine whether the target business channel triggered a business anomaly analysis process in the previous collection cycle;

[0022] If not, trigger the abnormal analysis business that matches the target business channel.

[0023] Optionally, determining a fault root cause analysis result corresponding to the target business channel based on the abnormality ratio of the service or application instance includes:

[0024] According to the abnormality ratio of the service or application instance, determining whether there is a target service or target application instance with an abnormality ratio greater than a preset threshold;

[0025] If yes, then analyzing the target service or target application instance to obtain a fault root cause analysis result corresponding to the target business channel;

[0026] If not, a multi-threaded analysis is performed on the service or application instance to obtain a fault root cause analysis result corresponding to the target business channel.

[0027] Optionally, analyzing the target service or target application instance to obtain a fault root cause analysis result corresponding to the target business channel includes:

[0028] Obtaining the current node corresponding to the target service or target application instance, as well as the associated resources and resource alarm information of the current node;

[0029] Determine whether the call type of the current node is a local call;

[0030] If not, obtain the downstream nodes of the current node, as well as the associated resources and resource alarm information of the downstream nodes;

[0031] If the downstream node has resource alarm information, the associated resources of the downstream node are used as the root cause analysis result of the fault; if the downstream node resources are normal, the associated resources of the current node are used as the root cause analysis result of the fault.

[0032] Optionally, after determining the root cause analysis result corresponding to the target service channel, the method further includes:

[0033] The root cause analysis results of different business channels associated with the same fault are recorded in the same analysis record to conduct a combined analysis of the business channels.

[0034] According to another aspect of the present invention, a service fault locating device is provided, the device comprising:

[0035] A task triggering module is used to trigger a fault location task matching the target service channel after detecting a fault in the target service channel; the target service channel includes multiple system services;

[0036] A data acquisition module is used to acquire call chain data corresponding to the target business channel, and to establish matching tags for the call chain data to obtain target data to be analyzed corresponding to the target business channel;

[0037] An information aggregation module is used to determine the service node with the earliest fault occurrence time based on the fault occurrence time corresponding to the target data, aggregate the information of the service node, and determine the service or application instance with the largest abnormality proportion based on the aggregation result;

[0038] The root cause analysis module is used to determine the root cause analysis result of the fault corresponding to the target business channel based on the abnormal proportion of the service or application instance.

[0039] According to another aspect of the present invention, an electronic device is provided, comprising:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the service fault locating method according to any embodiment of the present invention.

[0043] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the service fault locating method according to any embodiment of the present invention when executed.

[0044] The technical solution provided by the embodiment of the present invention, after detecting that a fault has occurred in a target business channel, triggers a fault location task matching the target business channel, obtains call chain data corresponding to the target business channel, and establishes a matching label for the call chain data, obtains the target data to be analyzed corresponding to the target business channel, determines the business node with the earliest fault occurrence time according to the fault occurrence time corresponding to the target data, aggregates the information of the business node, determines the service or application instance with the largest abnormality ratio according to the aggregation result, and determines the fault root cause analysis result corresponding to the target business channel according to the abnormality ratio of the service or application instance. The technical means can improve the efficiency of locating business faults, improve the accuracy of locating results, and reduce resource consumption caused by the fault location process.

[0045] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1a This is a flow chart of a service fault locating method provided by an embodiment of the present invention;

[0048] Figure 1b This is a flowchart of a method for triggering a fault location task according to an embodiment of the present invention;

[0049] Figure 2is a flowchart of another service fault locating method provided by an embodiment of the present invention;

[0050] Figure 3 2 is a schematic structural diagram of a service fault locating device provided by an embodiment of the present invention;

[0051] Figure 4 It is a structural diagram of an electronic device for implementing the service fault locating method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0053] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] Figure 1a This is a flow chart of a service fault location method provided by an embodiment of the present invention. This embodiment is applicable to situations where the root cause of a service fault is analyzed. The method can be executed by a service fault location device. The service fault location device can be implemented in the form of hardware and / or software. The service fault location device can be configured in an electronic device. Figure 1a As shown, the method includes:

[0055] Step 110: After detecting that a target service channel has a fault, trigger a fault location task that matches the target service channel; the target service channel includes multiple system services.

[0056] In this embodiment, the operation of each service can be monitored according to the operation status of each service channel in the service system. Each service channel can include multiple system services, such as service A, service B, and service C. Each service channel is separated and independent from each other.

[0057] Specifically, in this step, upon detecting a failure in a service channel (i.e., the target service channel) among the multiple service channels, a fault location task matching the target service channel may be triggered and executed to perform a root cause analysis of the failure in the target service channel. Optionally, the failure in the target service channel may include a service timeout or a service anomaly.

[0058] In existing business fault location methods, the same fault will instantly trigger many fault demarcation tasks. The fault assessment indicators used may include "call volume, success rate, average call duration, and call failure volume". In other words, one fault will trigger at least four business alarms, and four business alarms will trigger four fault demarcation tasks. As a result, too many fault demarcation tasks are triggered during the fault period, which not only causes task congestion but also wastes computing resources.

[0059] To address the above issues, this embodiment simplifies the traditionally basic metrics of "call volume, success rate, average call duration, and call failures" into two metrics that best reflect service failures: service timeouts and service exceptions. When both service timeout and service exception alarms occur simultaneously, the root cause of service timeouts can be prioritized for root cause identification, as they can lead to further service errors.

[0060] Assume that, in existing technology, a single fault across five services affects them, triggering abnormalities in their metrics for call volume, success rate, average call duration, and call failures. Using existing localization methods, this would trigger 5*4=20 fault demarcation tasks for the same fault. This optimization, achieved through this embodiment, only triggers a service timeout analysis task for one channel. This reduces resource waste from concurrent, repeated analysis tasks while also preventing misleading results from different analysis tasks. Based on this logic, this embodiment saves 75% of computing resources.

[0061] Step 120: Acquire call chain data corresponding to the target business channel, establish matching tags for the call chain data, and obtain target data to be analyzed corresponding to the target business channel.

[0062] In existing service fault location methods, after a fault is detected within a certain time period, only the full amount of data within that time period can be analyzed. As a result, irrelevant interference data will be used for analysis, and the amount of data analyzed is large.

[0063] In this embodiment, since each service channel is independent of each other, a fault in one service channel typically won't propagate to another. Therefore, the call chain data corresponding to the target service channel can be tagged based on its characteristic information. Compared to existing methods, this minimizes the need for precise data analysis, reduces the amount of data analysis, and improves the efficiency of service fault location.

[0064] In a specific embodiment, after obtaining the call chain data corresponding to the target business channel, a regular algorithm may be used to clean the call chain data, and then a matching tag may be established for the cleaned call chain data to obtain the target data to be analyzed.

[0065] Step 130: Determine the service node with the earliest fault occurrence time based on the fault occurrence time corresponding to the target data, aggregate the information of the service node, and determine the service or application instance with the largest abnormality ratio based on the aggregation result.

[0066] In existing business fault location methods, the deepest abnormal node is usually found according to the earliest call time. The root cause is only found based on the fault symptoms, which makes it difficult to determine the most accurate root cause result.

[0067] In this embodiment, the earliest service node to experience a failure (e.g., an abnormal node or a timed-out node) can be identified based on the time of failure occurrence and cached. Information about this service node is then aggregated based on the service and application instance dimensions. The aggregated results then identify the service or application instance with the largest proportion of abnormalities. Compared to existing technologies, this embodiment improves the accuracy of subsequent root cause analysis results by identifying the service or application instance with the largest proportion of abnormalities.

[0068] Step 140: Determine a root cause analysis result corresponding to the target business channel based on the abnormality ratio of the service or application instance.

[0069] In this embodiment, optionally, the root cause analysis result of the fault corresponding to the target business channel can be determined based on the abnormality ratio of the service or application instance and the associated application resources.

[0070] In a specific embodiment, assuming that a timeout failure occurs in the target business channel, the corresponding root cause analysis result may be memory overflow, thread deadlock, or connection not released, etc.; assuming that an abnormal failure occurs in the target business channel, the corresponding root cause analysis result may include service exception and remote call failure. The service exception may be a code logic exception, and the remote call failure may be a remote call exception or SQL execution exception, etc.

[0071] The technical solution provided by the embodiment of the present invention, after detecting that a fault has occurred in a target business channel, triggers a fault location task matching the target business channel, obtains call chain data corresponding to the target business channel, and establishes a matching label for the call chain data, obtains the target data to be analyzed corresponding to the target business channel, determines the business node with the earliest fault occurrence time according to the fault occurrence time corresponding to the target data, aggregates the information of the business node, determines the service or application instance with the largest abnormality ratio according to the aggregation result, and determines the fault root cause analysis result corresponding to the target business channel according to the abnormality ratio of the service or application instance. The technical means can improve the efficiency of locating business faults, improve the accuracy of locating results, and reduce resource consumption caused by the fault location process.

[0072] Based on the above embodiments, Figure 1b This can be a flowchart of a fault location task triggered in this embodiment, such as Figure 1b As shown in the figure, after a fault is detected in the target service channel, a fault location task matching the target service channel is triggered, including:

[0073] Step 101: Collect the number of service timeouts and service exceptions of the target service channel within a preset time period;

[0074] Step 102 , determine whether the number of service timeouts is greater than a preset first threshold; if so, execute step 103 ; if not, execute step 106 .

[0075] Step 103 , determining whether the target business channel triggers a business timeout analysis process in the previous acquisition cycle; if so, executing step 104 ; if not, executing step 105 .

[0076] Step 104: Only generate trigger records.

[0077] Step 105: Trigger a timeout analysis service that matches the target service channel.

[0078] Step 106 , determine whether the number of business anomalies is greater than a preset second threshold. If so, execute step 107 ; if not, execute step 108 .

[0079] Step 107 : Determine whether the target business channel triggers a business anomaly analysis process in the previous collection cycle. If so, execute step 104 ; if not, execute step 109 .

[0080] Step 108: Only generate trigger records.

[0081] Step 109: Trigger an abnormality analysis service that matches the target service channel.

[0082] In this embodiment, the need to trigger a fault location task in the current cycle is determined based on the fault location task triggering status in the previous consecutive cycle. This solves the problem of triggering the same location task in repeated cycles, further reducing computing resource consumption and improving location efficiency. Assuming that each service channel failure lasts 5 minutes, this embodiment can reduce the number of fault location tasks by 80%.

[0083] Figure 2 A flowchart of another service fault locating method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0084] Step 210: After detecting that a fault occurs in the target service channel, obtain the fault type corresponding to the target service channel, and trigger a fault location task matching the target service channel according to the fault type.

[0085] Step 220: Acquire call chain data corresponding to the target business channel, and create matching tags for the call chain data to obtain target data to be analyzed corresponding to the target business channel.

[0086] Step 230: Determine the service node with the earliest fault occurrence time based on the fault occurrence time corresponding to the target data, aggregate the information of the service node, and determine the service or application instance with the largest abnormality ratio based on the aggregation result.

[0087] Step 240 : Based on the abnormality ratio of the service or application instance, determine whether there is a target service or target application instance with an abnormality ratio greater than a preset threshold. If so, execute step 250 ; otherwise, execute step 260 .

[0088] In this step, specifically, it may be determined whether there are target service or target application instances with a proportion greater than 80%. If so, step 250 is executed; if not, step 260 is executed.

[0089] Step 250: Analyze the target service or target application instance to obtain a fault root cause analysis result corresponding to the target business channel.

[0090] Step 260: Perform multi-threaded analysis on the service or application instance to obtain a root cause analysis result corresponding to the target business channel.

[0091] In this embodiment, if there are target services or target application instances that account for more than 80%, a single analysis may be performed on the target services or target application instances; otherwise, a multi-threaded analysis may be performed (the analysis method is the same as the single analysis).

[0092] In one implementation of this embodiment, the target service or target application instance is analyzed to obtain a root cause analysis result corresponding to the target business channel, including: obtaining the current node corresponding to the target service or target application instance, and the associated resources and resource alarm information of the current node; determining whether the call type of the current node is a local call; if not, obtaining the downstream node of the current node, and the associated resources and resource alarm information of the downstream node; if the downstream node has resource alarm information, using the associated resources of the downstream node as the root cause analysis result; if the downstream node resources are normal, using the associated resources of the current node as the root cause analysis result.

[0093] In this embodiment, the current node corresponding to the target service or target application instance and the associated resources of the current node (including application instances, hosts, services, pods, nodes, middleware, etc.) can be obtained, and the alarm information of the resources can be obtained. Then, it is determined whether the call type of the current node is a non-local call. If so, the downstream node is continued to be obtained.

[0094] After obtaining the downstream node, you can continue to obtain the downstream node's associated resources (including application instances, hosts, services, pods, nodes, middleware, etc.) and obtain alarm information for these resources. If there are alarms in the downstream node's resources, the associated resources of the downstream node are directly used as the root cause analysis result. If all downstream node resources are normal, the associated resources of the current node are used as the root cause analysis result.

[0095] The advantage of this setting is that it can dig out the deepest fault node in the call chain and improve the accuracy of the root cause analysis results.

[0096] Step 270: Record the root cause analysis results of the same fault in different service channels in the same analysis record, so as to perform a combined analysis on the service channels.

[0097] In practical applications, if a fault lasts for more than ten minutes or even longer, it is inevitable that many alarms will be generated due to the fault. In order to avoid the time spread leading to too many analysis results that are difficult to judge, this embodiment provides a way to merge the analysis results.

[0098] In this step, analysis results from different channels caused by the same fault can be combined into one analysis record. Specifically, all analysis results with overlapping time, the same analysis type, and automatically triggered can be grouped into the same combined analysis record.

[0099] The time overlap refers to the intersection of the first trigger time and the latest trigger time in the analysis results; the analysis type refers to service timeout or service anomaly. The merging frequency can be set to once every 5 minutes. The specific value can be preset based on actual conditions and is not limited in this embodiment.

[0100] The technical solution provided by the embodiment of the present invention detects that a fault has occurred in a target business channel, obtains the fault type corresponding to the target business channel, triggers a fault location task matching the target business channel according to the fault type, obtains call chain data corresponding to the target business channel, and establishes a matching label for the call chain data to obtain the target data to be analyzed corresponding to the target business channel, determines the business node with the earliest fault occurrence time according to the fault occurrence time corresponding to the target data, aggregates the information of the business nodes, determines the service or application instance with the largest abnormality ratio according to the aggregation result, and judges whether there is a target service or target application instance with an abnormality ratio greater than a preset threshold. If so, analyzes the target service or target application instance to obtain a fault root cause analysis result corresponding to the target business channel. If not, performs multi-threaded analysis on the service or application instance to obtain a fault root cause analysis result corresponding to the target business channel. This technical means can improve the efficiency of locating business faults, improve the accuracy of locating results, and reduce resource consumption caused by the fault location process.

[0101] Figure 3 A schematic diagram of a structure of a service fault location device provided by an embodiment of the present invention, wherein the device is applied to an electronic device, such as Figure 3 As shown, the apparatus includes: a task triggering module 310 , a data acquisition module 320 , an information aggregation module 330 and a root cause analysis module 340 .

[0102] The task triggering module 310 is used to trigger a fault location task matching the target service channel after detecting a fault in the target service channel; the target service channel includes multiple system services;

[0103] The data acquisition module 320 is used to acquire the call chain data corresponding to the target business channel, and establish a matching tag for the call chain data to obtain the target data to be analyzed corresponding to the target business channel;

[0104] An information aggregation module 330 is configured to determine the service node with the earliest fault occurrence time based on the fault occurrence time corresponding to the target data, aggregate information of the service node, and determine the service or application instance with the largest abnormality ratio based on the aggregation result;

[0105] The root cause analysis module 340 is configured to determine a root cause analysis result of a fault corresponding to the target business channel according to the abnormality ratio of the service or application instance.

[0106] The technical solution provided by the embodiment of the present invention, after detecting that a fault has occurred in a target business channel, triggers a fault location task matching the target business channel, obtains call chain data corresponding to the target business channel, and establishes a matching label for the call chain data, obtains the target data to be analyzed corresponding to the target business channel, determines the business node with the earliest fault occurrence time according to the fault occurrence time corresponding to the target data, aggregates the information of the business node, determines the service or application instance with the largest abnormality ratio according to the aggregation result, and determines the fault root cause analysis result corresponding to the target business channel according to the abnormality ratio of the service or application instance. The technical means can improve the efficiency of locating business faults, improve the accuracy of locating results, and reduce resource consumption caused by the fault location process.

[0107] Based on the above embodiment, the task triggering module 310 includes:

[0108] A fault type acquisition unit is configured to, upon detecting a fault in a target service channel, acquire the fault type corresponding to the target service channel and, based on the fault type, trigger a fault location task matching the target service channel;

[0109] A data collection unit is used to collect the number of service timeouts and service exceptions of the target service channel within a preset time period;

[0110] A timeout task triggering unit is configured to determine whether the number of service timeouts is greater than a preset first threshold; if so, determine whether the target service channel has triggered a service timeout analysis process in the previous acquisition cycle; if not, trigger a timeout analysis service that matches the target service channel;

[0111] The abnormal task triggering unit is used to determine whether the number of business timeouts is greater than a preset first threshold. If not, it determines whether the number of business abnormalities is greater than a preset second threshold; if so, it determines whether the target business channel triggers the business abnormality analysis process in the previous collection cycle; if not, it triggers the abnormality analysis business matching the target business channel.

[0112] The root cause analysis module 340 includes:

[0113] a proportion judgment unit, configured to judge, based on the abnormal proportion of the service or application instance, whether there is a target service or target application instance whose abnormal proportion is greater than a preset threshold; if so, analyze the target service or target application instance to obtain a fault root cause analysis result corresponding to the target business channel; if not, perform multi-threaded analysis on the service or application instance to obtain a fault root cause analysis result corresponding to the target business channel;

[0114] A node analysis unit is used to obtain the current node corresponding to the target service or target application instance, as well as the associated resources and resource alarm information of the current node; determine whether the call type of the current node is a local call; if not, obtain the downstream node of the current node, as well as the associated resources and resource alarm information of the downstream node; if the downstream node has resource alarm information, use the associated resources of the downstream node as the root cause analysis result of the fault; if the downstream node resources are normal, use the associated resources of the current node as the root cause analysis result of the fault.

[0115] The device further comprises:

[0116] The result merging module is used to record the root cause analysis results of different business channels associated with the same fault into the same analysis record, so as to perform a combined analysis on each of the business channels.

[0117] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in the embodiments of the present invention, please refer to the methods provided by all the above embodiments of the present invention.

[0118] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0119] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0120] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the service fault location method.

[0122] In some embodiments, the service fault localization method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the service fault localization method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the service fault localization method in any other appropriate manner (e.g., via firmware).

[0123] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0128] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0129] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0130] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A service fault location method, characterized in that: The method comprises: After detecting that a target service channel has a fault, triggering a fault location task matching the target service channel; the target service channel includes multiple system services; Acquire call chain data corresponding to the target business channel, and establish matching tags for the call chain data to obtain target data to be analyzed corresponding to the target business channel; According to the fault occurrence time corresponding to the target data, determine the service node with the earliest fault occurrence time, aggregate the information of the service node, and determine the service or application instance with the largest abnormality proportion according to the aggregation result; Determine a root cause analysis result of a fault corresponding to the target business channel according to the abnormal proportion of the service or application instance; After a target service channel failure is detected, a fault location task matching the target service channel is triggered, including: After detecting that a fault occurs in a target business channel, obtaining a fault type corresponding to the target business channel; According to the fault type, trigger a fault location task matching the target service channel; According to the abnormal proportion of the service or application instance, the root cause analysis result of the fault corresponding to the target business channel is determined, including: According to the abnormal proportion of the service or application instance, determine whether there is a target service or target application instance whose abnormal proportion is greater than a preset threshold; If yes, then analyzing the target service or target application instance to obtain a fault root cause analysis result corresponding to the target business channel; If not, a multi-threaded analysis is performed on the service or application instance to obtain a fault root cause analysis result corresponding to the target business channel.

2. The method according to claim 1, characterized in that After a fault is detected in the target service channel, a fault location task matching the target service channel is triggered, including: Collect the number of business timeouts and business anomalies of the target business channel within a preset time period; Determine whether the number of service timeouts is greater than a preset first threshold; If yes, determine whether the target service channel triggers a service timeout analysis process in the previous collection cycle; If the target business channel has triggered the business timeout analysis process in the previous collection cycle, only a trigger task is generated; otherwise, the timeout analysis business matching the target business channel is triggered.

3. The method according to claim 2, characterized in that After determining whether the number of service timeouts is greater than a preset first threshold, the method further includes: If not, determining whether the number of service anomalies is greater than a preset second threshold; If the number of business anomalies is greater than a preset second threshold, determining whether the target business channel triggers a business anomaly analysis process in the previous collection cycle, otherwise, only generating a trigger record; If the target business channel has triggered the business anomaly analysis process in the previous collection cycle, only a trigger task is generated; otherwise, the anomaly analysis business matching the target business channel is triggered.

4. The method according to claim 1, characterized in that: Analyze the target service or target application instance to obtain a fault root cause analysis result corresponding to the target business channel, including: Obtaining the current node corresponding to the target service or target application instance, and the associated resources and resource alarm information of the current node; Determine whether the call type of the current node is a local call; If not, obtain the downstream node of the current node, as well as the associated resources and resource alarm information of the downstream node; If the downstream node has resource alarm information, the associated resources of the downstream node are used as the root cause analysis result of the fault; if the downstream node resources are normal, the associated resources of the current node are used as the root cause analysis result of the fault.

5. The method according to claim 1, characterized in that After determining the fault root cause analysis result corresponding to the target business channel, the method further includes: The root cause analysis results of different business channels associated with the same type of fault are recorded in the same analysis record to conduct a combined analysis of the business channels.

6. A service fault location device, characterized in that: The device comprises: A task triggering module is used to trigger a fault location task matching the target service channel after detecting that a fault occurs in the target service channel; the target service channel includes multiple system services; A data acquisition module, used to acquire call chain data corresponding to the target business channel, and establish matching tags for the call chain data to obtain target data to be analyzed corresponding to the target business channel; An information aggregation module is used to determine the service node with the earliest fault occurrence time according to the fault occurrence time corresponding to the target data, aggregate the information of the service node, and determine the service or application instance with the largest abnormality proportion according to the aggregation result; A root cause analysis module, used to determine a fault root cause analysis result corresponding to the target business channel according to the abnormal proportion of the service or application instance; The task triggering module includes: a fault type acquisition unit, which is used to acquire the fault type corresponding to the target business channel after detecting that a fault occurs in the target business channel; triggering a fault location task matching the target business channel according to the fault type; The root cause analysis module includes: a proportion judgment unit, which is used to judge whether there is a target service or target application instance with an abnormal proportion greater than a preset threshold according to the abnormal proportion of the service or application instance; if so, the target service or target application instance is analyzed to obtain a fault root cause analysis result corresponding to the target business channel; if not, a multi-threaded analysis is performed on the service or application instance to obtain a fault root cause analysis result corresponding to the target business channel.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the service fault locating method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the service fault locating method according to any one of claims 1 to 5 when executed.

Images