Intelligent Operation and Maintenance Methods for the Entire IT System Chain for Precise Fault Location

Abstract

This invention relates to the field of fault diagnosis technology, and in particular to an intelligent operation and maintenance method for the entire IT system chain for precise fault location. It constructs a dual dependency topology including explicit logical call edges and implicit resource contention edges, simultaneously covering business call associations and resource contention associations within the same host machine. This eliminates the blind spots in traditional solutions and is adaptable to complex fault scenarios involving business call anomalies, resource contention anomalies, and both. A micro-level queuing entropy increase rate quantification model is constructed. Through integral calculations of system scheduling saturation and effective instruction execution rate, it achieves refined quantification of micro-level anomalies at the kernel scheduling level, enabling early detection of fault precursors and accurate differentiation between native node anomalies and propagated anomalies. Starting from the alarm node, it traverses all associated nodes in reverse, and adjusts the weights of nodes with different anomaly types based on the node's micro-level queuing entropy increase rate, accurately quantifying the true fault contribution of each node.

Description

Technical Field

[0001] This invention relates to the field of fault diagnosis technology, and in particular to an intelligent operation and maintenance method for the entire IT system chain aimed at precise fault location. Background Technology

[0002] With the widespread adoption of cloud-native technologies, distributed microservice architecture has become the mainstream architecture for core IT systems in finance, internet, and other fields. The scale of system nodes and the complexity of call chains are increasing exponentially. Traditional manual operation and maintenance (O&M) and static monitoring models can no longer meet the O&M needs of high-availability systems, making end-to-end intelligent O&M technology a core development direction for the industry. Accurate fault location is the core of end-to-end intelligent O&M, directly determining the recovery time and business impact of system failures. Existing end-to-end O&M solutions mostly focus on tracing and analyzing explicit call chains at the business layer, lacking sufficient awareness of the micro-level operating state at the operating system kernel level. They also ignore implicit resource contention between microservice nodes deployed on the same host machine without business calls, resulting in blind spots in fault root cause location, insufficient location accuracy, and delayed response, failing to meet the fault location needs of complex distributed systems.

[0003] Existing end-to-end intelligent operation and maintenance and fault location technologies suffer from three major flaws. First, topology construction has inherent blind spots. Most existing solutions rely on RPC call TraceIDs to construct topology graphs containing only explicit business call relationships, completely ignoring implicit resource contention relationships between microservice nodes within the same host machine that lack business calls. This makes it impossible to locate cross-business resource contention faults, and in complex fault scenarios, root cause errors are easily missed. Second, anomaly quantification capabilities are severely insufficient. Existing solutions rely on macro-level performance indicators such as CPU utilization and interface response latency, failing to capture micro-level queuing anomalies at the kernel scheduling level. This not only results in delayed fault warnings but also an inability to finely quantify the degree of node anomalies, and an inability to distinguish between root cause nodes and fault propagation nodes. Third, the fault contribution calculation logic is inaccurate. Existing solutions often use fixed call level weight decay rules without considering the real-time running status of nodes for differentiated correction. This easily leads to incorrect root cause sorting, hinders accurate fault location, and has poor adaptability to the dynamic scaling of microservice architectures. Summary of the Invention

[0004] The main objective of this invention is to provide an intelligent end-to-end operation and maintenance method for IT systems with precise fault location. This solution constructs a dual dependency topology including explicit logical call edges and implicit resource contention edges, simultaneously covering business call associations and resource contention associations within the same host machine. This eliminates the blind spots in traditional solutions and is adaptable to complex fault scenarios involving business call anomalies, resource contention anomalies, and the coexistence of both. Based on eBPF kernel-level non-intrusive data collection, a micro-level queuing entropy increase rate quantification model is constructed. Through integral calculations of system scheduling saturation and effective instruction execution rate, refined quantification of micro-level anomalies at the kernel scheduling level is achieved, enabling early detection of fault precursors and accurate differentiation between native node anomalies and propagated anomalies. A differentiated weight correction and normalized fault contribution calculation system is designed. Starting from the alarm node, all associated nodes are traversed in reverse. The weights of nodes with different anomaly types are corrected based on the node's micro-level queuing entropy increase rate, accurately quantifying the true fault contribution of each node. This significantly improves the accuracy of root cause ranking, eliminates the need for manual configuration of fixed rules, adapts to the dynamic scaling characteristics of microservices, and significantly improves operation and maintenance efficiency and system availability.

[0005] The technical solution of the present invention is as follows:

[0006] Firstly, a full-link intelligent operation and maintenance method for IT systems, oriented towards precise fault location, is proposed. This method includes the following steps:

[0007] S1. Collect operating system kernel events and network protocol stack data packets of the entire chain of the distributed microservice architecture IT system, extract the basic runtime sequence data of each microservice node, and construct a dependency topology graph containing explicit logical call edges and implicit resource contention edges.

[0008] S2. Based on the basic runtime sequence data, the system scheduling saturation and effective instruction execution rate within the sliding time window are calculated. Then, the micro-queue entropy increase rate of each microservice node in the dependent topology graph within the sliding time window is obtained through integration, and a set of node micro-queue entropy increase rates is generated.

[0009] S3. Starting from the microservice node that triggered the alarm, traverse the dependency topology graph in reverse to obtain the set of microservice nodes to be evaluated. Extract the micro-queue entropy increase rate of all microservice nodes to be evaluated in the set of microservice nodes to be evaluated, process the initial fault weights of all microservice nodes to be evaluated, and then calculate the true fault contribution of each microservice node to be evaluated through normalization.

[0010] S4. Sort the actual fault contribution of each microservice node in the set of microservice nodes to be evaluated in descending order, obtain the suspected root cause nodes, generate the corresponding root cause links in combination with the dependency topology graph, and output the fault diagnosis results.

[0011] A further improvement of the present invention is that step S1 includes the following specific steps:

[0012] S11. Intercept operating system kernel events of the IT system host machine using eBPF probes. The kernel events include sched_switch scheduling events. Extract basic runtime sequence data for each microservice node. The basic runtime sequence data includes the number of requests arriving per unit time for each microservice node. CPU run queue length Context switching rate Thread ready wait delay Compared with the actual execution latency of the thread t is the index of time;

[0013] S12. Parse the TraceID in the RPC header of the network protocol stack data packet to construct explicit logical call edges between microservice nodes. The explicit logic call edge Directed edges are used to represent the business call relationships between microservice nodes; at the same time, different microservice nodes deployed on the same host machine are linked by adding implicit resource contention edges. The implicit resource contention edge is associated. To characterize the directed edges that relate to resource contention among microservice nodes, a dependency topology graph is obtained. The dependency topology graph It is a dual topology containing explicit logical call edges and implicit resource contention edges.

[0014] A further improvement of the present invention is that step S2 includes the following specific steps:

[0015] S21, in the sliding time window Internally, the system scheduling saturation is calculated. The calculation formula is: ;in, This refers to the weighting coefficient corresponding to the context switching rate. The weighting coefficient corresponding to the CPU run queue length. This represents the upper limit of the context switching rate corresponding to the host CPU. This represents the maximum length of the run queue corresponding to the host CPU. The starting time of the sliding time window. This is the end time of the sliding time window;

[0016] S22, in the sliding time window Internally, calculate the effective instruction execution rate. The calculation formula is: ;in, For smoothing correction;

[0017] S23. Calculate the micro-queue entropy increase rate of each microservice node in the dependent topology graph within the sliding time window using integral operations. The calculation formula is: ;in, For balance coefficient, Use a reference time constant; generate a set of node micro-queue entropy increase rates. , where i is the index of the microservice node in the dependency topology graph.

[0018] A further improvement of the present invention is that step S3 includes the following specific steps:

[0019] S31. Starting from the microservice node that triggered the alarm, follow the dependency topology graph. The directed edges are traversed in reverse. During the traversal, all microservice nodes that have business call relationships or resource contention relationships with the microservice node that triggered the alarm are covered, resulting in a set of microservice nodes to be evaluated.

[0020] S32. Extract the micro-queue entropy increase rate of all microservice nodes to be evaluated in the set of microservice nodes to be evaluated. When the micro-queue entropy increase rate of the j-th microservice node to be evaluated in the set of microservice nodes to be evaluated... Greater than the preset threshold When it is determined that the microservice node has a resource contention relationship with the microservice node that triggered the alarm, an exponential decay penalty is applied to the microservice node based on its initial fault weight. The formula for calculating the exponential decay penalty is as follows: ; This represents the processed fault weight of the microservice node. Let k be the initial fault weight of the microservice node, and k be the decay coefficient; when the micro queuing entropy increase rate of the j-th microservice node in the set of microservice nodes to be evaluated is... Not greater than the preset threshold Furthermore, if the actual execution latency of the thread corresponding to the microservice node exceeds the normal operating range, it is determined that there is an abnormal business call relationship between the microservice node and the microservice node that triggered the alarm, and the initial fault weight of the microservice node is retained. The normal operating interval is calculated as the average of the actual thread execution latency of the microservice node during the same period over the past 7 days, plus or minus 3 times the standard deviation.

[0021] S33. Extract the processed fault weights of all microservice nodes to be evaluated from the set of microservice nodes to be evaluated, and calculate the true fault contribution of each microservice node to be evaluated through normalization. The formula for normalization is: ; is the actual failure contribution rate of the j-th microservice node in the set of microservice nodes to be evaluated, and n is the total number of microservice nodes to be evaluated in the set of microservice nodes to be evaluated.

[0022] A further improvement of this invention is that the specific content of S4 is as follows: extract the actual fault contribution of each microservice node to be evaluated from the set of microservice nodes to be evaluated and sort them in descending order, obtain the top N microservice nodes to be evaluated as suspected root cause nodes of the fault, and combine them with the dependency topology graph. Extract the directed paths connecting N suspected root cause nodes to generate root cause links and output the fault diagnosis results.

[0023] Secondly, a computer-readable storage medium is proposed, on which a computer program is stored. When the computer program is executed by a processor, it realizes the above-mentioned intelligent operation and maintenance method for the entire IT system with precise fault location.

[0024] Thirdly, an electronic device is proposed, including a memory for storing instructions and a processor for executing the instructions, so that the device performs the above-mentioned intelligent operation and maintenance method for the entire IT system chain oriented to precise fault location.

[0025] The technical effects of this invention are as follows:

[0026] A comprehensive intelligent operation and maintenance (O&M) method for IT systems, oriented towards precise fault localization, was constructed. This method incorporates a dual dependency topology of explicit logical call edges and implicit resource contention edges, simultaneously covering business call associations and resource contention associations within the same host machine. This eliminates the blind spots in traditional solutions and is adaptable to complex fault scenarios involving business call anomalies, resource contention anomalies, and both. Based on non-intrusive kernel-level data collection using eBPF, a micro-level queuing entropy increase rate quantification model was built. Through integral calculations of system scheduling saturation and effective instruction execution rate, refined quantification of micro-level anomalies at the kernel scheduling level was achieved, enabling early detection of fault precursors and accurate differentiation between native node anomalies and propagated anomalies. A differentiated weight correction and normalized fault contribution calculation system was designed. Starting from the alarm node, all associated nodes are traversed in reverse. Weights are corrected for nodes of different anomaly types based on the node's micro-level queuing entropy increase rate, accurately quantifying the true fault contribution of each node. This significantly improves the accuracy of root cause ranking, eliminates the need for manual configuration of fixed rules, adapts to the dynamic scaling characteristics of microservices, and significantly improves O&M efficiency and system availability. Attached Figure Description

[0027] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0028] Figure 1This is a flowchart illustrating the end-to-end intelligent operation and maintenance method for IT systems with precise fault location according to Embodiment 1 of the present invention. Detailed Implementation

[0029] This embodiment proposes an intelligent end-to-end IT system operation and maintenance method for precise fault location. It constructs a dual-dependency topology including explicit logical call edges and implicit resource contention edges, simultaneously covering business call associations and resource contention associations within the same host machine. This eliminates the blind spots in traditional solutions and is adaptable to complex fault scenarios involving business call anomalies, resource contention anomalies, and both. Based on eBPF kernel-level non-intrusive data collection, a micro-queue entropy increase rate quantification model is constructed. Through integral calculations of system scheduling saturation and effective instruction execution rate, it achieves refined quantification of micro-anomalies at the kernel scheduling level, enabling early detection of fault precursors and accurate differentiation between native node anomalies and propagated anomalies. A differentiated weight correction and normalized fault contribution calculation system is designed. Starting from the alarm node, it traverses all associated nodes in reverse, and combines the node's micro-queue entropy increase rate to adjust the weights of nodes with different anomaly types, accurately quantifying the true fault contribution of each node. This significantly improves the accuracy of root cause ranking, eliminates the need for manual configuration of fixed rules, adapts to the dynamic scaling characteristics of microservices, and significantly improves operation and maintenance efficiency and system availability. Specifically, as shown... Figure 1 As shown, the IT system end-to-end intelligent operation and maintenance method for precise fault location proposed in this embodiment includes the following specific steps:

[0030] S1. Collect operating system kernel events and network protocol stack data packets of the entire chain of the distributed microservice architecture IT system, extract the basic runtime sequence data of each microservice node, and construct a dependency topology graph containing explicit logical call edges and implicit resource contention edges.

[0031] S2. Based on the basic runtime sequence data, the system scheduling saturation and effective instruction execution rate within the sliding time window are calculated. Then, the micro-queue entropy increase rate of each microservice node in the dependent topology graph within the sliding time window is obtained through integration, and a set of node micro-queue entropy increase rates is generated.

[0032] S3. Starting from the microservice node that triggered the alarm, traverse the dependency topology graph in reverse to obtain the set of microservice nodes to be evaluated. Extract the micro-queue entropy increase rate of all microservice nodes to be evaluated in the set of microservice nodes to be evaluated, process the initial fault weights of all microservice nodes to be evaluated, and then calculate the true fault contribution of each microservice node to be evaluated through normalization.

[0033] S4. Sort the actual fault contribution of each microservice node in the set of microservice nodes to be evaluated in descending order, obtain the suspected root cause nodes, generate the corresponding root cause links in combination with the dependency topology graph, and output the fault diagnosis results.

[0034] In this embodiment, step S1 includes the following specific steps:

[0035] S11. Intercept operating system kernel events of the IT system host machine using eBPF probes. The kernel events include sched_switch scheduling events. Extract basic runtime sequence data for each microservice node. The basic runtime sequence data includes the number of requests arriving per unit time for each microservice node. CPU run queue length Context switching rate Thread ready wait delay Compared with the actual execution latency of the thread t is the index of time;

[0036] S12. Parse the TraceID in the RPC header of the network protocol stack data packet to construct explicit logical call edges between microservice nodes. The explicit logic call edge Directed edges are used to represent the business call relationships between microservice nodes; at the same time, different microservice nodes deployed on the same host machine are linked by adding implicit resource contention edges. The implicit resource contention edge is associated. To characterize the directed edges that relate to resource contention among microservice nodes, a dependency topology graph is obtained. The dependency topology graph It is a dual topology containing explicit logical call edges and implicit resource contention edges.

[0037] In this embodiment, an eBPF probe is mounted in the operating system kernel of each host machine. The eBPF probe intercepts operating system kernel events of the IT system host machine. Kernel events include sched_switch scheduling events. Basic runtime sequence data of each microservice node is extracted. The basic runtime sequence data includes the number of requests arriving per unit time for each microservice node. CPU run queue length Context switching rate Thread ready wait delay Compared with the actual execution latency of the thread , The number of service requests received by the microservice node per unit time within the sampling period corresponding to time t. The number of threads waiting to be scheduled and executed in the run queue of the CPU cores bound to the microservice node during the sampling period at time t. The number of context switches that occur per unit time within the sampling period corresponding to time t for the thread belonging to the microservice node. The time consumed from the thread belonging to the microservice node entering the ready state to being scheduled and executed by the CPU within the sampling period corresponding to time t. This corresponds to the time consumed by the thread belonging to the microservice node to execute instructions on the CPU without interruption within the sampling period at time t. After extracting the basic runtime sequence data, the TraceID in the RPC protocol header of the network protocol stack data packet is parsed. For RPC request and response data packets carrying the same TraceID, the call and called relationship between microservice nodes is determined according to the direction of the request and the direction of the response. Explicit logical call edges between microservice nodes are constructed. Explicit logical call edges are directed edges representing the business call relationship between microservice nodes. The direction of the directed edge is from the called microservice node to the calling microservice node to adapt to subsequent reverse traversal operations. At the same time, different microservice nodes deployed on the same host are associated by adding implicit resource contention edges. Implicit resource contention edges are directed edges representing the resource contention relationship between microservice nodes. The direction of the directed edge is from the microservice node with lower resource priority to the microservice node with higher resource priority. The resource priority is determined by the kernel scheduling nice value of the thread to which the microservice node belongs. The higher the nice value, the lower the resource priority. Finally, a dependency topology graph is obtained. The dependency topology graph is a dual topology structure containing explicit logical call edges and implicit resource contention edges. The vertices of the topology graph are all microservice nodes in the cluster, and the edges of the topology graph are the set of the aforementioned explicit logical call edges and implicit resource contention edges.

[0038] In this embodiment, step S2 includes the following specific steps:

[0039] S21, in the sliding time window Internally, the system scheduling saturation is calculated. The calculation formula is: ;in, This refers to the weighting coefficient corresponding to the context switching rate. The weighting coefficient corresponding to the CPU run queue length. This represents the upper limit of the context switching rate corresponding to the host CPU. This represents the maximum length of the run queue corresponding to the host CPU. The starting time of the sliding time window. This is the end time of the sliding time window;

[0040] S22, in the sliding time window Internally, calculate the effective instruction execution rate. The calculation formula is: ;in, For smoothing correction;

[0041] S23. Calculate the micro-queue entropy increase rate of each microservice node in the dependent topology graph within the sliding time window using integral operations. The calculation formula is: ;in, For balance coefficient, Use a reference time constant; generate a set of node micro-queue entropy increase rates. , where i is the index of the microservice node in the dependency topology graph.

[0042] In this embodiment, after completing the construction of the dependency topology graph and the collection of basic runtime sequence data, the step of calculating the micro-queue entropy increase rate is performed. The core idea of ​​this step is to quantify the change in system disorder caused by the queuing behavior of microservice nodes during scheduling and execution using time-series data at the kernel scheduling level. The higher the disorder, the higher the probability of node performance abnormalities or failures, thereby achieving advance capture and quantitative characterization of abnormal node states. First, a sliding time window is set. The sliding time window width is set to 5 seconds, and the sliding step size is set to 1 second. Within this sliding time window, the system scheduling saturation is calculated. In the system scheduling saturation formula, This is the weighting coefficient corresponding to the context switching rate, with a value of 0.6. This value is adapted to the weighting of the impact of context switching on scheduling performance in the Linux kernel scheduling system. This is the weighting coefficient corresponding to the CPU run queue length, with a value of 0.4. This value is appropriate for the impact of run queue length on scheduling performance in the Linux kernel scheduling system. This represents the upper limit of the context switching rate for the host CPU, and is calculated as the number of host CPU cores multiplied by 10000. This represents the upper limit of the run queue length corresponding to the host CPU, which is calculated by multiplying the number of host CPU cores by 8. This value is adapted to the performance critical threshold of the CPU run queue length in the Linux kernel. After calculating the system scheduling saturation, the effective instruction execution rate is calculated within the same sliding time window. In the effective instruction execution rate formula, a smoothing correction term is included, with a value of 1ms, to avoid the denominator from reaching zero and to smooth out the interference of extreme sampling data on the calculation results. After calculating the system scheduling saturation and effective instruction execution rate, the micro-queue entropy increase rate of each microservice node in the dependent topology graph within the sliding time window is calculated through integral operation. The core of the integral operation is to accumulate the combined effect of scheduling saturation, effective instruction execution rate, run queue change rate, and waiting latency to execution latency ratio within the sliding time window to obtain the cumulative disorder change. In the micro-queue entropy increase rate calculation formula, the balance coefficient is set to 0.8 to balance the contribution weight of the dynamic change item of the run queue and the waiting execution latency ratio to the entropy increase rate. The reference time constant is set to 1ms. After completing the calculation of the micro-queue entropy increase rate of a single microservice node, all microservice nodes in the dependent topology graph are traversed to generate a set of node micro-queue entropy increase rates.

[0043] In this embodiment, step S3 includes the following specific steps:

[0044] S31. Starting from the microservice node that triggered the alarm, follow the dependency topology graph. The directed edges are traversed in reverse. During the traversal, all microservice nodes that have business call relationships or resource contention relationships with the microservice node that triggered the alarm are covered, resulting in a set of microservice nodes to be evaluated.

[0045] S32. Extract the micro-queue entropy increase rate of all microservice nodes to be evaluated in the set of microservice nodes to be evaluated. When the micro-queue entropy increase rate of the j-th microservice node to be evaluated in the set of microservice nodes to be evaluated... Greater than the preset threshold When it is determined that the microservice node has a resource contention relationship with the microservice node that triggered the alarm, an exponential decay penalty is applied to the microservice node based on its initial fault weight. The formula for calculating the exponential decay penalty is as follows: ; This represents the processed fault weight of the microservice node. Let k be the initial fault weight of the microservice node, and k be the decay coefficient; when the micro queuing entropy increase rate of the j-th microservice node in the set of microservice nodes to be evaluated is... Not greater than the preset threshold Furthermore, if the actual execution latency of the thread corresponding to the microservice node exceeds the normal operating range, it is determined that there is an abnormal business call relationship between the microservice node and the microservice node that triggered the alarm, and the initial fault weight of the microservice node is retained. The normal operating interval is calculated as the average of the actual thread execution latency of the microservice node during the same period over the past 7 days, plus or minus 3 times the standard deviation.

[0046] S33. Extract the processed fault weights of all microservice nodes to be evaluated from the set of microservice nodes to be evaluated, and calculate the true fault contribution of each microservice node to be evaluated through normalization. The formula for normalization is: ; is the actual failure contribution rate of the j-th microservice node in the set of microservice nodes to be evaluated, and n is the total number of microservice nodes to be evaluated in the set of microservice nodes to be evaluated.

[0047] In this embodiment, starting from the microservice node that triggered the alarm, a reverse traversal is performed along the directed edges of the dependency topology graph. The traversal process adopts a depth-first traversal method, and the traversal depth is set to 6 layers. During the traversal, all microservice nodes that have business call relationships or resource contention associations with the microservice node that triggered the alarm are covered, resulting in a set of microservice nodes to be evaluated. Each element in the set corresponds to a microservice node to be evaluated, where j is the index of the node in the set of microservice nodes to be evaluated, and the index value ranges from 1 to the total number of microservice nodes to be evaluated, n. After obtaining the set of microservice nodes to be evaluated, the micro-queue entropy increase rate of all microservice nodes in the set is extracted. When the micro-queue entropy increase rate of the j-th microservice node in the set exceeds a preset threshold, it is determined that the microservice node has a resource contention relationship with the microservice node that triggered the alarm. An exponential decay penalty is applied to this microservice node based on its initial fault weight. The preset threshold is 0.5, which is suitable for the fluctuation range of the micro-queue entropy increase rate under normal operating conditions. The calculation formula for the exponential decay penalty is as follows: The initial fault weight is set for the j-th microservice node to be evaluated. The value of the initial fault weight is related to the level of the reverse traversal. For each additional level, the initial fault weight decreases by 0.2. The initial fault weight of the first-level node directly associated with the alarm node is 1. k is the decay coefficient, with a value of 0.3, used to control the rate of exponential decay and adapt to the impact of resource contention on the fault contribution. When the micro-queue entropy increase rate of the j-th microservice node in the set of microservice nodes to be evaluated is not greater than a preset threshold and the actual execution latency of the thread corresponding to this microservice node exceeds the normal operating range, it is determined that there is an abnormal business call relationship between this microservice node and the microservice node that triggered the alarm. The initial fault weight of this microservice node is retained. The normal operating range is set to the average actual execution latency of the thread of this microservice node in the same period over the past 7 days plus or minus 3 standard deviations. The time granularity of the same period is 1 hour, used to match the periodic fluctuation characteristics of business traffic. After processing the fault weights of all microservice nodes to be evaluated, the processed fault weights of all microservice nodes to be evaluated in the set of microservice nodes to be evaluated are extracted, and the true fault contribution of each microservice node to be evaluated is obtained by normalization calculation.

[0048] In this embodiment, the specific content of S4 is as follows: extract the actual fault contribution of each microservice node to be evaluated from the set of microservice nodes to be evaluated and sort them in descending order, obtain the top N microservice nodes to be evaluated as suspected root cause nodes of the fault, and combine them with the dependency topology graph. Extract the directed paths connecting N suspected root cause nodes to generate root cause links and output the fault diagnosis results.

[0049] In this embodiment, after calculating the actual fault contribution of all microservice nodes to be evaluated, the steps of fault root cause localization and result output are performed. The actual fault contribution of each microservice node to be evaluated in the set of microservice nodes to be evaluated is extracted and sorted in descending order. The top N microservice nodes to be evaluated are obtained as suspected fault root cause nodes, where N is 3, which is used to cover the core fault root cause nodes and associated transmission nodes. Combined with the dependency topology graph, the directed path connecting the N suspected fault root cause nodes is extracted to generate the fault root cause link. The extraction of the directed path follows the direction of reverse traversal, covering the explicit logical call edges and implicit resource contention edges between nodes. Finally, the fault diagnosis result is output. The fault diagnosis result includes the identification information of the suspected fault root cause node, the actual fault contribution value, and the topology information of the fault root cause link.

[0050] The threshold can be set using the default settings according to the present invention, or it can be set by the operator.

[0051] Example 2

[0052] This embodiment provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that can be called by the processor; the processor executes the above-mentioned intelligent operation and maintenance method for the entire IT system for precise fault location by calling the computer program stored in the memory.

[0053] The electronic device can vary considerably depending on its configuration or performance. It may include one or more Central Processing Units (CPUs) and one or more memories, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the end-to-end intelligent operation and maintenance method for IT systems with precise fault location provided in the above-described embodiment. The electronic device may also include other components for implementing its functions; for example, it may have wired or wireless network interfaces and input / output interfaces for data input and output. Further details are omitted in this embodiment.

[0054] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0055] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0056] This invention is described with reference to flowchart illustrations and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and block diagrams, as well as combinations of blocks in the flowchart illustrations and block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An intelligent operation and maintenance method for the entire IT system chain aimed at precise fault location, characterized by: The specific steps include the following: S1. Collect operating system kernel events and network protocol stack data packets of the entire chain of the distributed microservice architecture IT system, extract the basic runtime sequence data of each microservice node, and construct a dependency topology graph containing explicit logical call edges and implicit resource contention edges. S2. Based on the basic runtime sequence data, the system scheduling saturation and effective instruction execution rate within the sliding time window are calculated. Then, the micro-queue entropy increase rate of each microservice node in the dependent topology graph within the sliding time window is obtained through integration, and a set of node micro-queue entropy increase rates is generated. S3. Starting from the microservice node that triggered the alarm, traverse the dependency topology graph in reverse to obtain the set of microservice nodes to be evaluated. Extract the micro-queue entropy increase rate of all microservice nodes to be evaluated in the set of microservice nodes to be evaluated, process the initial fault weights of all microservice nodes to be evaluated, and then calculate the true fault contribution of each microservice node to be evaluated through normalization. S4. Sort the actual fault contribution of each microservice node in the set of microservice nodes to be evaluated in descending order, obtain the suspected root cause nodes, generate the corresponding root cause links in combination with the dependency topology graph, and output the fault diagnosis results.

2. The IT system end-to-end intelligent operation and maintenance method for precise fault location according to claim 1, characterized in that: S1 includes the following specific steps: S11. Intercept operating system kernel events of the IT system host machine using eBPF probes. The kernel events include sched_switch scheduling events. Extract basic runtime sequence data for each microservice node. The basic runtime sequence data includes the number of requests arriving per unit time for each microservice node. CPU run queue length Context switching rate Thread ready wait delay Compared with the actual execution latency of the thread t is the index of time; S12. Parse the TraceID in the RPC header of the network protocol stack data packet to construct explicit logical call edges between microservice nodes. The explicit logic call edge Directed edges are used to represent the business call relationships between microservice nodes; Simultaneously, different microservice nodes deployed on the same host machine will have implicit resource contention edges added. The implicit resource contention edge is associated. To characterize the directed edges that relate to resource contention among microservice nodes, a dependency topology graph is obtained. The dependency topology graph It is a dual topology containing explicit logical call edges and implicit resource contention edges.

3. The IT system end-to-end intelligent operation and maintenance method for precise fault location according to claim 2, characterized in that: S2 includes the following specific steps: S21, in the sliding time window Internally, the system scheduling saturation is calculated. The calculation formula is: ;in, This refers to the weighting coefficient corresponding to the context switching rate. The weighting coefficient corresponding to the CPU run queue length. This represents the upper limit of the context switching rate corresponding to the host CPU. This represents the maximum length of the run queue corresponding to the host CPU. The starting time of the sliding time window. This is the end time of the sliding time window; S22, in the sliding time window Internally, calculate the effective instruction execution rate. The calculation formula is: ;in, For smoothing correction; S23. Calculate the micro-queue entropy increase rate of each microservice node in the dependent topology graph within the sliding time window using integral operations. The calculation formula is: ;in, For balance coefficient, Use a reference time constant; generate a set of node micro-queue entropy increase rates. , where i is the index of the microservice node in the dependency topology graph.

4. The IT system end-to-end intelligent operation and maintenance method for precise fault location according to claim 3, characterized in that: S3 includes the following specific steps: S31. Starting from the microservice node that triggered the alarm, follow the dependency topology graph. The directed edges are traversed in reverse. During the traversal, all microservice nodes that have business call relationships or resource contention relationships with the microservice node that triggered the alarm are covered, resulting in a set of microservice nodes to be evaluated. S32. Extract the micro-queue entropy increase rate of all microservice nodes to be evaluated in the set of microservice nodes to be evaluated. When the micro-queue entropy increase rate of the j-th microservice node to be evaluated in the set of microservice nodes to be evaluated... Greater than the preset threshold When it is determined that the microservice node has a resource contention relationship with the microservice node that triggered the alarm, an exponential decay penalty is applied to the microservice node based on its initial fault weight. The formula for calculating the exponential decay penalty is as follows: ; This represents the processed fault weight of the microservice node. Let k be the initial fault weight of the microservice node, and k be the decay coefficient; when the micro queuing entropy increase rate of the j-th microservice node in the set of microservice nodes to be evaluated is... Not greater than the preset threshold Furthermore, if the actual execution latency of the thread corresponding to the microservice node exceeds the normal operating range, it is determined that there is an abnormal business call relationship between the microservice node and the microservice node that triggered the alarm, and the initial fault weight of the microservice node is retained. The normal operating interval is calculated as the average of the actual thread execution latency of the microservice node during the same period over the past 7 days, plus or minus 3 times the standard deviation. S33. Extract the processed fault weights of all microservice nodes to be evaluated from the set of microservice nodes to be evaluated, and calculate the true fault contribution of each microservice node to be evaluated through normalization. The formula for normalization is: ; is the actual failure contribution rate of the j-th microservice node in the set of microservice nodes to be evaluated, and n is the total number of microservice nodes to be evaluated in the set of microservice nodes to be evaluated.

5. The IT system end-to-end intelligent operation and maintenance method for precise fault location according to claim 4, characterized in that: The specific content of S4 is as follows: extract the actual fault contribution of each microservice node to be evaluated from the set of microservice nodes to be evaluated and sort them in descending order, obtain the top N microservice nodes to be evaluated as suspected root cause nodes of the fault, and combine them with the dependency topology graph. Extract the directed paths connecting N suspected root cause nodes to generate root cause links and output the fault diagnosis results.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the IT system end-to-end intelligent operation and maintenance method for precise fault location as described in any one of claims 1-5.

7. An electronic device, characterized in that, It includes a memory for storing instructions; and a processor for executing the instructions, causing the device to perform the IT system end-to-end intelligent operation and maintenance method for precise fault location as described in any one of claims 1 to 5.

Images