Distributed system fault location and diagnosis method and system based on log analysis

Abstract

The present invention provides a distributed system fault location and diagnosis method and system based on log analysis, which relates to the field of fault diagnosis technology. The method includes collecting call data between microservice nodes, constructing a service dependency graph, and calculating the influence weight of each node as a fault source. The service dependency graph is modeled using a graph convolutional neural network to construct a fault propagation prediction model. When an abnormal node is detected, the node is input into the prediction model, the fault propagation path and impact range are calculated, and a list of affected service nodes and their impact level is output. Risk level classification and alarms are performed, and the fault propagation path is visually displayed. The present invention can quickly and accurately locate the source of the fault, predict the impact of fault propagation, and improve the reliability and stability of the distributed system.

Description

Technical Field

[0001] The present invention relates to fault diagnosis technology, and in particular to a distributed system fault location diagnosis method and system based on log analysis. Background Art

[0002] With the prevalence of distributed systems and microservices architectures, system complexity continues to increase, making fault location and diagnosis increasingly difficult. Traditional troubleshooting methods often rely on manual experience and log analysis, which are inefficient and difficult to handle complex failure scenarios. Especially in a microservices architecture, where inter-service call relationships are complex, a service failure can quickly propagate along the call chain, paralyzing a wide range of services. Therefore, quickly and accurately locating the source of a fault and predicting its impact are crucial.

[0003] First, traditional log analysis methods require manual analysis of massive amounts of log data line by line, which is time-consuming, labor-intensive, and error-prone. In a microservices architecture, logs are dispersed across different service nodes, further increasing the difficulty of log analysis.

[0004] Secondly, some fault diagnosis methods based on monitoring indicators can only identify abnormal service nodes but cannot determine the root cause and propagation path of the fault. This makes it difficult for operations personnel to quickly locate the source of the fault and take effective remediation measures.

[0005] Finally, most existing fault prediction models are based on simple rules or statistical methods, lacking a deep understanding of service dependencies and fault propagation patterns. This results in low prediction accuracy and makes it difficult to effectively predict the impact scope and potential risks of faults. Summary of the Invention

[0006] The embodiments of the present invention provide a distributed system fault location and diagnosis method and system based on log analysis, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a distributed system fault location and diagnosis method based on log analysis, including:

[0009] Obtain information about all microservice nodes through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service ID, callee service ID, call interface name, call parameters, and call status; based on the structured stored call data, use a graph database to build a service dependency graph, where graph nodes represent microservice nodes and graph edges represent call relationships between services, and annotate the graph edges with call frequency, average response time, and success rate;

[0010] Based on the service dependency graph, for each microservice node, the impact weight of the node on other microservice nodes when it is a fault source is calculated. The impact weight is calculated based on the call frequency between services, the call link length, the service importance, and historical fault data. The service dependency graph is modeled using a graph convolutional neural network. The operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model.

[0011] The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node. The propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact level are output. The affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

[0012] Obtain all microservice node information through the service registration center in the distributed system, collect call data between microservice nodes based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service ID, callee service ID, call interface name, call parameters, and call status, including:

[0013] Obtain registration information of all microservice nodes through the service registration center in the distributed system. The registration information includes service name, instance identifier, network address, port number, and health check address. The service registration center stores the registration information in the form of temporary nodes and maintains the survival status of the microservice nodes through a heartbeat mechanism;

[0014] The application program interface call link tracker collects call data between each microservice node. The application program interface call link tracker injects tracing logic into the remote call entry point of the microservice node, generates a globally unique tracing identifier and call identifier, and propagates the tracing identifier and the call identifier in the call link; the application program interface call link tracker records the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status;

[0015] The call data is sent to a message queue for temporary storage. A dedicated consumer cluster obtains the call data from the message queue and performs sharding processing. The sharding processing divides the call data into multiple data shards according to the service dimension; the sharded call data is written into a distributed time series database for structured storage.

[0016] Based on the service dependency graph, for each microservice node, calculate its impact weight on other microservice nodes when it is a fault source. The impact weight is calculated based on the call frequency between services, call link length, service importance, and historical fault data, including:

[0017] Based on the service dependency graph, the microservice node to be analyzed is selected as the fault source node, and the following steps are performed to calculate the impact weight of the fault source node on other microservice nodes:

[0018] A graph traversal algorithm is used to analyze the call link from the fault source node to each of the microservice nodes, extract the call frequency data on the call link, and multiply the call frequencies on the call link in sequence to obtain an inter-service call frequency impact factor;

[0019] Determining the call link length based on the number of node hops of the call link, and calculating a call link length impact factor according to a preset decay function, wherein the call link length impact factor decreases as the number of node hops of the call link increases;

[0020] Obtaining service importance data of the microservice node, quantifying the service importance data based on business importance, user impact range, and business loss degree, and obtaining a service importance impact factor;

[0021] Analyze the historical fault records of the microservice node, extract the fault frequency, fault recovery time, and fault spread characteristics, and construct the historical fault impact factor;

[0022] Perform a weighted combination of the inter-service call frequency impact factor, the call link length impact factor, the service importance impact factor, and the historical fault impact factor to obtain the impact weight of the fault source node on the current microservice node;

[0023] Repeat the above steps until the influence weight of the fault source node on all other microservice nodes in the service dependency graph is calculated.

[0024] A graph convolutional neural network is used to model the service dependency graph. The operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model, including:

[0025] Based on the service call log, the call frequency between microservice nodes is counted; the call frequency is normalized to generate edge weight values; a weighted directed graph structure is constructed, wherein the directed graph structure includes a set of microservice nodes, a set of call relationship edges, and an edge weight matrix; self-loop connections are added to the diagonal of the adjacency matrix of the directed graph structure to generate an asymmetric adjacency matrix;

[0026] Collecting multi-dimensional features of nodes, collecting operating status indicators of each node in the microservice node set, wherein the operating status indicators include average response time, request success rate, and error rate; collecting system resource indicators of each node in the microservice node set, wherein the system resource indicators include CPU usage, memory usage, and network IO; collecting business indicators of each node in the microservice node set, wherein the business indicators include service call volume, business processing delay, and business success rate; calculating the historical mean and standard deviation of the operating status indicator, the system resource indicator, and the business indicator; and normalizing the operating status indicator, the system resource indicator, and the business indicator based on the historical mean and the standard deviation to generate an initial node feature matrix;

[0027] Constructing a graph convolutional feature extraction network, inputting the asymmetric adjacency matrix and the initial node feature matrix into a first graph convolutional layer; calculating the first layer of node features through the first graph convolutional layer; fusing the first layer of node features with the original features through a residual connection; inputting the fused features into a second graph convolutional layer; introducing an attention mechanism into the second graph convolutional layer, performing weighted aggregation on neighbor node features based on the edge weight values; and outputting the final node representation features;

[0028] A multi-task prediction model is constructed, and the node representation features are input into a multi-layer perceptron; in a first branch of the multi-layer perceptron, a fault state classifier is constructed based on the node representation features; in a second branch of the multi-layer perceptron, a similarity matrix of the node representation features is calculated; a fault propagation prediction model is constructed based on the similarity matrix; and the outputs of the fault state classifier and the fault propagation prediction model are combined to generate a final prediction result.

[0029] The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node. The propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network. The list of affected service nodes and their impact levels are output, including:

[0030] Collect microservice operation data and deploy monitoring probes to collect operation status indicators of microservice nodes, including response time, request success rate, and error rate; collect system resource indicators of microservice nodes, including CPU utilization, memory utilization, and network IO; collect business indicators of microservice nodes, including service call volume, business processing delay, and business success rate; use a sliding time window to aggregate the operation status indicators, system resource indicators, and business indicators, with a window size of 5 minutes and a step size of 1 minute; calculate the statistical characteristics of each indicator within the sliding time window to generate a real-time monitoring feature matrix;

[0031] Constructing an anomaly detection baseline, constructing a Gaussian distribution model for continuous indicators in the real-time monitoring feature matrix; constructing a frequency distribution model for discrete indicators in the real-time monitoring feature matrix; combining the Gaussian distribution model and the frequency distribution model to form a normal operation baseline; calculating a deviation measure between the real-time monitoring feature matrix and the normal operation baseline; performing a weighted combination of the deviation measures to obtain an anomaly score; and identifying a fault source node based on the anomaly score;

[0032] Based on the fault propagation model, the anomaly score of the fault source node and the real-time monitoring feature matrix are combined to form an initial node state vector; the initial node state vector is input into a pre-trained graph convolutional neural network; the node state is updated through propagation through the graph convolution layer; the influence strength between nodes is calculated based on the updated node state; and a fault impact propagation matrix is ​​generated;

[0033] Calculate the impact range, calculate the topological impact score between nodes based on the fault impact propagation matrix; calculate the dependency impact score between nodes based on the service call relationship; fuse historical fault data to calculate the vulnerability impact score of the node; perform weighted fusion of the topological impact score, the dependency impact score, and the vulnerability impact score; and generate a comprehensive risk score for the node;

[0034] Output prediction results, classify microservice nodes into risk levels based on the comprehensive risk score; identify high-risk nodes as key focus points; calculate the expected impact time of each node; generate a prediction result report including risk level and impact time; output the prediction result report to guide fault emergency response.

[0035] The affected service nodes are classified into risk levels based on the degree of impact, and alarm information of corresponding levels is triggered according to the preset risk threshold. At the same time, the fault propagation path is visualized, including:

[0036] Collect microservice fault propagation data, including fault occurrence time, fault impact range, fault propagation path, fault propagation speed, and fault duration;

[0037] Extracting impact characteristics based on the microservice fault propagation data, including calculating the probability of fault propagation between nodes, extracting fault time delay characteristics, and statistically analyzing the overlap of fault impact ranges to construct a fault impact feature matrix; performing feature dimensionality reduction and normalization on the fault impact feature matrix, extracting key influencing factors using principal component analysis, and calculating the weight coefficient of each influencing factor;

[0038] Calculating a comprehensive impact score for each affected service node based on the key impact factors and the weight coefficients, and dividing the comprehensive impact scores into different risk levels according to a preset risk level classification rule; setting corresponding risk thresholds for different risk levels, and triggering an alarm message of a corresponding level when the comprehensive impact score of a service node exceeds the risk threshold of the corresponding risk level;

[0039] Construct an alarm information template, which includes the fault impact range, fault propagation path, risk level determination basis, and processing suggestions, and generate standardized alarm information based on the alarm information template; use a graphical method to display the fault propagation path, including constructing a service dependency diagram, and marking the fault propagation direction, propagation speed, and impact range in the service dependency diagram.

[0040] According to a second aspect of the embodiments of the present invention,

[0041] Provides a distributed system fault location and diagnosis system based on log analysis, including:

[0042] The first unit is used to obtain information about all microservice nodes through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters, and call status; based on the structured stored call data, a service dependency graph is constructed using a graph database, where graph nodes represent microservice nodes, graph edges represent call relationships between services, and call frequency, average response time, and success rate are annotated on the graph edges;

[0043] The second unit is used to calculate, for each microservice node, the impact weight of the node on other microservice nodes when it is a fault source based on the service dependency graph. The impact weight is calculated based on the call frequency between services, the call link length, the service importance, and historical fault data. The service dependency graph is modeled using a graph convolutional neural network. The operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model.

[0044] The third unit is used to collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact level are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

[0045] According to a third aspect of the embodiments of the present invention,

[0046] An electronic device is provided, comprising:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0050] According to a fourth aspect of the embodiments of the present invention,

[0051] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0052] The beneficial effects of this application are as follows:

[0053] Fast and accurate fault location: This invention uses graph convolutional neural networks to model the service dependency graph and combines it with real-time operating status data to quickly identify the fault source node and predict the fault propagation path and impact range, thereby achieving accurate fault location.

[0054] Effectively reduce the false alarm rate: This invention comprehensively calculates the impact weight based on multi-dimensional information such as service call frequency, call link length, service importance and historical fault data, and combines it with the predictive ability of graph convolutional neural networks to effectively reduce the false alarm rate of faults and improve the accuracy of fault diagnosis.

[0055] Improve operation and maintenance efficiency: This invention provides a list of affected service nodes and their impact, and performs risk level classification and alarm. At the same time, it visualizes the fault propagation path, which can help operation and maintenance personnel quickly understand the fault situation and take timely response measures, thereby improving operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of a distributed system fault location and diagnosis method based on log analysis according to an embodiment of the present invention;

[0057] Figure 2 This is a structural diagram of a distributed system fault location and diagnosis system based on log analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 1 FIG is a flow chart of a distributed system fault location and diagnosis method based on log analysis according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0061] S11. Obtain information about all microservice nodes through the service registry in the distributed system. Collect call data between each microservice node using the application program interface call link tracker. This call data is structured and stored according to the call timestamp, caller service ID, callee service ID, call interface name, call parameters, and call status. Based on the structured call data, a service dependency graph is constructed using a graph database, where graph nodes represent microservice nodes and graph edges represent call relationships between services. Call frequency, average response time, and success rate are annotated on the graph edges.

[0062] S12. Based on the service dependency graph, for each microservice node, calculate its impact weight on other microservice nodes when acting as a fault source. This impact weight is calculated based on the inter-service call frequency, call link length, service importance, and historical fault data. A graph convolutional neural network is used to model the service dependency graph, using the operational status indicators, system resource usage, and business indicators of each microservice node as node feature inputs to construct a fault propagation prediction model.

[0063] S13. Collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, input the abnormal node as the fault source node into the fault propagation prediction model, calculate the propagation path and impact range of the fault in the service dependency graph based on the graph convolutional neural network, and output a list of affected service nodes and their impact level; classify the affected service nodes into risk levels according to the impact level, and trigger the corresponding level of alarm information according to the preset risk threshold, and visualize the fault propagation path.

[0064] In an optional embodiment, all microservice node information is obtained through a service registration center in a distributed system, call data between microservice nodes is collected based on an application program interface call link tracker, and the call data is structured and stored according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters, and call status, including:

[0065] Obtain registration information of all microservice nodes through the service registration center in the distributed system. The registration information includes service name, instance identifier, network address, port number, and health check address. The service registration center stores the registration information in the form of temporary nodes and maintains the survival status of the microservice nodes through a heartbeat mechanism;

[0066] The application program interface call link tracker collects call data between each microservice node. The application program interface call link tracker injects tracing logic into the remote call entry point of the microservice node, generates a globally unique tracing identifier and call identifier, and propagates the tracing identifier and the call identifier in the call link; the application program interface call link tracker records the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status;

[0067] The call data is sent to a message queue for temporary storage. A dedicated consumer cluster obtains the call data from the message queue and performs sharding processing. The sharding processing divides the call data into multiple data shards according to the service dimension; the sharded call data is written into a distributed time series database for structured storage.

[0068] A microservice call data collection and storage method based on a service registry and a call link tracker is used to collect and store call data between microservices in a distributed system for performance analysis, troubleshooting, and system optimization.

[0069] First, obtain the registration information of all microservice nodes through the service registry in the distributed system. The service registry, such as ZooKeeper, Consul, or Etcd, stores information about each microservice instance, including the service name, instance identifier (such as UUID), network address (such as IP address), port number, and health check address (such as HTTP interface). This information is stored in the form of temporary nodes, and the survival status of the microservice nodes is maintained through the heartbeat mechanism. The service registry regularly checks the health of each instance. If the heartbeat of an instance times out, it will be removed from the registry.

[0070] Next, the call link tracer is used to collect call data between microservice nodes through the application program interface (API). Common call link tracers, such as Jaeger, Zipkin, or SkyWalking, inject tracing logic into the remote call entry point of the microservice node. When one service calls another service, the tracer generates a globally unique trace identifier (trace ID) and call identifier (span ID) and propagates these identifiers in the call link. For example, a user request triggers service A to call service B, and service B calls service C. The entire call link shares the same trace ID, while the calls to services A, B, and C have different span IDs. The tracer records the timestamp, caller service ID, callee service ID, call interface name, call parameters, and call status (such as success or failure) of each call.

[0071] The collected call data is then sent to a message queue, such as Kafka or RabbitMQ, for temporary storage. A dedicated consumer cluster retrieves the call data from the message queue and performs sharding processing. Sharding divides the call data into multiple data shards based on the service dimension, such as the "order-service" shard and the "payment-service" shard.

[0072] Finally, the sharded call data is written to a distributed time series database for structured storage, such as InfluxDB, Prometheus, or OpenTSDB. Time series databases can efficiently store and query time series data.

[0073] The beneficial effects of this method are reflected in the following three aspects:

[0074] First, it improves system observability. By collecting and storing call data between microservices, we can clearly understand the call relationships, call counts, call duration, and other information of each service in the system, thereby better monitoring the system operation status and promptly identifying and resolving performance bottlenecks.

[0075] Second, it improves troubleshooting efficiency. When a system failure occurs, the cause can be quickly located by tracking call chain data, shortening troubleshooting time and improving system stability. For example, by analyzing error messages and call durations in the call chain, the service and code causing the failure can be quickly identified.

[0076] Third, it supports system performance optimization. By analyzing call chain data, we can identify performance bottlenecks in the system, such as services with excessive call counts or excessively long call times. This allows targeted optimization to improve overall system performance. For example, based on the call data analysis results, we can optimize code, cache, or database for certain services.

[0077] In an optional embodiment, based on the service dependency graph, for each microservice node, the impact weight of the microservice node as a fault source on other microservice nodes is calculated. The impact weight is calculated based on the call frequency between services, the call link length, the importance of the service, and historical fault data, including:

[0078] Based on the service dependency graph, the microservice node to be analyzed is selected as the fault source node, and the following steps are performed to calculate the impact weight of the fault source node on other microservice nodes:

[0079] A graph traversal algorithm is used to analyze the call link from the fault source node to each of the microservice nodes, extract the call frequency data on the call link, and multiply the call frequencies on the call link in sequence to obtain an inter-service call frequency impact factor;

[0080] Determining the call link length based on the number of node hops of the call link, and calculating a call link length impact factor according to a preset decay function, wherein the call link length impact factor decreases as the number of node hops of the call link increases;

[0081] Obtaining service importance data of the microservice node, quantifying the service importance data based on business importance, user impact range, and business loss degree, and obtaining a service importance impact factor;

[0082] Analyze the historical fault records of the microservice node, extract the fault frequency, fault recovery time, and fault spread characteristics, and construct the historical fault impact factor;

[0083] Perform a weighted combination of the inter-service call frequency impact factor, the call link length impact factor, the service importance impact factor, and the historical fault impact factor to obtain the impact weight of the fault source node on the current microservice node;

[0084] Repeat the above steps until the influence weight of the fault source node on all other microservice nodes in the service dependency graph is calculated.

[0085] A method for calculating node impact weights in service dependency graphs is used to assess the impact of each service node on other nodes when acting as a fault source in a microservices architecture. This method comprehensively considers inter-service call frequency, call link length, service importance, and historical fault data to more accurately quantify the impact of a fault.

[0086] First, build a service dependency graph. This graph uses microservices as nodes and the call relationships between services as edges. Each edge contains call frequency information, such as the number of calls per minute. For example, consider a system consisting of four microservices, A, B, C, and D. A calls B 10 times per minute, A also calls C 5 times per minute, B calls D 2 times per minute, and C also calls D 3 times per minute.

[0087] Next, select the microservice node to be analyzed as the fault source node. Suppose A is selected as the fault source node.

[0088] Next, analyze the call links from the fault source node to other microservice nodes. The call link from A to B has a length of 1 and a call frequency of 10 times per minute. The call link from A to C has a length of 1 and a call frequency of 5 times per minute. There are two call links from A to D: A->B->D, with a length of 2 and a call frequency of 10 times per minute * 2 times = 20 times; and A->C->D, with a length of 2 and a call frequency of 5 times per minute * 3 times = 15 times.

[0089] Calculate the impact factor of the service call frequency. For A to B, the impact factor is 10; for A to C, the impact factor is 5; for A to D, since there are two links, the call frequencies of the two links are added together, and the impact factor is 20 + 15 = 35.

[0090] Calculate the call link length impact factor. Assume the preset attenuation function is: when the link length is 1, the impact factor is 1; when the link length is 2, the impact factor is 0.5; and the impact factor decreases by half for every increase in link length. Therefore, the impact factors of the link lengths from A to B and A to C are both 1, and the impact factors of the link lengths from A to D are both 0.5.

[0091] Obtain service importance data for microservice nodes. For example, quantify service importance on a scale of 1 to 5 based on business importance, user impact, and business loss, with higher levels indicating greater importance. Suppose B's importance level is 3, C's importance level is 4, and D's importance level is 5. Convert the importance levels to service importance impact factors. For example, if level n corresponds to an impact factor of n*0.2, then B's service importance impact factors are 0.6, C's are 0.8, and D's are 1.

[0092] Analyze the historical failure records of microservice nodes. Extract features such as failure frequency, recovery time, and fault spread. Suppose B's historical failure records show an average of one failure per month, with an average recovery time of one hour and a small fault spread. C's historical failure records show an average of one failure per quarter, with an average recovery time of two hours and a medium fault spread. D's historical failure records show an average of one failure per year, with an average recovery time of four hours and a large fault spread. Based on this information, a historical failure impact factor can be constructed. For example, the frequency, recovery time, and spread of the failure can be quantified and weighted together. Suppose B's historical failure impact factor is 0.2, C's is 0.4, and D's is 0.8.

[0093] The influence factors of inter-service call frequency, call link length, service importance, and historical faults are weighted and combined to obtain the impact weight of the fault source node on other microservice nodes.

[0094] Repeat the above steps until the impact weight of the fault source node on all other microservice nodes is calculated.

[0095] Beneficial effects:

[0096] Accurate fault impact assessment: This method comprehensively considers multiple factors and can more accurately assess the scope and extent of the impact when a microservice node is the source of a fault, avoiding assessment bias caused by relying on a single indicator.

[0097] Effective fault warning: By calculating the impact weight, potential high-risk nodes can be identified, and early warning and preventive measures can be taken in advance to reduce the possibility of faults.

[0098] Optimized resource allocation: Based on the impact weight, resources can be allocated first to ensure the stability of key nodes and improve the reliability of the overall system.

[0099] In an optional embodiment, a graph convolutional neural network is used to model the service dependency graph, and the operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model, including:

[0100] Based on the service call log, the call frequency between microservice nodes is counted; the call frequency is normalized to generate edge weight values; a weighted directed graph structure is constructed, wherein the directed graph structure includes a set of microservice nodes, a set of call relationship edges, and an edge weight matrix; self-loop connections are added to the diagonal of the adjacency matrix of the directed graph structure to generate an asymmetric adjacency matrix;

[0101] Collecting multi-dimensional features of nodes, collecting operating status indicators of each node in the microservice node set, wherein the operating status indicators include average response time, request success rate, and error rate; collecting system resource indicators of each node in the microservice node set, wherein the system resource indicators include CPU usage, memory usage, and network IO; collecting business indicators of each node in the microservice node set, wherein the business indicators include service call volume, business processing delay, and business success rate; calculating the historical mean and standard deviation of the operating status indicator, the system resource indicator, and the business indicator; and normalizing the operating status indicator, the system resource indicator, and the business indicator based on the historical mean and the standard deviation to generate an initial node feature matrix;

[0102] Constructing a graph convolutional feature extraction network, inputting the asymmetric adjacency matrix and the initial node feature matrix into a first graph convolutional layer; calculating the first layer of node features through the first graph convolutional layer; fusing the first layer of node features with the original features through a residual connection; inputting the fused features into a second graph convolutional layer; introducing an attention mechanism into the second graph convolutional layer, performing weighted aggregation on neighbor node features based on the edge weight values; and outputting the final node representation features;

[0103] A multi-task prediction model is constructed, and the node representation features are input into a multi-layer perceptron; in a first branch of the multi-layer perceptron, a fault state classifier is constructed based on the node representation features; in a second branch of the multi-layer perceptron, a similarity matrix of the node representation features is calculated; a fault propagation prediction model is constructed based on the similarity matrix; and the outputs of the fault state classifier and the fault propagation prediction model are combined to generate a final prediction result.

[0104] The microservice system fault propagation prediction method includes the following steps:

[0105] First, construct a microservice dependency graph. Extract the call relationships and call counts between microservices from system logs. For example, service A calls service B 100 times, service B calls service C 50 times, and service A calls service C 20 times. Normalize the call counts to generate edge weights. For example, the weight of service A calling service B is 100 / (100+20)=0.83, and the weight of service A calling service C is 20 / (100+20)=0.17. Similarly, construct a weighted directed graph. This directed graph consists of the set of microservice nodes (A, B, C), the set of call relationship edges (A->B, B->C, A->C), and an edge weight matrix (containing weights such as 0.83, 0.17, and 0.5). To prevent information loss, add self-loops to the diagonal of the adjacency matrix, connecting each node to itself with a weight of 1. This ultimately generates an asymmetric adjacency matrix.

[0106] Next, collect and process node features. Collect the operating status indicators of the microservice nodes (for example, the average response time of service A is 0.5 seconds, the request success rate is 99%, and the error rate is 1%), system resource indicators (for example, the CPU utilization rate of service A is 70%, the memory utilization rate is 80%, and the network IO is 10Mbps) and business indicators (for example, the service call volume of service A is 1000 times / minute, the business processing delay is 0.2 seconds, and the business success rate is 98%). Calculate the historical mean and standard deviation for each indicator. For example, the historical mean of the average response time of service A is 0.4 seconds, and the standard deviation is 0.1 seconds. Use the Z-score standardization method to standardize the collected indicators. For example, the standardized value of the average response time of service A is (0.5-0.4) / 0.1=1. All standardized indicators form the initial node feature matrix.

[0107] Next, a graph convolutional feature extraction network is constructed. The asymmetric adjacency matrix and the initial node feature matrix are input into the first graph convolutional layer. The first graph convolutional layer performs weighted aggregation on the neighbor node features of each node to generate the first-layer node features. The first-layer node features are added to the initial node features via a residual connection, and the fused features are input into the second graph convolutional layer. The second graph convolutional layer introduces an attention mechanism to perform weighted aggregation on the neighbor node features based on the edge weights. For example, if the features of service A's neighbor nodes B and C are [0.8, 0.2, 0.5] and [0.1, 0.9, 0.3], respectively, and the edge weights are 0.83 and 0.17, respectively, then the second-layer features of service A are 0.83*[0.8, 0.2, 0.5]+0.17*[0.1, 0.9, 0.3]. The final output is a matrix containing the final representation features of each node.

[0108] Finally, a multi-task prediction model is constructed. The node representation features are input into a multilayer perceptron. The first branch of the multilayer perceptron constructs a fault status classifier to predict the failure probability of each node. For example, the failure probability of service A is 0.1. The second branch of the multilayer perceptron calculates the similarity matrix of the node representation features. For example, the similarity between service A and service B is 0.9. Based on the similarity matrix, a fault propagation prediction model is constructed to predict the probability of fault propagation between services. For example, the probability of a fault in service A propagating to service B is 0.8. The outputs of the fault status classifier and the fault propagation prediction model are combined to generate the final prediction result, for example, predicting that the probability of service B failing within the next hour is 0.08.

[0109] The beneficial effects of this method are reflected in the following three aspects:

[0110] 1. Improve prediction accuracy: By capturing the dependencies between microservices through graph convolutional networks and combining them with multi-dimensional features for modeling, fault status and propagation paths can be more accurately predicted.

[0111] 2. Reduce false alarm rate: Through the attention mechanism, the model can pay more attention to important neighboring nodes and features, thereby reducing interference with irrelevant information and reducing the false alarm rate.

[0112] 3. Enhanced explainability: The model's output includes failure probability and propagation path, which can help operation and maintenance personnel better understand the cause and impact of failures, thereby conducting more targeted troubleshooting and resolution.

[0113] In an optional embodiment, the operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as a fault source node. The propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact levels are output, including:

[0114] Collect microservice operation data and deploy monitoring probes to collect operation status indicators of microservice nodes, including response time, request success rate, and error rate; collect system resource indicators of microservice nodes, including CPU utilization, memory utilization, and network IO; collect business indicators of microservice nodes, including service call volume, business processing delay, and business success rate; use a sliding time window to aggregate the operation status indicators, system resource indicators, and business indicators, with a window size of 5 minutes and a step size of 1 minute; calculate the statistical characteristics of each indicator within the sliding time window to generate a real-time monitoring feature matrix;

[0115] Constructing an anomaly detection baseline, constructing a Gaussian distribution model for continuous indicators in the real-time monitoring feature matrix; constructing a frequency distribution model for discrete indicators in the real-time monitoring feature matrix; combining the Gaussian distribution model and the frequency distribution model to form a normal operation baseline; calculating a deviation measure between the real-time monitoring feature matrix and the normal operation baseline; performing a weighted combination of the deviation measures to obtain an anomaly score; and identifying a fault source node based on the anomaly score;

[0116] Based on the fault propagation model, the anomaly score of the fault source node and the real-time monitoring feature matrix are combined to form an initial node state vector; the initial node state vector is input into a pre-trained graph convolutional neural network; the node state is updated through propagation through the graph convolution layer; the influence strength between nodes is calculated based on the updated node state; and a fault impact propagation matrix is ​​generated;

[0117] Calculate the impact range, calculate the topological impact score between nodes based on the fault impact propagation matrix; calculate the dependency impact score between nodes based on the service call relationship; fuse historical fault data to calculate the vulnerability impact score of the node; perform weighted fusion of the topological impact score, the dependency impact score, and the vulnerability impact score; and generate a comprehensive risk score for the node;

[0118] Output prediction results, classify microservice nodes into risk levels based on the comprehensive risk score; identify high-risk nodes as key focus points; calculate the expected impact time of each node; generate a prediction result report including risk level and impact time; output the prediction result report to guide fault emergency response.

[0119] To predict the impact of microservice fault propagation in distributed systems in real time, this implementation provides a fault propagation prediction method based on a graph convolutional neural network. This method collects operational status data from microservice nodes, builds an anomaly detection baseline, identifies fault source nodes, and uses a graph convolutional neural network to predict the fault propagation path and impact range. Ultimately, it outputs a list of affected service nodes and their impact level.

[0120] First, deploy monitoring probes to collect operational status metrics from microservice nodes. These metrics, such as response time, request success rate, and error rate, reflect the performance and stability of microservices. Simultaneously, collect system resource metrics, such as CPU utilization, memory utilization, and network I / O, as well as business metrics, such as service call volume, business processing latency, and business success rate. These metrics depict the operational status of microservices from different dimensions.

[0121] Collected metric data is aggregated using a sliding time window. For example, setting the sliding time window size to 5 minutes with a step size of 1 minute means that statistical features of the data from the past 5 minutes are calculated every minute. This approach allows for capturing short-term trends in metric data. For example, if the response times of a microservice node over the past 5 minutes were 100ms, 120ms, 150ms, 180ms, and 200ms, respectively, statistical features such as the average and standard deviation can be calculated.

[0122] Next, we build an anomaly detection baseline. For continuous metrics, such as response time, we can construct a Gaussian distribution model. By analyzing historical data, we can obtain the metric's mean and standard deviation, thereby determining the parameters of the Gaussian distribution. For discrete metrics, such as request success rate, we can construct a frequency distribution model. By counting the number of occurrences of each value, we can obtain the frequency distribution. Combining the Gaussian and frequency distribution models creates a normal operation baseline.

[0123] The real-time monitoring feature matrix is ​​compared with the normal operating baseline to calculate a deviation metric. For example, the difference between the real-time metric value and the baseline average value can be calculated, or the probability of the real-time metric value falling outside the baseline distribution can be calculated. Different deviation metric values ​​are weighted and combined to generate an anomaly score. For example, different weights can be assigned based on the importance of the metric. Based on the anomaly score, the fault source node is identified. For example, a threshold can be set, and nodes exceeding the threshold are identified as fault sources.

[0124] The anomaly score of the fault source node and the real-time monitoring feature matrix are combined to form an initial node state vector. This vector is then fed into a pre-trained graph convolutional neural network. The graph convolutional neural network can learn the dependencies between services and simulate the propagation of faults in the dependency graph. The node state is updated through the graph convolutional layer. The node state vector contains information about the node's anomaly and the impact of neighboring nodes.

[0125] Based on the updated node states, the impact strength between nodes is calculated. For example, the similarity or distance between node state vectors can be calculated. This generates a fault impact propagation matrix, where each element represents the impact strength between nodes.

[0126] Based on the fault impact propagation matrix, the topology impact scores between nodes are calculated. For example, the influence of a node can be calculated based on its position and connection relationship in the dependency graph. The dependency impact scores between nodes are calculated based on the service call relationship. For example, if node A frequently calls node B, node B has a greater impact on node A. The vulnerability impact score of the node is calculated by integrating historical fault data. For example, if node C has a history of frequent faults, its vulnerability is high. The topology impact score, dependency impact score, and vulnerability impact score are weighted and integrated to generate a comprehensive risk score for the node.

[0127] Microservice nodes are classified into risk levels based on their comprehensive risk scores. For example, nodes can be categorized as high risk, medium risk, and low risk. High-risk nodes are identified as key targets for attention. The estimated impact time for each node is calculated. For example, the impact time can be estimated based on the fault propagation speed and the node's risk level. A prediction report is generated that includes the risk level and impact time. The prediction report is used to guide emergency response. For example, based on the prediction results, proactive measures can be taken to prevent the fault from escalating.

[0128] Beneficial effects:

[0129] 1. Improved fault prediction accuracy: By combining real-time monitoring data, service dependencies, and historical fault data, we can more accurately predict the fault propagation path and impact range. For example, we can identify potentially affected nodes even if they haven't yet shown obvious anomalies.

[0130] 2. Shorten fault response time: By predicting the impact of a fault in advance, you can shorten the response time and reduce the losses caused by the fault. For example, you can prepare an emergency plan in advance and implement it quickly when a fault occurs.

[0131] 3. Improve system stability: By identifying high-risk nodes and potential failure points, we can guide system optimization and improvement, improving overall system stability. For example, we can allocate resources or optimize code for high-risk nodes to reduce the probability of failure.

[0132] In an optional implementation, the affected service nodes are classified into risk levels according to the degree of impact, and alarm information of corresponding levels is triggered according to the preset risk threshold. At the same time, the fault propagation path is visualized, including:

[0133] Collect microservice fault propagation data, including fault occurrence time, fault impact range, fault propagation path, fault propagation speed, and fault duration;

[0134] Extracting impact characteristics based on the microservice fault propagation data, including calculating the probability of fault propagation between nodes, extracting fault time delay characteristics, and statistically analyzing the overlap of fault impact ranges to construct a fault impact feature matrix; performing feature dimensionality reduction and normalization on the fault impact feature matrix, extracting key influencing factors using principal component analysis, and calculating the weight coefficient of each influencing factor;

[0135] Calculating a comprehensive impact score for each affected service node based on the key impact factors and the weight coefficients, and dividing the comprehensive impact scores into different risk levels according to a preset risk level classification rule; setting corresponding risk thresholds for different risk levels, and triggering an alarm message of a corresponding level when the comprehensive impact score of a service node exceeds the risk threshold of the corresponding risk level;

[0136] Construct an alarm information template, which includes the fault impact range, fault propagation path, risk level determination basis, and processing suggestions, and generate standardized alarm information based on the alarm information template; use a graphical method to display the fault propagation path, including constructing a service dependency diagram, and marking the fault propagation direction, propagation speed, and impact range in the service dependency diagram.

[0137] Microservice fault propagation risk assessment and visual alarm method, the specific implementation is as follows:

[0138] First, collect microservice fault propagation data. Using distributed tracing systems and log collection tools, collect various data from the microservice system in real time. This data includes the time of fault occurrence, accurate to the millisecond; the scope of the fault, recording the names and number of affected microservices; the fault propagation path, tracing the call chain between microservices; the fault propagation speed, calculating the time interval for the fault to propagate from one microservice to the next; and the fault duration, recording the length of time from fault occurrence to system recovery. For example, at 10:00:00 on January 1, 2024, microservice A experienced a fault, affecting three microservices. The propagation path was A->B->C, with propagation speeds of 10ms and 20ms, respectively, and the fault duration was one minute.

[0139] Next, we extract impact characteristics based on the collected microservice fault propagation data. We calculate the inter-node fault propagation probability and count the probability of a microservice failure causing other microservices to fail, as seen in historical fault data. We also extract fault time delay characteristics and calculate the time interval for a fault to propagate from one microservice to the next. We also calculate the overlap of fault impact ranges and the proportion of the intersection of the impact ranges of different microservice faults to the overall impact range. These characteristics are then constructed into a fault impact feature matrix. For example, if microservice A fails, there's an 80% chance that microservice B will fail, and a 50% chance that microservice C will fail. The propagation delay from A to B is 10ms, and the propagation delay from B to C is 20ms. The overlap between the impact ranges of microservices A and B is 30%.

[0140] The fault impact feature matrix was then subjected to feature dimensionality reduction and normalization. Principal component analysis was used to extract key influencing factors and calculate the weight coefficient for each factor. Key influencing factors are the feature combinations that best reflect the fault impact. The weight coefficient represents the contribution of each key influencing factor to the overall impact score. For example, principal component analysis extracted two key influencing factors: fault propagation probability and overlap of fault impact ranges, with weight coefficients of 0.7 and 0.3, respectively.

[0141] Calculate the comprehensive impact score for each affected service node based on the key impact factors and weight coefficients. Multiply each key impact factor by its corresponding weight coefficient, and then add all the results together to obtain the comprehensive impact score for each affected service node. For example, if microservice B has an 80% fault propagation probability and a 30% overlap in fault impact ranges, its comprehensive impact score is 0.7*0.8+0.3*0.3=0.65.

[0142] The comprehensive impact score is divided into different risk levels according to the preset risk level classification rules. For example, the comprehensive impact score is divided into three risk levels: low, medium, and high, with corresponding score ranges of 0-0.3, 0.3-0.7, and 0.7-1 respectively.

[0143] Set corresponding risk thresholds for different risk levels. When the comprehensive impact score of a service node exceeds the risk threshold of the corresponding risk level, an alarm message of the corresponding level is triggered. For example, the threshold for low risk level is 0.2, the threshold for medium risk level is 0.5, and the threshold for high risk level is 0.8.

[0144] Build an alarm information template that includes the fault impact scope, fault propagation path, risk level determination basis, and handling recommendations. Generate standardized alarm information based on the alarm information template. For example, if microservice B's comprehensive impact score is 0.65, exceeding the medium risk threshold of 0.5, a medium-level alarm will be triggered. The alarm information includes: Affected microservices: B and C; Fault propagation path: A->B->C; Risk level: Medium; Determination basis: Comprehensive impact score of 0.65 exceeds the threshold of 0.5; Handling recommendation: Check the logs of microservices B and C to identify the cause of the fault.

[0145] Use a graphical approach to display fault propagation paths. Build a service dependency diagram and annotate the fault propagation direction, speed, and impact range. For example, use arrows to indicate fault propagation direction, numbers on the arrows to indicate propagation speed, and different colors to indicate the impact range of different risk levels.

[0146] Beneficial effects:

[0147] 1. Improve fault response efficiency: Through real-time monitoring and alarms, faults can be discovered and handled promptly, shortening fault recovery time and avoiding business interruptions.

[0148] 2. Reduce the scope of fault impact: By analyzing the fault propagation path and impact range, the root cause of the fault can be quickly located, effective measures can be taken to prevent the fault from spreading, and the number of affected service nodes can be reduced.

[0149] 3. Optimize system stability: By analyzing and accumulating fault data, we can identify weak links in the system and perform targeted optimization to improve the overall stability and reliability of the system.

[0150] Figure 2 FIG. 1 is a structural diagram of a distributed system fault location and diagnosis system based on log analysis according to an embodiment of the present invention. Figure 2 As shown, the system includes:

[0151] The first unit is used to obtain information about all microservice nodes through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters, and call status; based on the structured stored call data, a service dependency graph is constructed using a graph database, where graph nodes represent microservice nodes, graph edges represent call relationships between services, and call frequency, average response time, and success rate are annotated on the graph edges;

[0152] The second unit is used to calculate, for each microservice node, the impact weight of the node on other microservice nodes when it is a fault source based on the service dependency graph. The impact weight is calculated based on the call frequency between services, the call link length, the service importance, and historical fault data. The service dependency graph is modeled using a graph convolutional neural network. The operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model.

[0153] The third unit is used to collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact level are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

[0154] According to a third aspect of the embodiments of the present invention,

[0155] An electronic device is provided, comprising:

[0156] processor;

[0157] a memory for storing processor-executable instructions;

[0158] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0159] According to a fourth aspect of the embodiments of the present invention,

[0160] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0161] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed system fault location and diagnosis method based on log analysis, characterized in that: include: Obtain all microservice node information through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service ID, callee service ID, call interface name, call parameters and call status; Based on the structured stored call data, a graph database is used to build a service dependency graph, where graph nodes represent microservice nodes and graph edges represent the call relationships between services. The call frequency, average response time, and success rate are annotated on the graph edges. Based on the service dependency graph, for each microservice node, the impact weight of the node on other microservice nodes when it is a fault source is calculated. The impact weight is calculated based on the call frequency between services, the call link length, the importance of the service, and historical fault data. The service dependency graph is modeled using a graph convolutional neural network. The operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs. The fault propagation prediction model is constructed, specifically including the following steps: A multi-task prediction model is constructed, and node representation features are input into a multi-layer perceptron; in a first branch of the multi-layer perceptron, a fault state classifier is constructed based on the node representation features; in a second branch of the multi-layer perceptron, a similarity matrix of the node representation features is calculated; Constructing a fault propagation prediction model based on the similarity matrix; combining the output of the fault state classifier and the fault propagation prediction model to generate a final prediction result; The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node. The propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact level are output. The affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

2. The method according to claim 1, characterized in that Obtain all microservice node information through the service registration center in the distributed system, collect call data between microservice nodes based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service ID, callee service ID, call interface name, call parameters, and call status, including: Obtain registration information of all microservice nodes through the service registration center in the distributed system. The registration information includes service name, instance identifier, network address, port number, and health check address. The service registration center stores the registration information in the form of temporary nodes and maintains the survival status of the microservice nodes through a heartbeat mechanism; The application program interface call link tracker collects call data between each microservice node. The application program interface call link tracker injects tracing logic into the remote call entry point of the microservice node, generates a globally unique tracing identifier and call identifier, and propagates the tracing identifier and the call identifier in the call link; the application program interface call link tracker records the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status; The call data is sent to a message queue for temporary storage. A dedicated consumer cluster obtains the call data from the message queue and performs sharding processing. The sharding processing divides the call data into multiple data shards according to the service dimension; the sharded call data is written into a distributed time series database for structured storage.

3. The method according to claim 1, characterized in that Based on the service dependency graph, for each microservice node, calculate its impact weight on other microservice nodes when it is a fault source. The impact weight is calculated based on the call frequency between services, call link length, service importance, and historical fault data, including: Based on the service dependency graph, the microservice node to be analyzed is selected as the fault source node, and the following steps are performed to calculate the impact weight of the fault source node on other microservice nodes: A graph traversal algorithm is used to analyze the call link from the fault source node to each of the microservice nodes, extract the call frequency data on the call link, and multiply the call frequencies on the call link in sequence to obtain an inter-service call frequency impact factor; Determining the call link length based on the number of node hops of the call link, and calculating a call link length impact factor according to a preset decay function, wherein the call link length impact factor decreases as the number of node hops of the call link increases; Obtaining service importance data of the microservice node, quantifying the service importance data based on business importance, user impact range, and business loss degree, and obtaining a service importance impact factor; Analyze the historical fault records of the microservice node, extract the fault frequency, fault recovery time, and fault spread characteristics, and construct the historical fault impact factor; Perform a weighted combination of the inter-service call frequency impact factor, the call link length impact factor, the service importance impact factor, and the historical fault impact factor to obtain the impact weight of the fault source node on the current microservice node; Repeat the above steps until the influence weight of the fault source node on all other microservice nodes in the service dependency graph is calculated.

4. The method according to claim 1, wherein A graph convolutional neural network is used to model the service dependency graph. The operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model, including: Based on the service call log, the call frequency between microservice nodes is counted; the call frequency is normalized to generate edge weight values; a weighted directed graph structure is constructed, wherein the directed graph structure includes a set of microservice nodes, a set of call relationship edges, and an edge weight matrix; self-loop connections are added to the diagonal of the adjacency matrix of the directed graph structure to generate an asymmetric adjacency matrix; Collecting multi-dimensional features of nodes, collecting operating status indicators of each node in the microservice node set, wherein the operating status indicators include average response time, request success rate, and error rate; collecting system resource indicators of each node in the microservice node set, wherein the system resource indicators include CPU usage, memory usage, and network IO; collecting business indicators of each node in the microservice node set, wherein the business indicators include service call volume, business processing delay, and business success rate; calculating the historical mean and standard deviation of the operating status indicator, the system resource indicator, and the business indicator; and normalizing the operating status indicator, the system resource indicator, and the business indicator based on the historical mean and the standard deviation to generate an initial node feature matrix; Constructing a graph convolutional feature extraction network, inputting the asymmetric adjacency matrix and the initial node feature matrix into a first graph convolutional layer; calculating the first layer of node features through the first graph convolutional layer; fusing the first layer of node features with the original features through a residual connection; inputting the fused features into a second graph convolutional layer; introducing an attention mechanism into the second graph convolutional layer, performing weighted aggregation on neighbor node features based on the edge weight values; and outputting the final node representation features; A multi-task prediction model is constructed, and the node representation features are input into a multi-layer perceptron; in a first branch of the multi-layer perceptron, a fault state classifier is constructed based on the node representation features; in a second branch of the multi-layer perceptron, a similarity matrix of the node representation features is calculated; a fault propagation prediction model is constructed based on the similarity matrix; and the outputs of the fault state classifier and the fault propagation prediction model are combined to generate a final prediction result.

5. The method according to claim 1, wherein The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node. The propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network. The list of affected service nodes and their impact levels are output, including: Collect microservice operation data and deploy monitoring probes to collect operation status indicators of microservice nodes, including response time, request success rate, and error rate; collect system resource indicators of microservice nodes, including CPU utilization, memory utilization, and network IO; collect business indicators of microservice nodes, including service call volume, business processing delay, and business success rate; use a sliding time window to aggregate the operation status indicators, system resource indicators, and business indicators, with a window size of 5 minutes and a step size of 1 minute; calculate the statistical characteristics of each indicator within the sliding time window to generate a real-time monitoring feature matrix; Constructing an anomaly detection baseline, constructing a Gaussian distribution model for continuous indicators in the real-time monitoring feature matrix; constructing a frequency distribution model for discrete indicators in the real-time monitoring feature matrix; combining the Gaussian distribution model and the frequency distribution model to form a normal operation baseline; calculating a deviation measure between the real-time monitoring feature matrix and the normal operation baseline; performing a weighted combination of the deviation measures to obtain an anomaly score; and identifying a fault source node based on the anomaly score; Based on the fault propagation model, the anomaly score of the fault source node and the real-time monitoring feature matrix are combined to form an initial node state vector; the initial node state vector is input into a pre-trained graph convolutional neural network; the node state is updated through propagation through the graph convolution layer; the influence strength between nodes is calculated based on the updated node state; and a fault impact propagation matrix is ​​generated; Calculate the impact range, calculate the topological impact score between nodes based on the fault impact propagation matrix; calculate the dependency impact score between nodes based on the service call relationship; fuse historical fault data to calculate the vulnerability impact score of the node; perform weighted fusion of the topological impact score, the dependency impact score, and the vulnerability impact score; and generate a comprehensive risk score for the node; Output prediction results, classify microservice nodes into risk levels based on the comprehensive risk score; identify high-risk nodes as key focus points; calculate the expected impact time of each node; generate a prediction result report including risk level and impact time; output the prediction result report to guide fault emergency response.

6. The method according to claim 1, characterized in that The affected service nodes are classified into risk levels based on the degree of impact, and alarm information of corresponding levels is triggered according to the preset risk threshold. At the same time, the fault propagation path is visualized, including: Collect microservice fault propagation data, including fault occurrence time, fault impact range, fault propagation path, fault propagation speed, and fault duration; Extracting impact characteristics based on the microservice fault propagation data, including calculating the probability of fault propagation between nodes, extracting fault time delay characteristics, and statistically analyzing the overlap of fault impact ranges to construct a fault impact feature matrix; performing feature dimensionality reduction and normalization on the fault impact feature matrix, extracting key influencing factors using principal component analysis, and calculating the weight coefficient of each influencing factor; Calculating a comprehensive impact score for each affected service node based on the key impact factors and the weight coefficients, and dividing the comprehensive impact scores into different risk levels according to a preset risk level classification rule; setting corresponding risk thresholds for different risk levels, and triggering an alarm message of a corresponding level when the comprehensive impact score of a service node exceeds the risk threshold of the corresponding risk level; Construct an alarm information template, which includes the fault impact range, fault propagation path, risk level determination basis, and processing suggestions, and generate standardized alarm information based on the alarm information template; use a graphical method to display the fault propagation path, including constructing a service dependency diagram, and marking the fault propagation direction, propagation speed, and impact range in the service dependency diagram.

7. A distributed system fault location and diagnosis system based on log analysis, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to obtain all microservice node information through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status; Based on the structured stored call data, a graph database is used to build a service dependency graph, where graph nodes represent microservice nodes and graph edges represent the call relationships between services. The call frequency, average response time, and success rate are annotated on the graph edges. The second unit is used to calculate, for each microservice node, the impact weight of the node on other microservice nodes when it is a fault source based on the service dependency graph. The impact weight is calculated based on the call frequency between services, the call link length, the service importance, and historical fault data. The service dependency graph is modeled using a graph convolutional neural network. The operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model. The third unit is used to collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact level are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images