Distributed system fault positioning diagnosis method and system based on log analysis

Abstract

The invention provides a distributed system fault positioning diagnosis method and system based on log analysis, and relates to the technical field of fault diagnosis, and the method comprises the steps: collecting call data between micro-service nodes, constructing a service dependency graph, and calculating the influence weight of each node as a fault source; and modeling the service dependency relationship graph by using a graph convolutional neural network, and constructing a fault propagation prediction model. And when an abnormal node is detected, inputting the node into the prediction model, calculating a fault propagation path and an influence range, outputting an influenced service node list and an influence degree thereof, carrying out risk grade division and alarm, and meanwhile, visually displaying the fault propagation path. According to the method, the fault source can be quickly and accurately positioned, the fault propagation influence is predicted, and the reliability and stability of the distributed system are improved.

Description

Technical Field

[0001] The present invention relates to fault diagnosis technology, and in particular to a distributed system fault location diagnosis method and system based on log analysis. Background Art

[0002] With the popularity of distributed systems and microservice architectures, system complexity continues to increase, and fault location and diagnosis are becoming increasingly difficult. Traditional troubleshooting methods usually rely on manual experience and log analysis, which are inefficient and difficult to deal with complex fault scenarios. Especially in a microservice architecture, the call relationship between services is intricate, and the failure of a service may spread quickly along the call link, causing a large range of services to be paralyzed. Therefore, it is crucial to quickly and accurately locate the source of the fault and predict the scope of the fault.

[0003] First, the traditional log analysis method requires manual analysis of massive log data line by line, which is time-consuming, labor-intensive and error-prone. In the microservice architecture, logs are scattered across different service nodes, further increasing the difficulty of log analysis.

[0004] Secondly, some fault diagnosis methods based on monitoring indicators can only identify abnormal service nodes, but cannot determine the root cause and propagation path of the fault. This makes it difficult for operation and maintenance personnel to quickly locate the source of the fault and take effective repair measures.

[0005] Finally, most existing fault prediction models are based on simple rules or statistical methods, lacking a deep understanding of service dependencies and fault propagation patterns. This results in low prediction accuracy and makes it difficult to effectively predict the impact scope and potential risks of faults. Summary of the invention

[0006] The embodiments of the present invention provide a distributed system fault location and diagnosis method and system based on log analysis, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a distributed system fault location and diagnosis method based on log analysis, including:

[0009] Obtain all microservice node information through the service registration center in the distributed system, collect call data between microservice nodes based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service ID, callee service ID, call interface name, call parameters, and call status; based on the structured stored call data, use a graph database to build a service dependency graph, in which graph nodes represent microservice nodes, graph edges represent call relationships between services, and annotate the call frequency, average response time, and success rate on the graph edges;

[0010] Based on the service dependency graph, for each microservice node, the influence weight on other microservice nodes when it is a fault source is calculated, and the influence weight is calculated comprehensively based on the call frequency between services, the call link length, the service importance and the historical fault data; the service dependency graph is modeled using a graph convolutional neural network, and the operating status indicators, system resource usage and business indicators of each microservice node are used as node feature inputs to build a fault propagation prediction model;

[0011] The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as a fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact levels are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

[0012] Obtain all microservice node information through the service registration center in the distributed system, collect call data between microservice nodes based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status, including:

[0013] The registration information of all microservice nodes is obtained through the service registration center in the distributed system. The registration information includes the service name, instance identifier, network address, port number and health check address. The service registration center stores the registration information in the form of a temporary node and maintains the survival status of the microservice node through a heartbeat mechanism;

[0014] The call data between each microservice node is collected through the application program interface call link tracker, and the application program interface call link tracker injects the tracking logic into the remote call entry point of the microservice node, generates a globally unique tracking identifier and a call identifier, and propagates the tracking identifier and the call identifier in the call link; the application program interface call link tracker records the call timestamp, the caller service identifier, the callee service identifier, the call interface name, the call parameters and the call status;

[0015] The call data is sent to a message queue for temporary storage, and a dedicated consumer cluster obtains the call data from the message queue and performs sharding processing, wherein the sharding processing divides the call data into multiple data shards according to the service dimension; the sharded call data is written into a distributed time series database for structured storage.

[0016] Based on the service dependency graph, for each microservice node, the impact weight on other microservice nodes when it is a fault source is calculated. The impact weight is calculated based on the call frequency between services, the call link length, the importance of the service, and the historical fault data, including:

[0017] Based on the service dependency graph, the microservice node to be analyzed is selected as the fault source node, and the following steps are performed to calculate the impact weight of the fault source node on other microservice nodes:

[0018] A graph traversal algorithm is used to analyze the call link from the fault source node to each of the microservice nodes, the call frequency data on the call link is extracted, and the call frequencies on the call link are multiplied in sequence to obtain the service-to-service call frequency impact factor;

[0019] Determine the call link length based on the number of node hops of the call link, and calculate the call link length impact factor according to a preset attenuation function, wherein the call link length impact factor decreases as the number of node hops of the call link increases;

[0020] Obtaining service importance data of the microservice node, quantifying the service importance data based on business importance, user impact scope, and business loss degree, and obtaining a service importance impact factor;

[0021] Analyze the historical fault records of the microservice nodes, extract the fault frequency, fault recovery time, and fault spread characteristics, and construct historical fault impact factors;

[0022] The influence factor of the service call frequency, the influence factor of the call link length, the influence factor of the service importance and the influence factor of the historical fault are weighted and combined to obtain the influence weight of the fault source node on the current microservice node;

[0023] The above steps are repeated until the influence weight of the fault source node on all other microservice nodes in the service dependency graph is calculated.

[0024] The service dependency graph is modeled using a graph convolutional neural network, and the operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to build a fault propagation prediction model, including:

[0025] Based on the service call log, the call frequency between microservice nodes is counted; the call frequency is normalized to generate edge weight values; a weighted directed graph structure is constructed, wherein the directed graph structure includes a microservice node set, a call relationship edge set, and an edge weight matrix; self-loop connections are added to the diagonal of the adjacency matrix of the directed graph structure to generate an asymmetric adjacency matrix;

[0026] Collect node multi-dimensional features, collect the operating status indicators of each node in the microservice node set, the operating status indicators include average response time, request success rate and error rate; collect system resource indicators of each node in the microservice node set, the system resource indicators include CPU usage, memory usage and network IO; collect business indicators of each node in the microservice node set, the business indicators include service call volume, business processing delay and business success rate; calculate the historical mean and standard deviation of the operating status indicator, the system resource indicator and the business indicator; standardize the operating status indicator, the system resource indicator and the business indicator based on the historical mean and the standard deviation to generate an initial node feature matrix;

[0027] Constructing a graph convolutional feature extraction network, inputting the asymmetric adjacency matrix and the initial node feature matrix into the first graph convolutional layer; calculating the first layer node features through the first graph convolutional layer; fusing the first layer node features with the original features through residual connection; inputting the fused features into the second graph convolutional layer; introducing an attention mechanism into the second graph convolutional layer, performing weighted aggregation on neighbor node features based on the edge weight values; and outputting the final node representation features;

[0028] A multi-task prediction model is constructed, and the node representation features are input into a multi-layer perceptron; in a first branch of the multi-layer perceptron, a fault state classifier is constructed based on the node representation features; in a second branch of the multi-layer perceptron, a similarity matrix of the node representation features is calculated; a fault propagation prediction model is constructed based on the similarity matrix; and the outputs of the fault state classifier and the fault propagation prediction model are combined to generate a final prediction result.

[0029] The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as a fault source node. The propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and the list of affected service nodes and their impact levels are output, including:

[0030] Collect microservice operation data, deploy monitoring probes to collect operation status indicators of microservice nodes, the operation status indicators include response time, request success rate, and error rate; collect system resource indicators of the microservice nodes, the system resource indicators include CPU usage, memory usage, and network IO; collect business indicators of the microservice nodes, the business indicators include service call volume, business processing delay, and business success rate; use a sliding time window to aggregate the operation status indicators, the system resource indicators, and the business indicators, the window size of the sliding time window is 5 minutes, and the step size is 1 minute; calculate the statistical characteristics of each indicator in the sliding time window, and generate a real-time monitoring feature matrix;

[0031] Construct an anomaly detection baseline, construct a Gaussian distribution model for the continuous indicators in the real-time monitoring feature matrix; construct a frequency distribution model for the discrete indicators in the real-time monitoring feature matrix; combine the Gaussian distribution model and the frequency distribution model to form a normal operation baseline; calculate the deviation measurement value between the real-time monitoring feature matrix and the normal operation baseline; perform weighted combination of the deviation measurement values ​​to obtain an anomaly score; identify the fault source node based on the anomaly score;

[0032] Based on the fault propagation model, the abnormal score of the fault source node and the real-time monitoring feature matrix are combined to form an initial node state vector; the initial node state vector is input into a pre-trained graph convolutional neural network; the node state is updated through the graph convolutional layer propagation; the influence strength between nodes is calculated based on the updated node state; and a fault impact propagation matrix is ​​generated;

[0033] Calculate the impact range, calculate the topological impact score between nodes based on the fault impact propagation matrix; calculate the dependency impact score between nodes in combination with the service call relationship; fuse the historical fault data to calculate the vulnerability impact score of the node; perform weighted fusion on the topological impact score, the dependency impact score and the vulnerability impact score; generate a comprehensive risk score for the node;

[0034] Output the prediction results, classify the risk levels of microservice nodes based on the comprehensive risk scores; identify high-risk nodes as key focus objects; calculate the estimated impact time of each node; generate a prediction result report including risk level and impact time; output the prediction result report to guide fault emergency response.

[0035] The affected service nodes are classified into risk levels according to the degree of impact, and the corresponding level of alarm information is triggered according to the preset risk threshold. At the same time, the fault propagation path is visualized, including:

[0036] Collecting microservice fault propagation data, wherein the microservice fault propagation data includes fault occurrence time, fault impact scope, fault propagation path, fault propagation speed, and fault duration;

[0037] Extracting the impact degree characteristics based on the microservice fault propagation data, including calculating the fault propagation probability between nodes, extracting the fault time delay characteristics, and counting the overlap of the fault impact range, and constructing a fault impact feature matrix; performing feature dimension reduction and normalization processing on the fault impact feature matrix, extracting key influencing factors using the principal component analysis method, and calculating the weight coefficient of each influencing factor;

[0038] Calculate the comprehensive impact score of each affected service node based on the key impact factor and the weight coefficient, and divide the comprehensive impact score into different risk levels according to the preset risk level classification rule; set corresponding risk thresholds for different risk levels, and when the comprehensive impact score of the service node exceeds the risk threshold of the corresponding risk level, trigger the corresponding level of alarm information;

[0039] Construct an alarm information template, which includes the fault impact scope, fault propagation path, risk level determination basis, and processing suggestions, and generate standardized alarm information based on the alarm information template; use a graphical method to display the fault propagation path, including constructing a service dependency diagram, and marking the fault propagation direction, propagation speed, and impact scope in the service dependency diagram.

[0040] According to a second aspect of the embodiments of the present invention,

[0041] Provides a distributed system fault location and diagnosis system based on log analysis, including:

[0042] The first unit is used to obtain all microservice node information through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters, and call status; based on the structured stored call data, a service dependency graph is constructed using a graph database, in which graph nodes represent microservice nodes, graph edges represent call relationships between services, and call frequency, average response time, and success rate are annotated on the graph edges;

[0043] The second unit is used to calculate the influence weight of each microservice node on other microservice nodes when it is a fault source based on the service dependency graph, and the influence weight is calculated comprehensively according to the call frequency between services, the call link length, the service importance and the historical fault data; the service dependency graph is modeled by using a graph convolutional neural network, and the operation status indicator, system resource usage and business indicator of each microservice node are used as node feature inputs to build a fault propagation prediction model;

[0044] The third unit is used to collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact levels are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

[0045] According to a third aspect of the embodiments of the present invention,

[0046] An electronic device is provided, comprising:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0050] A fourth aspect of the embodiments of the present invention is:

[0051] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0052] The beneficial effects of this application are as follows:

[0053] Fast and accurate fault location: The present invention models the service dependency graph through a graph convolutional neural network and combines it with real-time operating status data to quickly identify the fault source node and predict the fault propagation path and impact range, thereby achieving accurate fault location.

[0054] Effectively reduce the false alarm rate: The present invention comprehensively calculates the impact weight based on multi-dimensional information such as service call frequency, call link length, service importance, and historical fault data, and combines the predictive ability of graph convolutional neural networks to effectively reduce the false alarm rate of faults and improve the accuracy of fault diagnosis.

[0055] Improve operation and maintenance efficiency: The present invention provides a list of affected service nodes and their impact levels, and performs risk level classification and alarms. At the same time, it visualizes the fault propagation path, which can help operation and maintenance personnel quickly understand the fault situation and take countermeasures in time, thereby improving operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a flow chart of a distributed system fault location and diagnosis method based on log analysis according to an embodiment of the present invention;

[0057] Figure 2 The present invention is a structural diagram of a distributed system fault location and diagnosis system based on log analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 1 FIG. 1 is a flow chart of a distributed system fault location and diagnosis method based on log analysis according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0061] S11. Obtain all microservice node information through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters, and call status; based on the structured stored call data, use a graph database to build a service dependency graph, where graph nodes represent microservice nodes, graph edges represent call relationships between services, and annotate the call frequency, average response time, and success rate on the graph edges;

[0062] S12. Based on the service dependency graph, for each microservice node, calculate the impact weight of the microservice node on other microservice nodes when it is a fault source, and the impact weight is calculated based on the call frequency between services, the call link length, the importance of the service, and the historical fault data; use a graph convolutional neural network to model the service dependency graph, take the operating status indicators, system resource usage, and business indicators of each microservice node as node feature inputs, and build a fault propagation prediction model;

[0063] S13. Collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, input the abnormal node as the fault source node into the fault propagation prediction model, calculate the propagation path and impact range of the fault in the service dependency graph based on the graph convolutional neural network, and output a list of affected service nodes and their impact levels; divide the affected service nodes into risk levels according to the impact level, and trigger the corresponding level of alarm information according to the preset risk threshold, and visualize the fault propagation path.

[0064] In an optional implementation, all microservice node information is obtained through a service registration center in a distributed system, call data between microservice nodes is collected based on an application program interface call link tracker, and the call data is structured and stored according to call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status, including:

[0065] The registration information of all microservice nodes is obtained through the service registration center in the distributed system. The registration information includes the service name, instance identifier, network address, port number and health check address. The service registration center stores the registration information in the form of a temporary node and maintains the survival status of the microservice node through a heartbeat mechanism;

[0066] The call data between each microservice node is collected through the application program interface call link tracker, and the application program interface call link tracker injects the tracking logic into the remote call entry point of the microservice node, generates a globally unique tracking identifier and a call identifier, and propagates the tracking identifier and the call identifier in the call link; the application program interface call link tracker records the call timestamp, the caller service identifier, the callee service identifier, the call interface name, the call parameters and the call status;

[0067] The call data is sent to a message queue for temporary storage, and a dedicated consumer cluster obtains the call data from the message queue and performs sharding processing, wherein the sharding processing divides the call data into multiple data shards according to the service dimension; the sharded call data is written into a distributed time series database for structured storage.

[0068] A microservice call data collection and storage method based on a service registration center and a call link tracker is used to collect and store call data between microservices in a distributed system for performance analysis, troubleshooting, and system optimization.

[0069] First, obtain the registration information of all microservice nodes through the service registry in the distributed system. The service registry, such as ZooKeeper, Consul, or Etcd, stores the information of each microservice instance, including the service name, instance identifier (such as UUID), network address (such as IP address), port number, and health check address (such as HTTP interface). This information is stored in the form of temporary nodes, and the survival status of the microservice nodes is maintained through the heartbeat mechanism. The service registry will regularly check the health status of each instance, and if the heartbeat of an instance times out, it will be removed from the registry.

[0070] Next, the call link tracker is called through the application program interface (API) to collect call data between each microservice node. Common call link trackers such as Jaeger, Zipkin or SkyWa l king inject tracing logic into the remote call entry point of the microservice node. When one service calls another service, the tracker generates a globally unique trace ID and call ID (span ID) and propagates these IDs in the call link. For example, a user request triggers service A to call service B, and service B calls service C. The entire call link shares the same trace ID, while the calls of services A, B, and C have different span IDs. The tracker records the timestamp, caller service ID, callee service ID, call interface name, call parameters, and call status (such as success or failure) of each call.

[0071] Then, the collected call data is sent to a message queue for temporary storage, such as Kafka or RabbitMQ. A dedicated consumer cluster obtains the call data from the message queue and performs sharding processing. The sharding process divides the call data into multiple data shards according to the service dimension, such as "order-service" shard, "payment-service" shard, etc.

[0072] Finally, the sharded call data is written to a distributed time series database for structured storage, such as InfluxDB, Prometheus, or OpenTSDB. The time series database can efficiently store and query time series data.

[0073] The beneficial effects of this method are reflected in the following three aspects:

[0074] First, it improves system observability. By collecting and storing call data between microservices, we can clearly understand the call relationship, call times, call duration and other information of each service in the system, so as to better monitor the system operation status and timely discover and solve performance bottlenecks.

[0075] Second, it improves troubleshooting efficiency. When a system fails, the cause of the failure can be quickly located by tracking data in the call link, shortening the troubleshooting time and improving system stability. For example, by analyzing the error information and call duration in the call link, the service and code that caused the failure can be quickly found.

[0076] Third, it supports system performance optimization. By analyzing the call link data, we can identify the performance bottlenecks in the system, such as services that are called too many times or take too long to call, so as to optimize them in a targeted manner and improve the overall performance of the system. For example, we can optimize the code, cache, or database of certain services based on the call data analysis results.

[0077] In an optional implementation, based on the service dependency graph, for each microservice node, the impact weight of the microservice node as a fault source on other microservice nodes is calculated, and the impact weight is calculated based on the call frequency between services, the call link length, the importance of the service, and the historical fault data, including:

[0078] Based on the service dependency graph, the microservice node to be analyzed is selected as the fault source node, and the following steps are performed to calculate the impact weight of the fault source node on other microservice nodes:

[0079] A graph traversal algorithm is used to analyze the call link from the fault source node to each of the microservice nodes, the call frequency data on the call link is extracted, and the call frequencies on the call link are multiplied in sequence to obtain the service-to-service call frequency impact factor;

[0080] Determine the call link length based on the number of node hops of the call link, and calculate the call link length impact factor according to a preset attenuation function, wherein the call link length impact factor decreases as the number of node hops of the call link increases;

[0081] Obtaining service importance data of the microservice node, quantifying the service importance data based on business importance, user impact scope, and business loss degree, and obtaining a service importance impact factor;

[0082] Analyze the historical fault records of the microservice nodes, extract the fault frequency, fault recovery time, and fault spread characteristics, and construct historical fault impact factors;

[0083] The influence factor of the service call frequency, the influence factor of the call link length, the influence factor of the service importance and the influence factor of the historical fault are weighted and combined to obtain the influence weight of the fault source node on the current microservice node;

[0084] The above steps are repeated until the influence weight of the fault source node on all other microservice nodes in the service dependency graph is calculated.

[0085] The node impact weight calculation method of the service dependency graph is used to evaluate the impact of each service node on other nodes when it is a fault source in the microservice architecture. This method comprehensively considers the call frequency between services, the call link length, the importance of the service, and the historical fault data to more accurately quantify the impact of the fault.

[0086] First, build a service dependency graph. The graph uses microservices as nodes and the call relationships between services as edges. Each edge contains call frequency information, such as the number of calls per minute. For example, a system containing four microservices A, B, C, and D. A calls B 10 times per minute, A also calls C 5 times per minute, B calls D 2 times per minute, and C also calls D 3 times per minute.

[0087] Next, select the microservice node to be analyzed as the fault source node. Assume that A is selected as the fault source node.

[0088] Then, analyze the call links from the fault source node to other microservice nodes. The call link length from A to B is 1, and the call frequency is 10 times per minute; the call link length from A to C is 1, and the call frequency is 5 times per minute; there are two call links from A to D: A->B->D, with a length of 2 and a call frequency of 10 times per minute * 2 times = 20 times; A->C->D, with a length of 2 and a call frequency of 5 times per minute * 3 times = 15 times.

[0089] Calculate the impact factor of the call frequency between services. For A to B, the impact factor is 10; for A to C, the impact factor is 5; for A to D, since there are two links, the call frequencies of the two links are added together, and the impact factor is 20+15=35.

[0090] Calculate the call link length impact factor. Assume that the preset attenuation function is: when the link length is 1, the impact factor is 1; when the link length is 2, the impact factor is 0.5; for every increase of 1 in the link length, the impact factor decays by half. Then the link length impact factors from A to B and from A to C are both 1, and the two link length impact factors from A to D are both 0.5.

[0091] Get the service importance data of the microservice node. For example, based on the business importance, user impact range, and business loss degree, quantify the service importance into a level from 1 to 5. The higher the level, the higher the importance. Assume that the importance level of B is 3, the importance level of C is 4, and the importance level of D is 5. Convert the importance level into a service importance impact factor. For example, the impact factor corresponding to level n is n*0.2. Then the service importance impact factor of B is 0.6, C is 0.8, and D is 1.

[0092] Analyze the historical fault records of microservice nodes. Extract features such as fault frequency, fault recovery time, and fault propagation range. Assume that B's historical fault records show that it has an average of one fault per month, with an average recovery time of 1 hour and a small fault propagation range; C's historical fault records show that it has an average of one fault per quarter, with an average recovery time of 2 hours and a medium fault propagation range; D's historical fault records show that it has an average of one fault per year, with an average recovery time of 4 hours and a large fault propagation range. Based on this information, a historical fault impact factor can be constructed, for example, the fault frequency, recovery time, and propagation range are quantified and weighted summed. Assume that the historical fault impact factor of B is 0.2, that of C is 0.4, and that of D is 0.8.

[0093] The influence factors of service call frequency, call link length, service importance and historical faults are weighted and combined to obtain the influence weight of the fault source node on other microservice nodes.

[0094] Repeat the above steps until the impact weight of the fault source node on all other microservice nodes is calculated.

[0095] Beneficial effects:

[0096] Accurate fault impact assessment: This method takes into account multiple factors and can more accurately assess the scope and extent of the impact when the microservice node is the source of the fault, avoiding the assessment bias caused by relying on a single indicator.

[0097] Effective fault warning: By calculating the impact weight, potential high-risk nodes can be identified, and early warning and preventive measures can be taken in advance to reduce the possibility of faults.

[0098] Optimized resource allocation: Based on the impact weight, resources can be allocated preferentially to ensure the stability of key nodes and improve the reliability of the overall system.

[0099] In an optional implementation, a graph convolutional neural network is used to model the service dependency graph, and the operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to construct a fault propagation prediction model, including:

[0100] Based on the service call log, the call frequency between microservice nodes is counted; the call frequency is normalized to generate edge weight values; a weighted directed graph structure is constructed, wherein the directed graph structure includes a microservice node set, a call relationship edge set, and an edge weight matrix; self-loop connections are added to the diagonal of the adjacency matrix of the directed graph structure to generate an asymmetric adjacency matrix;

[0101] Collect node multi-dimensional features, collect the operating status indicators of each node in the microservice node set, the operating status indicators include average response time, request success rate and error rate; collect system resource indicators of each node in the microservice node set, the system resource indicators include CPU usage, memory usage and network IO; collect business indicators of each node in the microservice node set, the business indicators include service call volume, business processing delay and business success rate; calculate the historical mean and standard deviation of the operating status indicator, the system resource indicator and the business indicator; standardize the operating status indicator, the system resource indicator and the business indicator based on the historical mean and the standard deviation to generate an initial node feature matrix;

[0102] Constructing a graph convolutional feature extraction network, inputting the asymmetric adjacency matrix and the initial node feature matrix into the first graph convolutional layer; calculating the first layer node features through the first graph convolutional layer; fusing the first layer node features with the original features through residual connection; inputting the fused features into the second graph convolutional layer; introducing an attention mechanism into the second graph convolutional layer, performing weighted aggregation on neighbor node features based on the edge weight values; and outputting the final node representation features;

[0103] A multi-task prediction model is constructed, and the node representation features are input into a multi-layer perceptron; in a first branch of the multi-layer perceptron, a fault state classifier is constructed based on the node representation features; in a second branch of the multi-layer perceptron, a similarity matrix of the node representation features is calculated; a fault propagation prediction model is constructed based on the similarity matrix; and the outputs of the fault state classifier and the fault propagation prediction model are combined to generate a final prediction result.

[0104] The microservice system fault propagation prediction method includes the following steps:

[0105] First, construct a microservice dependency graph. Extract the call relationship and call count between microservices from the system log. For example, service A calls service B 100 times, service B calls service C 50 times, and service A calls service C 20 times. Normalize the call counts and generate edge weights. For example, the weight of service A calling service B is 100 / (100+20)=0.83, and the weight of service A calling service C is 20 / (100+20)=0.17. Similarly, construct a weighted directed graph. The directed graph contains a set of microservice nodes (A, B, C), a call relationship edge set (A->B, B->C, A->C), and an edge weight matrix (including weight values ​​such as 0.83, 0.17, and 0.5). To avoid information loss, add self-loop connections on the diagonal of the adjacency matrix, that is, connect each node to itself with a weight of 1. Finally, an asymmetric adjacency matrix is ​​generated.

[0106] Next, collect and process node features. Collect the operating status indicators of microservice nodes (for example, the average response time of service A is 0.5 seconds, the request success rate is 99%, and the error rate is 1%), system resource indicators (for example, the CPU utilization rate of service A is 70%, the memory utilization rate is 80%, and the network IO is 10Mbps) and business indicators (for example, the service call volume of service A is 1000 times / minute, the business processing delay is 0.2 seconds, and the business success rate is 98%). Calculate the historical mean and standard deviation for each indicator. For example, the historical mean of the average response time of service A is 0.4 seconds, and the standard deviation is 0.1 seconds. Use the Z-score standardization method to standardize the collected indicators. For example, the standardized value of the average response time of service A is (0.5-0.4) / 0.1=1. All standardized indicators form the initial node feature matrix.

[0107] Then, a graph convolutional feature extraction network is constructed. The asymmetric adjacency matrix and the initial node feature matrix are input into the first graph convolutional layer. The first graph convolutional layer performs weighted aggregation on the neighbor node features of each node to generate the first-layer node features. The first-layer node features are added to the initial node features through residual connection, and the fused features are input into the second graph convolutional layer. The second graph convolutional layer introduces the attention mechanism to perform weighted aggregation on the neighbor node features according to the edge weight value. For example, the features of the neighbor nodes B and C of service A are [0.8, 0.2, 0.5] and [0.1, 0.9, 0.3], respectively, and the edge weights are 0.83 and 0.17, respectively. Then the second-layer features of service A are 0.83*[0.8, 0.2, 0.5]+0.17*[0.1, 0.9, 0.3]. The final output is a matrix containing the final representation features of each node.

[0108] Finally, a multi-task prediction model is constructed. The node representation features are input into the multilayer perceptron. The first branch of the multilayer perceptron constructs a fault state classifier to predict the failure probability of each node. For example, the failure probability of service A is 0.1. The second branch of the multilayer perceptron calculates the similarity matrix of the node representation features. For example, the similarity between service A and service B is 0.9. A fault propagation prediction model is constructed based on the similarity matrix to predict the probability of fault propagation between services. For example, the probability that the fault of service A propagates to service B is 0.8. The outputs of the fault state classifier and the fault propagation prediction model are combined to generate the final prediction result, for example, the probability of service B failing in the next hour is predicted to be 0.08.

[0109] The beneficial effects of this method are reflected in the following three aspects:

[0110] 1. Improve prediction accuracy: By capturing the dependencies between microservices through graph convolutional networks and combining them with multi-dimensional features for modeling, fault status and propagation paths can be predicted more accurately.

[0111] 2. Reduce false alarm rate: Through the attention mechanism, the model can pay more attention to important neighbor nodes and features, thereby reducing interference with irrelevant information and reducing the false alarm rate.

[0112] 3. Enhanced explainability: The output of the model includes failure probability and propagation path, which can help operation and maintenance personnel better understand the cause and impact of the failure, so as to conduct more targeted troubleshooting and processing.

[0113] In an optional implementation, the operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as a fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network. The list of affected service nodes and their impact levels are output, including:

[0114] Collect microservice operation data, deploy monitoring probes to collect operation status indicators of microservice nodes, the operation status indicators include response time, request success rate, and error rate; collect system resource indicators of the microservice nodes, the system resource indicators include CPU usage, memory usage, and network IO; collect business indicators of the microservice nodes, the business indicators include service call volume, business processing delay, and business success rate; use a sliding time window to aggregate the operation status indicators, the system resource indicators, and the business indicators, the window size of the sliding time window is 5 minutes, and the step size is 1 minute; calculate the statistical characteristics of each indicator in the sliding time window, and generate a real-time monitoring feature matrix;

[0115] Construct an anomaly detection baseline, construct a Gaussian distribution model for the continuous indicators in the real-time monitoring feature matrix; construct a frequency distribution model for the discrete indicators in the real-time monitoring feature matrix; combine the Gaussian distribution model and the frequency distribution model to form a normal operation baseline; calculate the deviation measurement value between the real-time monitoring feature matrix and the normal operation baseline; perform weighted combination of the deviation measurement values ​​to obtain an anomaly score; identify the fault source node based on the anomaly score;

[0116] Based on the fault propagation model, the abnormal score of the fault source node and the real-time monitoring feature matrix are combined to form an initial node state vector; the initial node state vector is input into a pre-trained graph convolutional neural network; the node state is updated through the graph convolutional layer propagation; the influence strength between nodes is calculated based on the updated node state; and a fault impact propagation matrix is ​​generated;

[0117] Calculate the impact range, calculate the topological impact score between nodes based on the fault impact propagation matrix; calculate the dependency impact score between nodes in combination with the service call relationship; fuse the historical fault data to calculate the vulnerability impact score of the node; perform weighted fusion on the topological impact score, the dependency impact score and the vulnerability impact score; generate a comprehensive risk score for the node;

[0118] Output the prediction results, classify the risk levels of microservice nodes based on the comprehensive risk scores; identify high-risk nodes as key focus objects; calculate the estimated impact time of each node; generate a prediction result report including risk level and impact time; output the prediction result report to guide fault emergency response.

[0119] In order to predict the impact of fault propagation of microservices in distributed systems in real time, this embodiment provides a fault propagation prediction method based on graph convolutional neural network. This method collects the running status data of microservice nodes, builds anomaly detection baseline, identifies fault source nodes, and uses graph convolutional neural network to predict fault propagation path and impact range, and finally outputs a list of affected service nodes and their impact degree.

[0120] First, deploy monitoring probes to collect operating status indicators of microservice nodes. These indicators include response time, request success rate, error rate, etc., which reflect the performance and stability of microservices. At the same time, collect system resource indicators, such as CPU usage, memory usage, network IO, etc., as well as business indicators, such as service call volume, business processing delay, business success rate, etc. These indicators characterize the operating status of microservices from different dimensions.

[0121] The collected indicator data is aggregated using a sliding time window. For example, the sliding time window size is set to 5 minutes and the step size is 1 minute. This means that the statistical features of the data in the past 5 minutes will be calculated every minute. In this way, the short-term change trend of the indicator data can be captured. Assuming that the response time of a microservice node in the past 5 minutes is 100ms, 120ms, 150ms, 180ms, and 200ms, its statistical features such as average value and standard deviation can be calculated.

[0122] Next, build an anomaly detection baseline. For continuous indicators, such as response time, you can build a Gaussian distribution model. By counting historical data, you can get the average and standard deviation of the indicator, and then determine the parameters of the Gaussian distribution. For discrete indicators, such as request success rate, you can build a frequency distribution model. Count the number of occurrences of each value to get the frequency distribution. Combine the Gaussian distribution model and the frequency distribution model to form a normal operation baseline.

[0123] Compare the real-time monitoring feature matrix with the normal operation baseline and calculate the deviation metric value. For example, you can calculate the difference between the real-time indicator value and the baseline average value, or calculate the probability that the real-time indicator value falls outside the baseline distribution. Perform a weighted combination of different deviation metric values ​​to obtain anomaly scores. For example, you can assign different weights based on the importance of the indicator. Based on the anomaly score, identify the fault source node. For example, set a threshold, and the node that exceeds the threshold is identified as the fault source node.

[0124] The anomaly score of the fault source node and the real-time monitoring feature matrix are combined to form the initial node state vector. This vector is input into the pre-trained graph convolutional neural network. The graph convolutional neural network can learn the dependencies between services and simulate the propagation process of faults in the dependency graph. The node state is updated by propagation through the graph convolution layer. The node state vector contains the abnormal information of the node and the influence of neighboring nodes.

[0125] Based on the updated node states, the influence strength between nodes is calculated. For example, the similarity or distance between node state vectors can be calculated. A fault impact propagation matrix is ​​generated, in which each element represents the influence strength between nodes.

[0126] Based on the fault impact propagation matrix, calculate the topological impact scores between nodes. For example, the influence of a node can be calculated based on its position and connection relationship in the dependency graph. Calculate the dependency impact scores between nodes in combination with the service call relationship. For example, if node A frequently calls node B, node B has a greater impact on node A. Calculate the vulnerability impact score of the node by integrating historical fault data. For example, if node C has frequently failed in history, its vulnerability is high. Perform weighted fusion of the topological impact score, dependency impact score, and vulnerability impact score to generate a comprehensive risk score for the node.

[0127] Based on the comprehensive risk score, the microservice nodes are classified into risk levels. For example, nodes can be classified into three levels: high risk, medium risk, and low risk. High-risk nodes are identified as the focus of attention. The estimated impact time of each node is calculated. For example, the impact time can be estimated based on the fault propagation speed and the risk level of the node. A prediction result report containing the risk level and impact time is generated. The prediction result report is output to guide the emergency response to the fault. For example, measures can be taken in advance based on the prediction results to prevent the fault from expanding.

[0128] Beneficial effects:

[0129] 1. Improve fault prediction accuracy: By combining real-time monitoring data, service dependencies, and historical fault data, the fault propagation path and impact range can be predicted more accurately. For example, potentially affected nodes can be identified even if they have not yet shown obvious abnormalities.

[0130] 2. Shorten fault response time: By predicting the impact of a fault in advance, the fault response time can be shortened and the losses caused by the fault can be reduced. For example, emergency plans can be prepared in advance and implemented quickly when a fault occurs.

[0131] 3. Improve system stability: By identifying high-risk nodes and potential failure points, we can guide system optimization and improvement to improve the overall stability of the system. For example, we can allocate resources or optimize codes for high-risk nodes to reduce the probability of failure.

[0132] In an optional implementation, the affected service nodes are classified into risk levels according to the degree of impact, and corresponding levels of alarm information are triggered according to a preset risk threshold, and the fault propagation path is visualized, including:

[0133] Collecting microservice fault propagation data, wherein the microservice fault propagation data includes fault occurrence time, fault impact scope, fault propagation path, fault propagation speed, and fault duration;

[0134] Extracting the impact degree characteristics based on the microservice fault propagation data, including calculating the fault propagation probability between nodes, extracting the fault time delay characteristics, and counting the overlap of the fault impact range, and constructing a fault impact feature matrix; performing feature dimension reduction and normalization processing on the fault impact feature matrix, extracting key influencing factors using the principal component analysis method, and calculating the weight coefficient of each influencing factor;

[0135] Calculate the comprehensive impact score of each affected service node based on the key impact factor and the weight coefficient, and divide the comprehensive impact score into different risk levels according to the preset risk level classification rule; set corresponding risk thresholds for different risk levels, and when the comprehensive impact score of the service node exceeds the risk threshold of the corresponding risk level, trigger the corresponding level of alarm information;

[0136] Construct an alarm information template, which includes the fault impact scope, fault propagation path, risk level determination basis, and processing suggestions, and generate standardized alarm information based on the alarm information template; use a graphical method to display the fault propagation path, including constructing a service dependency diagram, and marking the fault propagation direction, propagation speed, and impact scope in the service dependency diagram.

[0137] Microservice fault propagation risk assessment and visual alarm method, the specific implementation method is as follows:

[0138] First, collect microservice fault propagation data. Use distributed tracing systems and log collection tools to collect various data in the microservice system in real time. These data include the time when the fault occurred, accurate to milliseconds; the scope of the fault, recording the name and number of affected microservices; the fault propagation path, tracking the call link between the faults; the fault propagation speed, calculating the time interval for the fault to propagate from one microservice to the next microservice; the fault duration, recording the length of time from the occurrence of the fault to the system returning to normal. For example, at 10:00:00 on January 1, 2024, microservice A failed, affecting 3 microservices, the propagation path is A->B->C, the propagation speeds are 10ms and 20ms respectively, and the fault duration is 1 minute.

[0139] Next, extract the impact characteristics based on the collected microservice fault propagation data. Calculate the probability of fault propagation between nodes, and count the probability of other microservices failing after a microservice fails in historical fault data. Extract the fault time delay characteristics and calculate the time interval for the fault to propagate from one microservice to the next. Count the overlap of fault impact ranges and calculate the proportion of the intersection of different microservice fault impact ranges to the overall impact range. Construct these features into a fault impact feature matrix. For example, after microservice A fails, the probability of causing microservice B to fail is 80%, and the probability of causing microservice C to fail is 50%; the time delay for the fault to propagate from A to B is 10ms, and the time delay for the fault to propagate from B to C is 20ms; the overlap of the fault impact ranges of microservices A and B is 30%.

[0140] Then, the fault impact feature matrix is ​​subjected to feature dimension reduction and normalization. The principal component analysis method is used to extract key influencing factors and calculate the weight coefficient of each influencing factor. The key influencing factor is a feature combination that can best reflect the degree of fault impact. The weight coefficient represents the contribution of each key influencing factor to the comprehensive impact score. For example, after principal component analysis, two key influencing factors are extracted: fault propagation probability and fault impact range overlap, and their weight coefficients are 0.7 and 0.3 respectively.

[0141] The comprehensive impact score of each affected service node is calculated based on the key impact factors and weight coefficients. Each key impact factor is multiplied by the corresponding weight coefficient, and then all the results are added together to obtain the comprehensive impact score of each affected service node. For example, if the fault propagation probability of microservice B is 80% and the overlap of fault impact range is 30%, its comprehensive impact score is 0.7*0.8+0.3*0.3=0.65.

[0142] The comprehensive impact score is divided into different risk levels according to the preset risk level classification rules. For example, the comprehensive impact score is divided into three risk levels: low, medium, and high, and the corresponding score ranges are 0-0.3, 0.3-0.7, and 0.7-1 respectively.

[0143] Set corresponding risk thresholds for different risk levels. When the comprehensive impact score of the service node exceeds the risk threshold of the corresponding risk level, the corresponding level of alarm information is triggered. For example, the threshold for low risk level is 0.2, the threshold for medium risk level is 0.5, and the threshold for high risk level is 0.8.

[0144] Construct an alarm information template, including the fault impact scope, fault propagation path, risk level determination basis, and handling suggestions. Generate standardized alarm information based on the alarm information template. For example, if the comprehensive impact score of microservice B is 0.65, which exceeds the medium risk level threshold of 0.5, a medium-level alarm information is triggered. The alarm information content includes: Affected microservices: B, C; Fault propagation path: A->B->C; Risk level: Medium; Determination basis: The comprehensive impact score of 0.65 exceeds the threshold of 0.5; Handling suggestions: Check the logs of microservices B and C to troubleshoot the cause of the fault.

[0145] Use graphical methods to display the fault propagation path. Build a service dependency graph and mark the fault propagation direction, propagation speed, and impact range in the graph. For example, use arrows to indicate the fault propagation direction, the numbers on the arrows indicate the propagation speed, and use different colors to indicate the impact range of different risk levels.

[0146] Beneficial effects:

[0147] 1. Improve fault response efficiency: Through real-time monitoring and alarms, faults can be discovered and handled in a timely manner, shortening fault recovery time and avoiding business interruptions.

[0148] 2. Reduce the scope of fault impact: By analyzing the fault propagation path and impact scope, the root cause of the fault can be quickly located, effective measures can be taken to prevent the fault from spreading, and the number of affected service nodes can be reduced.

[0149] 3. Optimize system stability: Through the analysis and accumulation of fault data, the weak links in the system can be identified and targeted optimization can be performed to improve the overall stability and reliability of the system.

[0150] Figure 2 FIG. 1 is a schematic diagram of the structure of a distributed system fault location and diagnosis system based on log analysis according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0151] The first unit is used to obtain all microservice node information through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters, and call status; based on the structured stored call data, a service dependency graph is constructed using a graph database, in which graph nodes represent microservice nodes, graph edges represent call relationships between services, and call frequency, average response time, and success rate are annotated on the graph edges;

[0152] The second unit is used to calculate the influence weight of each microservice node on other microservice nodes when it is a fault source based on the service dependency graph, and the influence weight is calculated comprehensively according to the call frequency between services, the call link length, the service importance and the historical fault data; the service dependency graph is modeled by using a graph convolutional neural network, and the operation status indicator, system resource usage and business indicator of each microservice node are used as node feature inputs to build a fault propagation prediction model;

[0153] The third unit is used to collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact levels are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

[0154] According to a third aspect of the embodiments of the present invention,

[0155] An electronic device is provided, comprising:

[0156] processor;

[0157] a memory for storing processor-executable instructions;

[0158] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0159] A fourth aspect of the embodiments of the present invention is:

[0160] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0161] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed system fault location and diagnosis method based on log analysis, characterized in that: include: Obtain all microservice node information through the service registration center in the distributed system, collect call data between microservice nodes based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service ID, callee service ID, call interface name, call parameters and call status; Based on the structured stored call data, a graph database is used to build a service dependency graph, where graph nodes represent microservice nodes, graph edges represent the call relationships between services, and the call frequency, average response time, and success rate are annotated on the graph edges; Based on the service dependency graph, for each microservice node, the influence weight on other microservice nodes when it is a fault source is calculated, and the influence weight is calculated comprehensively based on the call frequency between services, the call link length, the service importance and the historical fault data; the service dependency graph is modeled using a graph convolutional neural network, and the operating status indicators, system resource usage and business indicators of each microservice node are used as node feature inputs to build a fault propagation prediction model; The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as a fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact levels are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

2. The method according to claim 1, characterized in that Obtain all microservice node information through the service registration center in the distributed system, collect call data between microservice nodes based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status, including: The registration information of all microservice nodes is obtained through the service registration center in the distributed system. The registration information includes the service name, instance identifier, network address, port number and health check address. The service registration center stores the registration information in the form of a temporary node and maintains the survival status of the microservice node through a heartbeat mechanism; The call data between each microservice node is collected through the application program interface call link tracker, and the application program interface call link tracker injects the tracking logic into the remote call entry point of the microservice node, generates a globally unique tracking identifier and a call identifier, and propagates the tracking identifier and the call identifier in the call link; the application program interface call link tracker records the call timestamp, the caller service identifier, the callee service identifier, the call interface name, the call parameters and the call status; The call data is sent to a message queue for temporary storage, and a dedicated consumer cluster obtains the call data from the message queue and performs sharding processing, wherein the sharding processing divides the call data into multiple data shards according to the service dimension; the sharded call data is written into a distributed time series database for structured storage.

3. The method according to claim 1, characterized in that: Based on the service dependency graph, for each microservice node, the impact weight on other microservice nodes when it is a fault source is calculated. The impact weight is calculated based on the call frequency between services, the call link length, the importance of the service, and the historical fault data, including: Based on the service dependency graph, the microservice node to be analyzed is selected as the fault source node, and the following steps are performed to calculate the impact weight of the fault source node on other microservice nodes: A graph traversal algorithm is used to analyze the call link from the fault source node to each of the microservice nodes, the call frequency data on the call link is extracted, and the call frequencies on the call link are multiplied in sequence to obtain the service-to-service call frequency impact factor; Determine the call link length based on the number of node hops of the call link, and calculate the call link length impact factor according to a preset attenuation function, wherein the call link length impact factor decreases as the number of node hops of the call link increases; Obtaining service importance data of the microservice node, quantifying the service importance data based on business importance, user impact scope, and business loss degree, and obtaining a service importance impact factor; Analyze the historical fault records of the microservice nodes, extract the fault frequency, fault recovery time, and fault spread characteristics, and construct historical fault impact factors; The influence factor of the service call frequency, the influence factor of the call link length, the influence factor of the service importance and the influence factor of the historical fault are weighted and combined to obtain the influence weight of the fault source node on the current microservice node; The above steps are repeated until the influence weight of the fault source node on all other microservice nodes in the service dependency graph is calculated.

4. The method according to claim 1, characterized in that The service dependency graph is modeled using a graph convolutional neural network, and the operating status indicators, system resource usage, and business indicators of each microservice node are used as node feature inputs to build a fault propagation prediction model, including: Based on the service call log, the call frequency between microservice nodes is counted; the call frequency is normalized to generate edge weight values; a weighted directed graph structure is constructed, wherein the directed graph structure includes a microservice node set, a call relationship edge set, and an edge weight matrix; self-loop connections are added to the diagonal of the adjacency matrix of the directed graph structure to generate an asymmetric adjacency matrix; Collect node multi-dimensional features, collect the operating status indicators of each node in the microservice node set, the operating status indicators include average response time, request success rate and error rate; collect system resource indicators of each node in the microservice node set, the system resource indicators include CPU usage, memory usage and network IO; collect business indicators of each node in the microservice node set, the business indicators include service call volume, business processing delay and business success rate; calculate the historical mean and standard deviation of the operating status indicator, the system resource indicator and the business indicator; standardize the operating status indicator, the system resource indicator and the business indicator based on the historical mean and the standard deviation to generate an initial node feature matrix; Constructing a graph convolutional feature extraction network, inputting the asymmetric adjacency matrix and the initial node feature matrix into the first graph convolutional layer; calculating the first layer node features through the first graph convolutional layer; fusing the first layer node features with the original features through residual connection; inputting the fused features into the second graph convolutional layer; introducing an attention mechanism into the second graph convolutional layer, performing weighted aggregation on neighbor node features based on the edge weight values; and outputting the final node representation features; A multi-task prediction model is constructed, and the node representation features are input into a multi-layer perceptron; in a first branch of the multi-layer perceptron, a fault state classifier is constructed based on the node representation features; in a second branch of the multi-layer perceptron, a similarity matrix of the node representation features is calculated; a fault propagation prediction model is constructed based on the similarity matrix; and the outputs of the fault state classifier and the fault propagation prediction model are combined to generate a final prediction result.

5. The method according to claim 1, characterized in that The operating status data of each microservice node in the distributed system is collected in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as a fault source node. The propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and the list of affected service nodes and their impact levels are output, including: Collect microservice operation data, deploy monitoring probes to collect operation status indicators of microservice nodes, the operation status indicators include response time, request success rate, and error rate; collect system resource indicators of the microservice nodes, the system resource indicators include CPU usage, memory usage, and network IO; collect business indicators of the microservice nodes, the business indicators include service call volume, business processing delay, and business success rate; use a sliding time window to aggregate the operation status indicators, the system resource indicators, and the business indicators, the window size of the sliding time window is 5 minutes, and the step size is 1 minute; calculate the statistical characteristics of each indicator in the sliding time window, and generate a real-time monitoring feature matrix; Construct an anomaly detection baseline, construct a Gaussian distribution model for the continuous indicators in the real-time monitoring feature matrix; construct a frequency distribution model for the discrete indicators in the real-time monitoring feature matrix; combine the Gaussian distribution model and the frequency distribution model to form a normal operation baseline; calculate the deviation measurement value between the real-time monitoring feature matrix and the normal operation baseline; perform weighted combination of the deviation measurement values ​​to obtain an anomaly score; identify the fault source node based on the anomaly score; Based on the fault propagation model, the abnormal score of the fault source node and the real-time monitoring feature matrix are combined to form an initial node state vector; the initial node state vector is input into a pre-trained graph convolutional neural network; the node state is updated through the graph convolutional layer propagation; the influence strength between nodes is calculated based on the updated node state; and a fault impact propagation matrix is ​​generated; Calculate the impact range, calculate the topological impact score between nodes based on the fault impact propagation matrix; calculate the dependency impact score between nodes in combination with the service call relationship; fuse the historical fault data to calculate the vulnerability impact score of the node; perform weighted fusion on the topological impact score, the dependency impact score and the vulnerability impact score; generate a comprehensive risk score for the node; Output the prediction results, classify the risk levels of microservice nodes based on the comprehensive risk scores; identify high-risk nodes as key focus objects; calculate the estimated impact time of each node; generate a prediction result report including risk level and impact time; output the prediction result report to guide fault emergency response.

6. The method according to claim 1, characterized in that The affected service nodes are classified into risk levels according to the degree of impact, and the corresponding level of alarm information is triggered according to the preset risk threshold. At the same time, the fault propagation path is visualized, including: Collecting microservice fault propagation data, wherein the microservice fault propagation data includes fault occurrence time, fault impact scope, fault propagation path, fault propagation speed, and fault duration; Extracting the impact degree characteristics based on the microservice fault propagation data, including calculating the fault propagation probability between nodes, extracting the fault time delay characteristics, and counting the overlap of the fault impact range, and constructing a fault impact feature matrix; performing feature dimension reduction and normalization processing on the fault impact feature matrix, extracting key influencing factors using the principal component analysis method, and calculating the weight coefficient of each influencing factor; Calculate the comprehensive impact score of each affected service node based on the key impact factor and the weight coefficient, and divide the comprehensive impact score into different risk levels according to the preset risk level classification rule; set corresponding risk thresholds for different risk levels, and when the comprehensive impact score of the service node exceeds the risk threshold of the corresponding risk level, trigger the corresponding level of alarm information; Construct an alarm information template, which includes the fault impact scope, fault propagation path, risk level determination basis, and processing suggestions, and generate standardized alarm information based on the alarm information template; use a graphical method to display the fault propagation path, including constructing a service dependency diagram, and marking the fault propagation direction, propagation speed, and impact scope in the service dependency diagram.

7. A distributed system fault location and diagnosis system based on log analysis, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to obtain all microservice node information through the service registration center in the distributed system, collect call data between each microservice node based on the application interface call link tracker, and store the call data in a structured manner according to the call timestamp, caller service identifier, callee service identifier, call interface name, call parameters and call status; Based on the structured stored call data, a graph database is used to build a service dependency graph, where graph nodes represent microservice nodes, graph edges represent the call relationships between services, and the call frequency, average response time, and success rate are annotated on the graph edges; The second unit is used to calculate the influence weight of each microservice node on other microservice nodes when it is a fault source based on the service dependency graph, and the influence weight is calculated comprehensively according to the call frequency between services, the call link length, the service importance and the historical fault data; the service dependency graph is modeled by using a graph convolutional neural network, and the operation status indicator, system resource usage and business indicator of each microservice node are used as node feature inputs to build a fault propagation prediction model; The third unit is used to collect the operating status data of each microservice node in the distributed system in real time. When an abnormality is detected in a microservice node, the abnormal node is input into the fault propagation prediction model as the fault source node, and the propagation path and impact range of the fault in the service dependency graph are calculated based on the graph convolutional neural network, and a list of affected service nodes and their impact levels are output; the affected service nodes are divided into risk levels according to the impact level, and the corresponding level of alarm information is triggered according to the preset risk threshold, and the fault propagation path is visualized.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images