System and method for analyzing and blocking fault propagation path of distributed system based on large model technology

Abstract

The invention provides a system and method for analyzing and blocking a fault propagation path of a distributed system based on a large model technology, and the method comprises the steps: monitoring the distributed system, recognizing a fault of the distributed system, and giving an alarm; after the alarm is received, measuring a single service fault event and analyzing a causal relationship between different service fault events through service operation data of the distributed system to obtain service fault contents; marking nodes and edges in the distributed system, and constructing a service fault propagation graph; according to the constructed service fault propagation graph and the output service fault content, fault paths sorted according to propagation probabilities are identified; and after the fault path identified by the fault propagation path identification module is received, a preset fault blocking strategy is adopted to block the fault. According to the invention, the efficiency and accuracy of fault diagnosis are improved, further propagation of faults is effectively prevented, the system loss is reduced, and the fault processing capability of the system can be continuously optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of fault propagation path analysis and blocking, and specifically relates to a system and method for realizing distributed system fault propagation path analysis and blocking based on large model technology. Background Art

[0002] With the wide application of distributed systems, their scale and complexity are increasing continuously, and the occurrence of faults is inevitable. The propagation of faults in distributed systems may lead to serious performance degradation or even system crashes. Traditional fault diagnosis and handling methods are often inefficient and difficult to accurately and quickly analyze the fault propagation path and effectively block it. Summary of the Invention

[0003] Aiming at the deficiencies in the prior art, the present invention provides a system and method for realizing distributed system fault propagation path analysis and blocking based on large model technology.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] In the first aspect, the present invention provides a system for realizing distributed system fault propagation path analysis and blocking based on large model technology, including: a monitoring module, a fault metric module, a fault propagation graph construction module, a fault propagation path identification module, a fault blocking module, and an operation and maintenance large model;

[0006] The monitoring module collects various data of the distributed system for monitoring the distributed system, identifying the faults occurring in the distributed system and sending out alarms;

[0007] After receiving the alarm sent by the monitoring module, the fault metric module measures a single service fault event and analyzes the causal relationship between different service fault events through the service operation data of the distributed system to obtain the service fault content;

[0008] The fault propagation graph construction module labels the nodes and edges in the distributed system to construct a service fault propagation graph;

[0009] The fault propagation path identification module identifies the fault paths sorted by propagation probability according to the service fault propagation graph constructed by the fault propagation graph construction module and the service fault content output by the fault metric module;

[0010] After receiving the fault paths identified by the fault propagation path identification module, the fault blocking module takes a preset fault blocking strategy to block the faults;

[0011] The operation and maintenance large model collects, trains, and analyzes various data of the distributed system for use by the fault metric module, the fault propagation graph construction module, the fault propagation path identification module, and the fault blocking module.

[0012] Optionally, the monitoring module collects various data of the distributed system, specifically:

[0013] Measure the response time of each service interface in the distributed system, including the average response time, the maximum response time, and the minimum response time; collect the data supporting the operation of the distributed system, including the operation data of the operating system, the operation data of network devices, server devices, storage devices, and the operation data of databases, storage, and middleware.

[0014] Optionally, when the data collected by the monitoring module triggers a certain threshold, the monitoring module issues an alarm, and the data format of the alarm information is xml format or json format, and jsonp cross-domain access is supported.

[0015] Optionally, the operation and maintenance large model includes a data collection sub-module, a data processing sub-module, and a data training sub-module;

[0016] The data collection sub-module collects various data provided by the monitoring module on the one hand, and on the other hand, collects operation data from each component of the distributed system, including system logs, performance metrics, and service call records;

[0017] The data processing sub-module converts the data in different formats collected by the data collection sub-model into a unified format;

[0018] The data training sub-module analyzes the data processed by the data processing sub-module using natural language processing technology based on a model with a Transformer architecture: by performing semantic understanding on the log text, identifying key events including service startup, stop, and errors; sorting out the sequence of events from the logs of multiple nodes to determine the service call chain.

[0019] Optionally, the fault metric module measures a single service fault event through the service operation data of the distributed system, specifically:

[0020] Take the statistic As the service fault event metric value, the calculation formula is:

[0021]

[0022] In the formula, u and Σ are the mean vector and covariance matrix composed of the mean values of each index in the service normal operation data set respectively, And are the mean vector composed of the mean values of each index in the current service operation data set and the data volume of the data set respectively;

[0023] When the of the service is higher than the threshold, it is considered that the service has failed.

[0024] Optionally, the fault metric module identifies the factors leading to service failure and the causal relationships between the factors through a linear structural causal model, specifically:

[0025] In the linear structural causal model, if the failure of service x causes the failure of service y, the causal relationship is expressed as y = wx + e, where e is a random disturbance term, representing the influence of random events other than x on y, and w is the causal effect strength; for the failure events x1, x2,..., x n , the causal relationship is in the following form:

[0026]

[0027] In the formula, e1, e2,..., e n represents the corresponding random disturbance terms of x1, x2,..., x n , and w nn-1 represents the causal effect strength of x n-1 on x n .

[0028] Infer the causal relationship of service failure events according to the DirectLiNGAM algorithm.

[0029] Optionally, the service failure propagation graph constructed by the failure propagation graph construction module is a weighted directed graph, where the nodes represent the failed services, the direction of the edges represents the failure propagation direction, and the weights represent the causal effect strength of the failure events.

[0030] Optionally, the failure propagation path identification module combines the service failure content output by the failure metric module, calculates the weights of the service failure propagation graph through the failure propagation path identification algorithm, traverses the service failure propagation graph, filters out all failure paths starting from the failed service, calculates the causal effect strength of the failure paths according to the weights of all the edges on the failure paths, sorts all the failure paths in descending order of the causal effect strength, and the higher the ranking of the failure path, the greater the propagation probability of the failure path.

[0031] Optionally, the failure blocking strategy adopted by the failure blocking module is set according to the role of the service node, including: for applications that do not belong to the core business process, call the traffic limiting function in the distributed system to eliminate the application; for applications that belong to the cache and there is a database at the back end, call the circuit breaking function in the distributed system to break through the application and switch all traffic to the database.

[0032] Second aspect, the present invention provides a method for analyzing and blocking the fault propagation path of a distributed system based on large model technology, including the following steps:

[0033] S1: Collect various data of the distributed system for monitoring the distributed system, identify the faults occurring in the distributed system and issue alarms;

[0034] S2: After receiving the alarm, measure a single service fault event and analyze the causal relationship between different service fault events through the service operation data of the distributed system to obtain the service fault content;

[0035] S3: Label the nodes and edges in the distributed system to construct a service fault propagation graph;

[0036] S4: According to the constructed service fault propagation graph and the output service fault content, identify the fault paths sorted by propagation probability;

[0037] S5: After receiving the fault paths identified by the fault propagation path identification module, adopt a preset fault blocking strategy to block the faults.

[0038] The beneficial effects of the present invention are as follows: The present invention proposes a system and method for analyzing and blocking the fault propagation path of a distributed system based on large model technology, introduces advanced large model technology, and can more accurately analyze the fault propagation path by using large model technology, improve the efficiency and accuracy of fault diagnosis, generate and implement effective blocking strategies in a timely manner, prevent the further propagation of faults, and reduce system losses; and through continuous effect evaluation and model adjustment, the fault handling ability of the system can also be continuously optimized. The present invention proposes an integrated comprehensive solution for fault monitoring, fault measurement, fault propagation graph construction, fault path identification and fault blocking, provides full-process management for the generation, quantification, propagation and blocking of faults, and innovatively designs data processing and analysis methods in each link, and can quickly locate the fault location, analyze the fault propagation path and effectively block it. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a framework diagram of a system for analyzing and blocking the fault propagation path of a distributed system based on large model technology.

[0040] Figure 2 is a flowchart of a method for analyzing and blocking the fault propagation path of a distributed system based on large model technology. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] Now, the present invention will be further described in detail with reference to the accompanying drawings.

[0042] In one embodiment, the present invention proposes a system for analyzing and blocking the fault propagation path of a distributed system based on large model technology, as Figure 1 shown, including: a monitoring module, a fault metric module, a fault propagation graph construction module, a fault propagation path identification module, a fault blocking module, and an operation and maintenance large model.

[0043] The monitoring module is used to monitor the distributed system. When problems or faults occur in the distributed system, the monitoring module can accurately identify them and issue alarms.

[0044] In this embodiment, the monitoring module is used to collect various types of data of the distributed system, measure the response time of each service interface in the distributed system, including the average response time, the maximum response time, the minimum response time, etc., and at the same time collect the data supporting the operation of the distributed system, including the operation data of the operating system, the operation data of network devices, server devices, and storage devices, and the operation data of databases, storage, and middleware. When a certain threshold is triggered, such as when the average response time exceeds 20ms or the CPU usage rate exceeds 80%, an alarm is issued. The request format of the alarm data format sent by the monitoring module is in xml format or json format, and jsonp cross-domain access is supported.

[0045] After receiving the alarm sent by the monitoring module, the fault metric module measures a single service fault event through the service operation data of the distributed system, reflects the fault situation of a single service, and can also analyze the causal relationship between different service fault events. The fault metric module identifies various factors leading to service faults and their causal relationships, and determines the contribution degree of each factor to the fault by analyzing the causal relationship, so as to measure the fault more accurately.

[0046] In this embodiment, the causal relationship is analyzed through a causal model. The causal model identifies various factors leading to service faults and their causal relationships, and determines the contribution degree of each factor to the fault by analyzing the causal relationship, so as to measure the fault more accurately. The causal relationship is composed of the deviation of the service operation data index expectation and the change of the service scheduling logic. Usually, the statistic is used as the service fault event metric value. After removing the indicators with a variance of 0 and non-stationary indicators during normal service operation, the principal component analysis (PCA) algorithm is used to remove the perfect collinearity in the data to avoid the metric value being less than 0, and then the covariance matrix is used to eliminate the linear relationship between the indicators, so that the deviation of the indicator mean can be directly used to measure these two types of situations, and finally a fault metric value is formed. The formula for the statistic is:

[0047]

[0048] Wherein, \(u\) and \(\sum\) are respectively the mean vector and covariance matrix composed of the means of various indicators in the normal service operation dataset. And \(\overline{x}\) and \(n\) are respectively the mean vector composed of the means of various indicators in the current service operation dataset and the data volume of the dataset. When a certain service is higher than the threshold (such as 1.15), it is considered that the service has a fault.

[0049] Due to the linear correlation of service fault event measurement vectors caused by service fault propagation, service fault events follow a linear structural causal model. In the linear structural causal model, if the fault of service \(x\) causes the fault of service \(y\), the causal relationship is expressed as \(y = wx + e\), where \(e\) is a random disturbance term representing the influence of random events other than \(x\) on \(y\), and \(w\) is the causal effect intensity; for the fault events \(x_1, x_2, \cdots, x_n\) of \(n\) services n , the causal relationship is in the following form:

[0050]

[0051] Wherein, \(e_1, e_2, \cdots, e_n\) n represent the corresponding random disturbance terms of \(x_1, x_2, \cdots, x_n\), and \(w_{ij}\) n represents the causal effect intensity of \(x_i\) nn-1 on \(x_j\). n-1 n

[0052] The causal relationship of service fault events is inferred according to the DirectLiNGAM algorithm, which is used to construct a service fault propagation graph subsequently to determine the propagation direction of service faults and the causal effect intensity between fault events.

[0053] The fault propagation graph construction module conducts a detailed analysis of the distributed system to determine all components included in the system. These components can be hardware devices (such as servers, network devices, etc.), software modules (such as different functional modules of applications), or human-computer interaction interfaces, etc. By adopting the technology of annotating nodes and edges, the nodes and edges in the distributed system are annotated in detail. Node annotation can include information such as component name and fault mode, and edge annotation can include content such as fault propagation conditions, probabilities, and delay times. Finally, the constructed fault propagation graph can more clearly and accurately reflect the actual situation of fault propagation.

[0054] In this embodiment, the service fault propagation graph is a weighted directed graph, where nodes represent faulty services, the direction of edges represents the fault propagation direction, and the weights represent the causal effect intensity of fault events.

[0055] The fault propagation path identification module obtains the service fault propagation path from the service fault propagation graph. According to the fault service content output by the fault metric module, it calculates the weight of the service fault propagation graph through the fault propagation path identification algorithm, traverses the service fault propagation graph, filters out all paths starting from the fault service, multiplies the weights of all edges on the path to obtain the causal effect strength of the path, and finally sorts all paths in descending order of the causal effect strength. The earlier the path is arranged, the greater the probability of its propagation.

[0056] In this embodiment, the fault propagation path identification module first screens out the fault services according to the service fault event measurement method, then standardizes the weights of the service fault propagation graph, then traverses the service fault propagation graph, filters out all paths starting from the fault services, and finally multiplies the weights of all edges on the path as the causal effect strength of the path, and sorts all paths in descending order of the causal effect strength. The earlier the path is arranged, the greater the probability of the fault being propagated by this path.

[0057] The service fault propagation path identification algorithm is as follows:

[0058] Input: Service fault event measurement matrix x, service fault propagation graph adjacency matrix W, fault service list FaultServiceList

[0059] Output: Service fault propagation path FaultPropagationPath

[0060] for i = 0 to len(x) do;

[0061] x[i] ← (x[i] - x[i].mean()) / x[i].std() / / Standardize the service fault event measurement vector

[0062] end for;

[0063] W ← Re-estimate the parameters of W according to the standardized matrix x;

[0064] for j in FaultServiceList:

[0065] FaultPropagationPath ← All paths in W starting from FaultServiceList[j];

[0066] end for;

[0067] for k in FaultPropagationPath:

[0068] CausalStrength←FaultPropagationPath[k] product of all edge weights

[0069] end for;

[0070] FaultPropagationPath←Sort FaultPropagationPath in descending order by CausalStrength

[0071] return FaultPropagationPath

[0072] After receiving the fault path identified by the fault propagation path identification module, the fault blocking module adopts a preset fault blocking strategy to block the fault.

[0073] In this embodiment, the fault blocking strategy can be set according to the role of the service node. For example, if application A does not belong to the core business process, the current limiting function in the distributed system is called to eliminate application A. If application B belongs to the cache and there is a database in the back end to provide a backup, the fuse function in the distributed system is called to break through application B, so that all traffic is switched to the database. Fault blocking can be performed by function. In a distributed system, when it is confirmed that the fault propagation path involves a certain functional service, the service discovery mechanism is dynamically adjusted through configuration management tools (such as Consul, Etcd) to isolate the faulty service from the service call chain. For example, in an e-commerce system, if a product recommendation service fails and may spread to affect the order service, the recommendation service can be temporarily removed from the call link to allow the order service to handle business normally and avoid further propagation of the fault.

[0074] The fault-blocking circuit breaker mechanism in this embodiment can be used to circuit breaker downstream services. In the service call chain, when a downstream service is found to have a fault and may cause an upstream service fault, the circuit breaker mechanism is used. For example, in a microservice architecture, using the Hystrix framework, when the error rate or response time of a downstream service exceeds a set threshold, the upstream service automatically cuts off the call to the downstream service and returns a preset degraded response (such as default data or a friendly error prompt). This prevents the upstream service from timing out while waiting for a response from the downstream fault service, thereby blocking the propagation of the fault.

[0075] The operation and maintenance big model is used to collect various types of data from distributed systems, including but not limited to distributed system operation data, status data supporting distributed system operation, business system data, system link data, and traffic data. It trains and analyzes various types of data. These data will be used by the fault measurement module, fault propagation graph construction module, fault propagation path identification module, and fault blocking module.

[0076] In this embodiment, the operation and maintenance large model includes a data collection sub-module, which comprehensively and accurately collects various types of data provided by the monitoring module, and also includes collecting operation data from various components of the distributed system (such as servers, databases, message queues, etc.). These data include system logs, performance metrics (CPU usage, memory occupancy, network traffic, etc.), service call records, etc.; it includes a data processing sub-module, which converts the collected data in different formats into a unified format for subsequent analysis. For example, it unifies the timestamps into a specific format and unifies the units of performance metrics; it includes a data training sub-module, which uses natural language processing technology to analyze the relevant data based on the Transformer architecture model, extracts key information, and through semantic understanding of the log text, identifies key events such as service startup, stop, and error. For a distributed system, it can sort out the sequence of events from the logs of numerous nodes, determine the service call chain. For example, in a microservice architecture, by analyzing the service names and call IDs in the logs, it determines the transfer link of a user request between different microservices. These data will be provided to the fault metric module, fault propagation graph construction module, fault propagation path identification module, and fault blocking module for use.

[0077] In another embodiment, the present invention also provides a method for analyzing and blocking the fault propagation path of a distributed system based on large model technology, as Figure 2 shown, specifically including the following steps:

[0078] S1: The monitoring module monitors the distributed system. When there is a problem or fault in the distributed system, the monitoring module can accurately identify it and issue an alarm.

[0079] S2: After receiving the alarm issued by the monitoring module, the fault metric module measures a single service fault event through the service operation data of the distributed system, reflects the fault situation of a single service, and can also analyze the causal relationship between different service fault events. The fault metric module determines the contribution degree of each factor to the fault by identifying various factors leading to service faults and their causal relationships, so as to measure the fault more accurately.

[0080] S3: The fault propagation graph construction module conducts a detailed analysis of the distributed system, determines all components included in the system, and adopts the technology of labeling nodes and edges to conduct detailed labeling of the nodes and edges in the distributed system. Node labeling can include information such as component name and fault mode, and edge labeling can include content such as fault propagation conditions, probabilities, and delay times. Finally, a fault propagation graph is constructed to reflect the actual situation of fault propagation.

[0081] S4: The fault propagation path identification module obtains the service fault propagation path from the service fault propagation graph. According to the faulty service content output by the fault metric module, it calculates the weight of the service fault propagation graph through the fault propagation path identification algorithm, traverses the service fault propagation graph, filters out all paths starting from the faulty service, multiplies the weights of all edges on the path to obtain the causal effect strength of the path, and finally sorts all paths in descending order of the causal effect strength. The earlier the path is arranged, the greater the probability of its propagation;

[0082] S5: After receiving the fault path identified by the fault propagation path identification module, the fault blocking module adopts a preset fault blocking strategy to block the fault and ensure the stable operation of the distributed system.

[0083] The method for analyzing and blocking the fault propagation path of a distributed system based on large model technology proposed in this embodiment corresponds to the system for analyzing and blocking the fault propagation path of a distributed system based on large model technology in the foregoing embodiment. The steps of the method are the same as the functions of the modules of the system, so they will not be elaborated here.

[0084] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should be regarded as within the protection scope of the present invention.

Claims

1. A system for analyzing and blocking the fault propagation path of a distributed system based on large model technology, characterized in that Including: A monitoring module, a fault metric module, a fault propagation graph construction module, a fault propagation path identification module, a fault blocking module, and an operation and maintenance large model; The monitoring module collects various data of the distributed system, is used to monitor the distributed system, identify faults occurring in the distributed system, and issue alarms; After receiving the alarm sent by the monitoring module, the fault metric module measures individual service fault events through the service operation data of the distributed system and analyzes the causal relationship between different service fault events to obtain service fault content; The fault propagation graph construction module labels the nodes and edges in the distributed system to construct a service fault propagation graph; The fault propagation path identification module identifies the fault paths sorted by propagation probability according to the service fault propagation graph constructed by the fault propagation graph construction module and the service fault content output by the fault metric module; After receiving the fault paths identified by the fault propagation path identification module, the fault blocking module takes a preset fault blocking strategy to block the faults; The operation and maintenance large model collects, trains, and analyzes various data of the distributed system, and is used to provide data for the fault metric module, the fault propagation graph construction module, the fault propagation path identification module, and the fault blocking module for use.

2. The system for analyzing and blocking the fault propagation path of a distributed system based on large model technology according to claim 1, wherein: The monitoring module collects various data of the distributed system, specifically: Measure the response time of each service interface in the distributed system, including the average response time, the maximum response time, and the minimum response time; Collect data supporting the operation of the distributed system, including the operation data of the operating system, the operation data of network devices, server devices, storage devices, and the operation data of databases, storage, and middleware.

3. A system for analyzing and blocking the fault propagation path of a distributed system based on large model technology according to claim 1, characterized in that: When the data collected by the monitoring module triggers a certain threshold, the monitoring module issues an alarm, and the data format of the alarm information is xml format or json format, and jsonp cross-domain access is supported.

4. A system for analyzing and blocking the fault propagation path of a distributed system based on large model technology according to claim 1, characterized in that: The operation and maintenance large model includes a data collection sub-module, a data processing sub-module, and a data training sub-module; The data collection sub-module, on the one hand, collects various data provided by the monitoring module, and on the other hand, collects operation data from each component of the distributed system, including system logs, performance metrics, and service call records; The data processing sub-module converts the data in different formats collected by the data collection sub-model into a unified format; The data training sub-module analyzes the data processed by the data processing sub-module using natural language processing technology based on a model with a Transformer architecture: by semantically understanding the log text, identifying key events including service start, stop, and errors; Sort out the sequence of events from the logs of multiple nodes to determine the service call chain.

5. The system for analyzing and blocking the fault propagation path of a distributed system based on large model technology as described in claim 1, wherein: The fault metric module measures individual service fault events through the service operation data of the distributed system, specifically: Take the statistic as the service failure event metric value, and the calculation formula is: where \(u\) and \(\sum\) are the mean vector and covariance matrix respectively, which are composed of the means of each index in the normal service operation dataset, and \(n\) are the mean vector composed of the means of each index in the current service operation dataset and the amount of data in the dataset respectively; When the is higher than the threshold, it is considered that the service has failed.

6. The system for analyzing and blocking the fault propagation path of a distributed system based on large model technology according to claim 1, characterized in that: The fault metric module uses a linear structural causal model to identify the factors leading to service faults and the causal relationship between the factors, specifically: In a linear structural causal model, if the failure of service x causes the failure of service y, the causal relationship is expressed as y = wx + e, where e is a random disturbance term representing the impact of random events other than x on y, and w is the causal effect strength; for the failure events x1, x2,..., x of n services n , the causal relationship is in the following form: where, e1, e2,..., e n represent the corresponding random disturbance terms of x1, x2,..., x n , and w nn-1 represents the causal effect intensity of x n-1 on x n ; Infer the causal relationship of service fault events according to the DirectLiNGAM algorithm.

7. A system for analyzing and blocking the fault propagation path of a distributed system based on large model technology as claimed in claim 1, wherein: The service fault propagation graph constructed by the fault propagation graph construction module is a weighted directed graph, where the nodes represent faulty services, the direction of the edges represents the fault propagation direction, and the weights represent the causal effect intensity of the fault events.

8. The system for analyzing and blocking the fault propagation path of a distributed system based on large model technology according to claim 7, characterized in that: The fault propagation path identification module combines the service fault content output by the fault metric module, calculates the weights of the service fault propagation graph through the fault propagation path identification algorithm, traverses the service fault propagation graph, filters out all fault paths starting from the faulty service, calculates the causal effect intensity of the fault paths based on the weights of all the edges on the fault paths, sorts all the fault paths in descending order of the causal effect intensity. The higher the ranking of the fault path, the greater the propagation probability of the fault path.

9. A system for analyzing and blocking the fault propagation path of a distributed system based on large model technology as described in claim 1, characterized in that: The fault blocking strategy adopted by the fault blocking module is set according to the role of the service nodes, including: for applications that do not belong to the core business process, call the rate limiting function in the distributed system to eliminate the application; for applications that belong to the cache and there is a database at the back end, call the circuit breaking function in the distributed system to break through the application and switch all traffic to the database.

10. A method for analyzing and blocking the fault propagation path of a distributed system based on large model technology, characterized in that, It includes the following steps: S1: Collect various data of the distributed system for monitoring the distributed system, identify the faults that occur in the distributed system and issue alarms; S2: After receiving the alarm, measure a single service fault event and analyze the causal relationship between different service fault events through the service operation data of the distributed system to obtain the service fault content; S3: Label the nodes and edges in the distributed system to construct a service fault propagation graph; S4: According to the constructed service fault propagation graph and the output service fault content, identify the fault paths sorted by the propagation probability; S5: After receiving the fault paths identified by the fault propagation path identification module, adopt a preset fault blocking strategy to block the faults.

Images