A fault propagation prediction method, device and related equipment

Abstract

This application provides a fault propagation prediction method, apparatus, and related equipment, belonging to the field of information technology. An embodiment of this application provides a fault propagation prediction method, comprising: acquiring real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range; constructing a service causal graph based on the historical operation and maintenance data and a preset topology constraint matrix, wherein the topology constraint matrix is ​​used to determine potential causal relationships between the multiple nodes; adjusting the edge weights of the causal edges of the faulty nodes according to the real-time operation and maintenance data to obtain an updated service causal graph, wherein the faulty nodes are one or more nodes among the multiple nodes that have experienced a fault; and predicting the fault propagation probability of the fault among the multiple nodes based on the updated service causal graph. This fault propagation prediction method can improve the accuracy of fault propagation prediction.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a method, apparatus and related equipment for predicting fault propagation. Background Technology

[0002] Fault propagation prediction is crucial for analyzing the propagation of faults across multiple nodes in modern, complex IT systems. Currently, fault propagation analysis primarily relies on static service topology analysis to predict propagation paths and impact ranges, but this approach offers relatively low accuracy in predicting fault propagation. Summary of the Invention

[0003] This application provides a fault propagation prediction method, apparatus, and related equipment, which can solve the technical problem of low accuracy in fault propagation prediction.

[0004] In a first aspect, embodiments of this application provide a fault propagation prediction method, the method comprising:

[0005] Acquire real-time operation and maintenance data from multiple nodes and historical operation and maintenance data within a preset first time range;

[0006] A service cause-effect graph is constructed based on the historical operation and maintenance data and the preset topology constraint matrix. The topology constraint matrix is ​​used to determine the potential causal relationships between the multiple nodes.

[0007] The updated service causal graph is obtained by adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data. The faulty nodes are one or more nodes that have failed among the plurality of nodes.

[0008] Based on the updated service causal graph, the probability of the fault propagation at the multiple nodes is predicted respectively.

[0009] Optionally, the step of constructing a service cause-effect graph based on the historical operation and maintenance data and the preset topology constraint matrix includes:

[0010] The historical operation and maintenance data is preprocessed to obtain first intermediate data, and a service cause-effect graph is constructed based on the first intermediate data and a preset topology constraint matrix.

[0011] And / or,

[0012] The step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0013] The real-time operation and maintenance data is preprocessed to obtain second intermediate data. The edge weights of the causal edges of the faulty nodes are adjusted according to the second intermediate data to obtain the updated service causal graph.

[0014] The preprocessing includes at least one of the following: deduplication; filling in default values; removing abnormal data points that exceed a preset numerical range; and normalization.

[0015] Optionally, the step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0016] Obtain the current failure rate of the faulty node and the average failure rate of the multiple nodes within a preset second time range;

[0017] The edge weights of the causal edges of the faulty nodes are adjusted according to the ratio of the current failure rate to the average failure rate to obtain the updated service causal graph.

[0018] Optionally, predicting the probability of fault propagation across the multiple nodes based on the updated service causal graph includes:

[0019] Determine the probability of the fault propagating from the faulty node to adjacent nodes;

[0020] Based on the preset fault propagation termination condition, the transition probability, and the updated service causal graph, the fault propagation is simulated a preset number of times using Monte Carlo simulation to obtain the fault propagation frequency of the multiple nodes.

[0021] Based on the ratio of the fault propagation frequency to the preset number of times, the probability of the fault propagation at the multiple nodes is predicted respectively.

[0022] Optionally, the method further includes:

[0023] Determine the business impact weights of the multiple nodes respectively;

[0024] Based on the ratio of the business impact weight to the fault propagation probability, the risk cost ratio of each of the multiple nodes is determined, and the risk cost ratios of the multiple nodes are sorted by size to obtain the sorting result.

[0025] The total propagation risk is determined by the sum of the failure propagation probabilities of the multiple nodes, and the constraint range is determined by the product of the total propagation risk and the preset risk coverage threshold.

[0026] Based on the sorting results and the constraint range, determine the set of isolated nodes among the plurality of nodes;

[0027] Isolate the nodes in the set of isolated nodes.

[0028] Optionally, determining the business impact weights of the plurality of nodes includes:

[0029] The revenue contribution value, service level agreement penalty (SLA) risk value, and user impact scope of each of the multiple nodes are determined respectively.

[0030] The revenue contribution value, the SLA risk value, and the user impact range are normalized to determine the normalized revenue contribution value, the normalized SLA risk value, and the normalized user impact range value for the multiple nodes, respectively.

[0031] The revenue contribution normalization value, SLA risk normalization value, and user impact scope normalization value of the multiple nodes are weighted and summed according to preset weights to obtain the business impact weight of the multiple nodes.

[0032] Secondly, embodiments of this application provide a fault propagation prediction device, the device comprising:

[0033] The acquisition module is used to acquire real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range;

[0034] The first processing module is used to construct a service cause-effect graph based on the historical operation and maintenance data and a preset topology constraint matrix, wherein the topology constraint matrix is ​​used to determine the potential causal relationships between the multiple nodes.

[0035] The second processing module is used to adjust the edge weights of the causal edges of the faulty nodes according to the real-time operation and maintenance data to obtain an updated service causal graph, wherein the faulty node is one or more nodes that have failed among the plurality of nodes.

[0036] The third processing module is used to predict the probability of the fault propagation in the multiple nodes based on the updated service causal graph.

[0037] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the fault propagation prediction method as described in the first aspect.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fault propagation prediction steps as described in the first aspect.

[0039] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the fault propagation prediction method as described in the first aspect.

[0040] In this embodiment, real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range are acquired. A service causal graph is constructed based on the historical operation and maintenance data and a preset topology constraint matrix, where the topology constraint matrix is ​​used to determine potential causal relationships between the multiple nodes. The edge weights of the causal edges of the faulty nodes are adjusted according to the real-time operation and maintenance data to obtain an updated service causal graph, where the faulty nodes are one or more nodes that have experienced a failure. Based on the updated service causal graph, the probability of the fault propagation across the multiple nodes is predicted. Because the service causal graph can be updated according to real-time operation and maintenance data, it can reflect changes in causal relationships between nodes in a timely manner. Therefore, it can more accurately predict the probability of the fault propagation across the multiple nodes, improving the accuracy of fault propagation prediction. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a fault propagation prediction method provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of a Monte Carlo simulation of fault propagation provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the structure of a fault propagation prediction device provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, "and / or" in this application indicates at least one of the connected objects. For example, the scope of protection of "A and / or B" covers at least three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. Additionally, the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0048] See Figure 1 , Figure 1 This is a flowchart of a fault propagation prediction method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0049] Step 101: Obtain real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range;

[0050] The plurality of nodes may be multiple service nodes or application nodes in systems such as cloud services and / or microservice architectures;

[0051] The real-time operation and maintenance data can be collected through an Application Programming Interface (API) gateway, service mesh, or distributed tracing system, and may specifically include key indicator data such as API call volume, request error rate, and / or response latency; the data format can be time series data, and can be recorded at a granularity of per second or per minute;

[0052] The historical operation and maintenance data can originate from the log systems, monitoring platforms (such as the Prometheus monitoring system and / or visual information system management tools) and operation and maintenance management platforms of the multiple nodes; this type of data can include fields such as service call relationships, fault occurrence time, fault type, impact scope, and recovery time, and its format can be a structured JavaScript Object Notation (JSON) file or a comma-separated values ​​(CSV) file.

[0053] The data structures of the historical operation and maintenance data and the real-time operation and maintenance data can be the same;

[0054] The preset first time range can be a time range specified by those skilled in the art, or a scrolling event window specified by those skilled in the art (e.g., the last two months or the last week).

[0055] In this step, real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range are obtained to provide a data foundation for building and updating the service cause-effect graph in step 102.

[0056] Step 102: Construct a service cause-effect graph based on the historical operation and maintenance data and the preset topology constraint matrix. The topology constraint matrix is ​​used to determine the potential causal relationships between the multiple nodes.

[0057] The preset topological constraint matrix can be a constraint matrix constructed based on existing knowledge or prior information in the field. The topological constraint matrix can be used to determine the potential causal relationship between the multiple nodes. For example, if it is known that service A will not directly affect service C, the corresponding position in the topological constraint matrix can be set to 0, thereby reducing the search space and improving the convergence speed.

[0058] The construction of the service causal graph based on the historical operation and maintenance data and the preset topological constraint matrix can be specifically as follows: Based on the historical operation and maintenance data, an undirected graph is constructed using the Peter-Clark (PC) algorithm (the PC algorithm can be used to learn the causal structure between variables from observation data). The PC algorithm can obtain an undirected graph by gradually deleting insignificant edges; the undirected graph, combined with directional constraints (such as the preset topological constraint matrix), determines the causal direction, thereby obtaining the service causal graph;

[0059] The graph nodes of the service cause-effect graph can be used to represent the multiple nodes;

[0060] In the service causal graph, causal edges can be determined based on predefined interaction relationships or preset monitoring indicators, and the edge weights are used to quantify the strength of causal influence.

[0061] For example, the causal relationship of a fault propagation at a certain node in the service causal graph can be referenced by the following first formula:

[0062]

[0063] In the first formula, This can represent the causal relationship of fault propagation at node V in the service causal graph; It can represent key metrics for node V, such as API call volume, request error rate, response latency, remote dictionary server Redis latency, and order service error rate; It can be used to represent the causal edges and causal directions in the service causal graph; Used to represent edge weights (which can be in the form of a weight matrix), it can be used to quantify the influence strength of causal edges.

[0064] It is worth noting that the fault propagation can be the same fault propagating between different nodes, or it can be cascading fault propagation within a node (i.e., one fault on the same node triggers another fault), or it can be heterogeneous fault propagation across nodes (e.g., one fault on one node propagates and triggers another fault on another node).

[0065] In the first formula above, taking the propagation of cascading failures within a node as an example, a failure i in node V (e.g., a Redis latency failure) may trigger another failure j (order error rate). It can be used It can be indicated that it can be used This means that a 1-millisecond increase in Redis latency will increase the order error rate by 0.82%.

[0066] In this step, a service cause-effect graph is constructed based on the historical operation and maintenance data and the preset topology constraint matrix, which provides a data basis for the propagation of the fault on the multiple nodes;

[0067] Step 103: Adjust the edge weights of the causal edges of the faulty nodes according to the real-time operation and maintenance data to obtain the updated service causal graph, wherein the faulty node is one or more nodes that have failed among the plurality of nodes.

[0068] In this step, the edge weights of the causal edges of the faulty nodes are adjusted using real-time operation and maintenance data to obtain an updated service causal graph. The service causal graph is updated according to the real-time operation and maintenance data, which can more accurately determine the current state of fault propagation among multiple nodes.

[0069] Step 104: Based on the updated service cause-effect graph, predict the probability of the fault propagation on the multiple nodes.

[0070] In this step, since the updated service cause-effect graph can be continuously updated based on the real-time operation and maintenance data, it can more accurately determine the current state of fault propagation in multiple nodes, and thus more accurately predict the probability of fault propagation in multiple nodes.

[0071] In this embodiment, since the service causal graph can be updated according to real-time operation and maintenance data, it can reflect the changes in causal relationships between nodes in a timely manner; therefore, it can more accurately predict the probability of the fault propagation in multiple nodes and improve the accuracy of fault propagation prediction.

[0072] In some implementations, constructing a service cause-effect graph based on the historical operation and maintenance data and a preset topology constraint matrix includes:

[0073] The historical operation and maintenance data is preprocessed to obtain first intermediate data, and a service cause-effect graph is constructed based on the first intermediate data and a preset topology constraint matrix.

[0074] And / or,

[0075] The step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0076] The real-time operation and maintenance data is preprocessed to obtain second intermediate data. The edge weights of the causal edges of the faulty nodes are adjusted according to the second intermediate data to obtain the updated service causal graph.

[0077] The preprocessing includes at least one of the following: deduplication; filling in default values; removing abnormal data points that exceed a preset numerical range; and normalization.

[0078] The deduplication operation may involve deleting duplicate data / records / elements from the historical and / or real-time operation and maintenance data, retaining only one copy to avoid duplicate counting.

[0079] The default value filling can be performed by filling the default values ​​in the historical operation and maintenance data and / or real-time operation and maintenance data with default values ​​according to their respective preset rules or preset values.

[0080] The removal of abnormal data points exceeding the preset value range may be the removal of abnormal data points in the historical operation and maintenance data and / or real-time operation and maintenance data that clearly exceed their respective preset value ranges. The preset value range may be a preset value range set by those skilled in the art according to preset rules or according to actual needs.

[0081] For example, because in a normal distribution, 99.73% of the data fall within... Within the range ( Used to represent the mean of a dataset. The standard deviation of the dataset is used to represent the probability that the data exceeds this range (less than 0.3%). Therefore, data exceeding this range can be considered acceptable. Data points within a certain range are considered outliers and are removed.

[0082] The preset value range corresponding to the historical operation and maintenance data can be... Within the range, This represents the average of the historical operation and maintenance data. The standard deviation used to represent historical operation and maintenance data can be used to represent the historical operation and maintenance data exceeding... Removed as outlier data points;

[0083] Alternatively, historical operation and maintenance data within a predetermined time range (this predetermined time range can be a specific period determined by those skilled in the art, or it can be a rolling time window (e.g., the most recent 30 days)) can be used. The range is used as a preset data range;

[0084] The preset value range corresponding to the real-time operation and maintenance data can be... Within the range, This represents the average value of the real-time operation and maintenance data. The standard deviation used to represent real-time operation and maintenance data can be used to exceed the real-time operation and maintenance data. Excluded as abnormal data points; it is understood that if the amount of real-time operation and maintenance data is insufficient, a preset value range corresponding to the historical operation and maintenance data can also be used. (range) is used as the preset numerical range of the real-time operation and maintenance data;

[0085] It should be noted that, in the process of determining the preset value range of the historical operation and maintenance data and / or the preset value range of the real-time operation and maintenance data based on the historical operation and maintenance data, the historical operation and maintenance data can be the full historical operation and maintenance data, or it can be the historical operation and maintenance data within a predetermined time range (the predetermined time range can be a certain period of time determined by those skilled in the art, or it can be a rolling time window (e.g., the most recent 30 days)).

[0086] For example, a standardized score algorithm (Z-score algorithm) can be used to determine the standardized score of a dataset (the dataset can be the historical operation and maintenance data and / or real-time operation and maintenance data). The specific calculation of the standardized score Z can be found in the second formula below:

[0087]

[0088] In the second formula, Z can represent the standardized score, specifically the standard deviation multiple of the original data point x from the mean u of the dataset (e.g., historical operation and maintenance data and / or real-time operation and maintenance data); x can represent the data point in the dataset; u can represent the mean of the dataset. It can represent the standard deviation of the dataset; the u and above A sliding window can be used to update the data, for example, the 1000 most recent data points;

[0089] The second formula can be used to determine the standard scores of data points in historical and / or real-time operation and maintenance data, respectively; where the number of data points in the historical and / or real-time operation and maintenance data exceeds 3. If the range is too wide, it is an outlier and the data point can be removed.

[0090] The normalization process can be applied to the historical and / or real-time operation and maintenance data. Specifically, it can be applied to numerical characteristics such as API call volume, request error rate, and response latency in the historical and / or real-time operation and maintenance data. Refer to the following normalization formula (Min-Max normalization formula):

[0091]

[0092] In the normalization formula, X can represent the original value; This can represent the minimum value of a dataset (such as the historical or real-time operation and maintenance data mentioned above); It can represent the maximum value of a dataset (such as the historical or real-time operation and maintenance data mentioned above). It can represent the normalized value (the value range can be [0, 1]);

[0093] In the second formula, the data boundary can also be updated incrementally, such as updating it every 100 new data points. and ;

[0094] The historical and / or real-time operation and maintenance data are normalized. Independent normalization intervals can be used according to different service types (to avoid diluting important service indicators). For example, a more sensitive interval is used for payment services. An error rate of more than 10% in payment services is considered serious, while other ordinary services have a higher tolerance.

[0095] In this step, the historical and / or real-time operation and maintenance data are preprocessed. While ensuring the authenticity and representativeness of the data, redundancy and noise can be eliminated, missing information can be supplemented, and the interference of outliers on the overall structure can be reduced. Data of different dimensions can also be mapped to a unified scale, thereby obtaining a cleaner, more stable, and more consistent data set, improving the consistency, stability, and accuracy of the service cause-effect graph, and thus improving the prediction accuracy of fault propagation.

[0096] In some implementations, the step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0097] Obtain the current failure rate of the faulty node and the average failure rate of the multiple nodes within a preset second time range;

[0098] The edge weights of the causal edges of the faulty nodes are adjusted according to the ratio of the current failure rate to the average failure rate to obtain the updated service causal graph.

[0099] In this embodiment, the edge weights of the causal edges of the faulty nodes are adjusted according to the ratio of the current failure rate to the average failure rate, so that the service causal graph can be updated more timely. Moreover, the updated service causal graph can reflect the changes in the causal relationship of fault propagation of each node more timely, thereby more accurately predicting the probability of fault propagation of the fault in multiple nodes and improving the accuracy of fault propagation prediction.

[0100] For example, adjusting the edge weights of the causal edges of the faulty nodes based on the ratio of the current failure rate to the average failure rate can be done using the following third formula:

[0101]

[0102] In the third formula, This can represent the edge weight of a causal edge at time t (e.g., the current time). It can represent the edge weight of the initial causal edge; It can represent the preset sensitivity coefficient ( The value can be 0.3. Too high a setting will amplify noise, while too low a setting will result in a delayed response. It can represent the current failure rate of node i at time t (e.g., the current time), for example, the current order error rate is 0.15; It can represent the average failure rate of multiple nodes within a preset second time range. The preset second time range can be a certain time period determined by those skilled in the art, or it can be a rolling time window (e.g., the most recent 30 days). For example, the average error rate of orders over 30 days is 0.05.

[0103] In the third formula above, In certain situations, amplifying the edge weight of a causal edge can strengthen fault propagation; In this case, the edge weight decay of causal edges can suppress fault propagation.

[0104] In some implementations, predicting the probability of fault propagation across multiple nodes based on the updated service causal graph includes:

[0105] Determine the probability of the fault propagating from the faulty node to adjacent nodes;

[0106] Based on the preset fault propagation termination condition, the transition probability, and the updated service causal graph, the fault propagation is simulated a preset number of times using Monte Carlo simulation to obtain the fault propagation frequency of the multiple nodes.

[0107] Based on the ratio of the fault propagation frequency to the preset number of times, the probability of the fault propagation at the multiple nodes is predicted respectively.

[0108] Specifically, the probability of the fault propagating from the faulty node to its neighboring nodes can be determined based on a preset graph traversal algorithm (e.g., a preset random walk transition rule). Taking the propagation of the fault from node i to neighboring node j as an example, the probability of the transition can be calculated using the following fourth formula:

[0109]

[0110] In the fourth formula, It can represent the transition probability from a fault propagating from node i to its neighboring node j; Let k represent the set of neighboring nodes of node i (e.g., the set of outgoing neighbors); k represents The k-th node; It can represent the edge weight after causal edge adjustment, that is, the probability that a fault propagates from node i to node j.

[0111] The preset fault propagation termination condition can be the preset maximum depth of fault propagation (for example, it can propagate up to five times between nodes, at which point the maximum depth L=5) and / or entering a node with no outgoing edges.

[0112] Specifically, the method of performing a preset number of Monte Carlo simulations on the propagation of the fault based on the preset fault propagation termination condition, the transition probability, and the updated service causality graph can be found in [reference needed]. Figure 2 , Figure 2 This is a schematic diagram of a Monte Carlo simulation of fault propagation provided in an embodiment of this application, as shown below. Figure 2 As shown:

[0113] The fault originates from the fault source (i.e., the fault node). According to the transition probability Select neighboring nodes (i.e., adjacent nodes). To simulate fault propagation;

[0114] The fault is from According to the transition probability Select neighboring nodes (i.e., adjacent nodes) Simulate fault propagation;

[0115] The simulation continues until the maximum depth L is reached (or other preset termination conditions are met), at which point the fault propagation terminates, completing one Monte Carlo simulation of the fault propagation.

[0116] The aforementioned fault propagation can be simulated a preset number of Monte Carlo simulations, and these preset number of Monte Carlo simulations can be independent of each other; the specific preset number can be set as needed by those skilled in the art.

[0117] After the Monte Carlo simulation is completed, the number of times each node is visited records the frequency of fault propagation to that node, that is, the fault propagation frequency of the multiple nodes is obtained respectively; the fault propagation probability of each node can be predicted by the ratio of the fault propagation frequency to the preset number of visits.

[0118] In this embodiment, based on the preset fault propagation termination condition, the transition probability, and the updated service causal graph, the propagation of the fault is simulated a preset number of times using Monte Carlo simulation. Even when the service causal graph is multi-dimensional, the node distribution is complex, and / or the path dependency is complex, a statistically significant fault propagation probability can still be obtained. Moreover, the preset number of Monte Carlo simulations for the propagation of the fault is simple to implement and easy to parallelize, which can greatly improve the prediction speed of the fault propagation probability.

[0119] In some embodiments, the method further includes:

[0120] Determine the business impact weights of the multiple nodes respectively;

[0121] Based on the ratio of the business impact weight to the fault propagation probability, the risk cost ratio of each of the multiple nodes is determined, and the risk cost ratios of the multiple nodes are sorted by size to obtain the sorting result.

[0122] The total propagation risk is determined by the sum of the failure propagation probabilities of the multiple nodes, and the constraint range is determined by the product of the total propagation risk and the preset risk coverage threshold.

[0123] Based on the sorting results and the constraint range, determine the set of isolated nodes among the plurality of nodes;

[0124] Isolate the nodes in the set of isolated nodes.

[0125] The business impact weight can be set based on multiple dimensions such as economic losses caused by the failure of a certain node (such as lost orders and compensation), the criticality of the node's service (such as the different criticalities of the core payment link and auxiliary log service), the scope of affected users (such as the different scope of impact between affecting all users and only affecting internal employees), and / or the level of affected users.

[0126] The total propagation risk is determined by summing the failure propagation probabilities of the multiple nodes, and the constraint range is determined by multiplying the total propagation risk by a preset risk coverage threshold. You can refer to the fifth formula below:

[0127]

[0128] In the fifth formula, It can represent a set of isolated nodes; It can represent The first in One node; It can represent the first The probability of fault propagation in each node; This can represent a preset risk coverage threshold (e.g.) (It can be 80%, or it can be set as needed by those skilled in the art). It can represent The sum of the failure propagation probabilities of the multiple nodes; It can represent the range of constraints;

[0129] In the fifth formula, simultaneously satisfying , It can represent a node The weight of business influence;

[0130] Taking node i as an example, the ;

[0131] In this embodiment, the set of isolated nodes among the plurality of nodes is determined based on the sorting result and the constraint range. The set of isolated nodes can be determined more accurately based on the constraint range, and the nodes in the set of isolated nodes can be isolated to achieve better risk and cost control, while reducing the ability of fault propagation.

[0132] Specifically, based on the sorting results and the constraint range, a set of isolated nodes is determined among the multiple nodes. Nodes with a high risk-cost ratio (i.e., nodes whose unit cost can cover more risks) can be preferentially selected to enter the set of isolated nodes until the requirements of the constraint range are met. This can cover more risk nodes under a preset cost condition and reduce the fault propagation capability.

[0133] In some implementations, determining the service impact weights of the plurality of nodes includes:

[0134] The revenue contribution value, Service Level Agreement (SLA) penalty risk value, and user impact scope of the aforementioned nodes are determined respectively.

[0135] The revenue contribution value, the SLA risk value, and the user impact range are normalized to determine the normalized revenue contribution value, the normalized SLA risk value, and the normalized user impact range value for the multiple nodes, respectively.

[0136] The revenue contribution normalization value, SLA risk normalization value, and user impact scope normalization value of the multiple nodes are weighted and summed according to preset weights to obtain the business impact weight of the multiple nodes.

[0137] The revenue contribution value can be the amount or percentage of revenue directly generated by a specific service or product, and can be used to quantify the economic importance of the node.

[0138] The SLA risk value can be the potential penalty amount that needs to be paid for failing to meet the standards agreed upon in the SLA (Service Level Agreement), and can be used to quantify the criticality or importance of a node;

[0139] The user's influence range can be the scope and / or level at which the node influences the user;

[0140] Taking revenue contribution value as an example, the normalized revenue contribution values ​​of the multiple nodes can be determined using the following normalization formula:

[0141]

[0142] in, This can represent the normalized value of node i's revenue contribution; This can represent the revenue contribution value of node i; This can represent the maximum revenue contribution of node i; This can represent the minimum revenue contribution of node i;

[0143] Regarding the SLA risk value and the scope of user influence Alternatively, the normalization formulas described above can be used to determine the normalized values ​​of SLA risk separately. Normalized value of user influence range ;

[0144] The preset harmonic weights can be used to adjust the revenue contribution normalization value, SLA risk normalization value, and user impact range normalization value, respectively (the specific size of the harmonic weights can be set as needed by those skilled in the art).

[0145] For example, taking node i as an example, the business impact weight of node i can be obtained by referring to the following sixth formula:

[0146]

[0147] In the sixth formula, It can represent a node The weight of business influence; , and Nodes can be represented separately. The normalized values ​​of revenue contribution, SLA risk, and user impact scope; , and They can be represented separately , and Each has its corresponding harmonic weight.

[0148] It is understandable that the multiple nodes can also determine their respective business impact weights by referring to the method of node i.

[0149] In this embodiment, multiple business indicator data (such as the aforementioned revenue contribution value, SLA risk value, and user impact range) are comprehensively calculated into a single business impact weight to quantify the impact of node business. This can more accurately reflect the multi-dimensional impact characteristics of the node, thereby comprehensively reflecting the criticality and importance of the node in the overall business system.

[0150] For example, the fault propagation prediction method provided in this application can be used as an example of a microservice system applied to an e-commerce platform to illustrate its specific implementation process:

[0151] Data Acquisition and Preprocessing: The microservice system may include modules such as user service, order service, payment service, and inventory service; historical operation and maintenance data can be stored in a distributed file system (Hadoop Distributed File System, HDFS); real-time operation and maintenance data can be collected through a service mesh (such as the Istio platform) and written to a time-series database (such as the InfluxDB database); in the data preprocessing stage, fields such as error rate and latency can be normalized and used as input for subsequent cause-effect graph construction;

[0152] Cause-effect graph construction: The PC algorithm is used to construct a service cause-effect graph. The initial graph contains all possible edges. A topological constraint matrix is ​​introduced. For example, according to the business logic, "User service → Order service" is set as a directed edge, while "Inventory service → Payment service" is set as an undirected edge, thereby reducing the search space. Finally, a cause-effect graph containing 12 nodes and 28 edges is obtained.

[0153] Fault propagation prediction; for example, when a fault is detected in the "order service", the microservice system reads real-time traffic data and finds that its error rate has increased by 40% compared with the historical average and the latency has increased by 30%; based on real-time operation and maintenance data, the edge weights of causal edges are updated, for example, the weight of "order service → payment service" is increased from 0.6 to 0.8; then, Monte Carlo simulation is performed, with a total of 1000 iterations, to determine the fault propagation frequency of each node, and then obtain the fault propagation probability of each node.

[0154] Isolation strategy generation: The set of isolation nodes can be determined based on the probability of fault propagation and the weight of business impact. Assuming that the propagation risks of "payment service" and "inventory service" are 0.72 and 0.65 respectively, and the business impact weights are 0.9 and 0.8 respectively, the isolation nodes can be determined as "payment service" and "inventory service" through a greedy algorithm, covering 82% of the propagation risk, with a total cost of 1.7 units.

[0155] In this embodiment, accurate prediction and efficient isolation of fault propagation paths in complex service systems are achieved, effectively improving the robustness and recovery capabilities of microservice systems.

[0156] It should be noted that the fault propagation prediction method disclosed herein can be executed by an electronic device, that is, all steps included in the above method are executed by the electronic device, which may be a server, computer, cloud service node or mobile phone or other electronic device.

[0157] See Figure 3 , Figure 3 This is a schematic diagram of the structure of a fault propagation prediction device provided in an embodiment of this application, as shown below. Figure 3 As shown, the fault propagation prediction device 300 includes:

[0158] The acquisition module 301 is used to acquire real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range;

[0159] The first processing module 302 is used to construct a service cause-effect graph based on the historical operation and maintenance data and a preset topology constraint matrix, wherein the topology constraint matrix is ​​used to determine the potential causal relationships between the multiple nodes.

[0160] The second processing module 303 is used to adjust the edge weights of the causal edges of the faulty nodes according to the real-time operation and maintenance data to obtain an updated service causal graph, wherein the faulty node is one or more nodes that have failed among the plurality of nodes.

[0161] The third processing module 304 is used to predict the probability of the fault propagation in the multiple nodes based on the updated service causal graph.

[0162] Optionally, the step of constructing a service cause-effect graph based on the historical operation and maintenance data and the preset topology constraint matrix includes:

[0163] The historical operation and maintenance data is preprocessed to obtain first intermediate data, and a service cause-effect graph is constructed based on the first intermediate data and a preset topology constraint matrix.

[0164] And / or,

[0165] The step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0166] The real-time operation and maintenance data is preprocessed to obtain second intermediate data. The edge weights of the causal edges of the faulty nodes are adjusted according to the second intermediate data to obtain the updated service causal graph.

[0167] The preprocessing includes at least one of the following: deduplication; filling in default values; removing abnormal data points that exceed a preset numerical range; and normalization.

[0168] Optionally, the step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0169] Obtain the current failure rate of the faulty node and the average failure rate of the multiple nodes within a preset second time range;

[0170] The edge weights of the causal edges of the faulty nodes are adjusted according to the ratio of the current failure rate to the average failure rate to obtain the updated service causal graph.

[0171] Optionally, predicting the probability of fault propagation across the multiple nodes based on the updated service causal graph includes:

[0172] Determine the probability of the fault propagating from the faulty node to adjacent nodes;

[0173] Based on the preset fault propagation termination condition, the transition probability, and the updated service causal graph, the fault propagation is simulated a preset number of times using Monte Carlo simulation to obtain the fault propagation frequency of the multiple nodes.

[0174] Based on the ratio of the fault propagation frequency to the preset number of times, the probability of the fault propagation at the multiple nodes is predicted respectively.

[0175] Optionally, the fault propagation prediction device 300 may also include a fourth processing module 305;

[0176] The fourth processing module 305 is used to determine the business impact weights of the multiple nodes respectively;

[0177] Based on the ratio of the business impact weight to the fault propagation probability, the risk cost ratio of each of the multiple nodes is determined, and the risk cost ratios of the multiple nodes are sorted by size to obtain the sorting result.

[0178] The total propagation risk is determined by the sum of the failure propagation probabilities of the multiple nodes, and the constraint range is determined by the product of the total propagation risk and the preset risk coverage threshold.

[0179] Based on the sorting results and the constraint range, determine the set of isolated nodes among the plurality of nodes;

[0180] Isolate the nodes in the set of isolated nodes.

[0181] Optionally, determining the business impact weights of the plurality of nodes includes:

[0182] The revenue contribution value, service level agreement penalty (SLA) risk value, and user impact scope of each of the multiple nodes are determined respectively.

[0183] The revenue contribution value, the SLA risk value, and the user impact range are normalized to determine the normalized revenue contribution value, the normalized SLA risk value, and the normalized user impact range value for the multiple nodes, respectively.

[0184] The revenue contribution normalization value, SLA risk normalization value, and user impact scope normalization value of the multiple nodes are weighted and summed according to preset weights to obtain the business impact weight of the multiple nodes.

[0185] The fault propagation prediction device 300 is capable of implementing the various processes applied to the various embodiments of the fault propagation prediction method described above. The technical features correspond one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0186] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described fault propagation prediction method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0187] For details, see Figure 4 This application also provides an electronic device, including a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405, and a memory 406.

[0188] The transceiver 402 is used to acquire real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range;

[0189] The processor 405 is used to construct a service cause-effect graph based on the historical operation and maintenance data and a preset topology constraint matrix, wherein the topology constraint matrix is ​​used to determine the potential causal relationships between the multiple nodes.

[0190] The updated service causal graph is obtained by adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data. The faulty nodes are one or more nodes that have failed among the plurality of nodes.

[0191] Based on the updated service causal graph, the probability of the fault propagation at the multiple nodes is predicted respectively.

[0192] Optionally, the step of constructing a service cause-effect graph based on the historical operation and maintenance data and the preset topology constraint matrix includes:

[0193] The historical operation and maintenance data is preprocessed to obtain first intermediate data, and a service cause-effect graph is constructed based on the first intermediate data and a preset topology constraint matrix.

[0194] And / or,

[0195] The step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0196] The real-time operation and maintenance data is preprocessed to obtain second intermediate data. The edge weights of the causal edges of the faulty nodes are adjusted according to the second intermediate data to obtain the updated service causal graph.

[0197] The preprocessing includes at least one of the following: deduplication; filling in default values; removing abnormal data points that exceed a preset numerical range; and normalization.

[0198] Optionally, the step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes:

[0199] Obtain the current failure rate of the faulty node and the average failure rate of the multiple nodes within a preset second time range;

[0200] The edge weights of the causal edges of the faulty nodes are adjusted according to the ratio of the current failure rate to the average failure rate to obtain the updated service causal graph.

[0201] Optionally, predicting the probability of fault propagation across the multiple nodes based on the updated service causal graph includes:

[0202] Determine the probability of the fault propagating from the faulty node to adjacent nodes;

[0203] Based on the preset fault propagation termination condition, the transition probability, and the updated service causal graph, the fault propagation is simulated a preset number of times using Monte Carlo simulation to obtain the fault propagation frequency of the multiple nodes.

[0204] Based on the ratio of the fault propagation frequency to the preset number of times, the probability of the fault propagation at the multiple nodes is predicted respectively.

[0205] Optionally, the processor 405 is further configured to determine the service impact weights of the plurality of nodes respectively;

[0206] Based on the ratio of the business impact weight to the fault propagation probability, the risk cost ratio of each of the multiple nodes is determined, and the risk cost ratios of the multiple nodes are sorted by size to obtain the sorting result.

[0207] The total propagation risk is determined by the sum of the failure propagation probabilities of the multiple nodes, and the constraint range is determined by the product of the total propagation risk and the preset risk coverage threshold.

[0208] Based on the sorting results and the constraint range, determine the set of isolated nodes among the plurality of nodes;

[0209] Isolate the nodes in the set of isolated nodes.

[0210] Optionally, determining the business impact weights of the plurality of nodes includes:

[0211] The revenue contribution value, service level agreement penalty (SLA) risk value, and user impact scope of each of the multiple nodes are determined respectively.

[0212] The revenue contribution value, the SLA risk value, and the user impact range are normalized to determine the normalized revenue contribution value, the normalized SLA risk value, and the normalized user impact range value for the multiple nodes, respectively.

[0213] The revenue contribution normalization value, SLA risk normalization value, and user impact scope normalization value of the multiple nodes are weighted and summed according to preset weights to obtain the business impact weight of the multiple nodes.

[0214] exist Figure 4 In this context, a bus architecture (represented by bus 401) is used. Bus 401 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 405 is transmitted over a wireless medium via antenna 403, which further receives data and transmits data to processor 405.

[0215] Processor 405 is responsible for managing bus 401 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 406 can be used to store data used by processor 405 during operation.

[0216] Optionally, the processor 405 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0217] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described fault propagation prediction method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0218] This application also provides a computer program product, including computer instructions. When executed by a processor, these computer instructions implement the various processes of the above-described fault propagation prediction method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0219] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0220] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0221] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method of failure propagation prediction, characterized by, The method includes: Acquire real-time operation and maintenance data from multiple nodes and historical operation and maintenance data within a preset first time range; A service cause-effect graph is constructed based on the historical operation and maintenance data and the preset topology constraint matrix. The topology constraint matrix is ​​used to determine the potential causal relationships between the multiple nodes. The updated service causal graph is obtained by adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data. The faulty nodes are one or more nodes that have failed among the plurality of nodes. Based on the updated service causal graph, the probability of the fault propagation at the multiple nodes is predicted respectively.

2. The method of claim 1, wherein, The construction of the service cause-effect graph based on the historical operation and maintenance data and the preset topology constraint matrix includes: The historical operation and maintenance data is preprocessed to obtain first intermediate data, and a service cause-effect graph is constructed based on the first intermediate data and a preset topology constraint matrix. And / or, The step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes: The real-time operation and maintenance data is preprocessed to obtain second intermediate data. The edge weights of the causal edges of the faulty nodes are adjusted according to the second intermediate data to obtain the updated service causal graph. The preprocessing includes at least one of the following: deduplication; filling in default values; removing abnormal data points that exceed a preset numerical range; and normalization.

3. The method of claim 1, wherein, The step of adjusting the edge weights of the causal edges of the faulty nodes based on the real-time operation and maintenance data to obtain the updated service causal graph includes: Obtain the current failure rate of the faulty node and the average failure rate of the multiple nodes within a preset second time range; The edge weights of the causal edges of the faulty nodes are adjusted according to the ratio of the current failure rate to the average failure rate to obtain the updated service causal graph.

4. The method of claim 1, wherein, The step of predicting the probability of fault propagation across multiple nodes based on the updated service causal graph includes: Determine the probability of the fault propagating from the faulty node to adjacent nodes; Based on the preset fault propagation termination condition, the transition probability, and the updated service causal graph, the fault propagation is simulated a preset number of times using Monte Carlo simulation to obtain the fault propagation frequency of the multiple nodes. Based on the ratio of the fault propagation frequency to the preset number of times, the probability of the fault propagation at the multiple nodes is predicted respectively.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Determine the business impact weights of the multiple nodes respectively; Based on the ratio of the business impact weight to the fault propagation probability, the risk cost ratio of each of the multiple nodes is determined, and the risk cost ratios of the multiple nodes are sorted by size to obtain the sorting result. The total propagation risk is determined by the sum of the failure propagation probabilities of the multiple nodes, and the constraint range is determined by the product of the total propagation risk and the preset risk coverage threshold. Based on the sorting results and the constraint range, determine the set of isolated nodes among the plurality of nodes; Isolate the nodes in the set of isolated nodes.

6. The method of claim 5, wherein, The step of determining the business impact weights of the multiple nodes includes: The revenue contribution value, service level agreement penalty (SLA) risk value, and user impact scope of each of the multiple nodes are determined respectively. The revenue contribution value, the SLA risk value, and the user impact range are normalized to determine the normalized revenue contribution value, the normalized SLA risk value, and the normalized user impact range value for the multiple nodes, respectively. The revenue contribution normalization value, SLA risk normalization value, and user impact scope normalization value of the multiple nodes are weighted and summed according to preset weights to obtain the business impact weight of the multiple nodes.

7. A fault propagation prediction device, characterized in that, The device includes: The acquisition module is used to acquire real-time operation and maintenance data of multiple nodes and historical operation and maintenance data within a preset first time range; The first processing module is used to construct a service cause-effect graph based on the historical operation and maintenance data and a preset topology constraint matrix, wherein the topology constraint matrix is ​​used to determine the potential causal relationships between the multiple nodes. The second processing module is used to adjust the edge weights of the causal edges of the faulty nodes according to the real-time operation and maintenance data to obtain an updated service causal graph, wherein the faulty node is one or more nodes that have failed among the plurality of nodes. The third processing module is used to predict the probability of the fault propagation in the multiple nodes based on the updated service causal graph.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 6.

Images