A full-link intelligent fault simulation and evaluation defense method in a micro-service scenario

Abstract

The application relates to the technical field of micro-service operation and maintenance, and discloses a full-link intelligent fault simulation and evaluation defense method in a micro-service scenario. The method comprises the following steps: constructing a dynamic dependency graph based on a service registration center and real-time communication traffic; matching and backtracking historical faults according to the graph, generating a fault injection point list and a propagation path set perceived by the graph; in an isolated environment, driving a programmable agent to perform multidimensional fault injection, and synchronously collecting full-quantity system response signals; performing multidimensional deviation calculation on the signals and a baseline to form observation records containing deviation intensity and propagation rate; training a graph neural network model with time sequence dependency understanding capability based on the records, which is used for fault mode identification and evolution prediction of an online real-time link, and outputs a risk evaluation report. The method improves the authenticity of fault simulation and the initiative of risk prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of micro-service operation and maintenance, in particular to a full-link intelligent fault simulation and evaluation defense method in a micro-service scenario. BACKGROUND

[0002] In the stability guarantee practice of micro-service architecture, fault testing is an important link to verify the system elasticity. The existing technology usually constructs a static service dependency model according to a preset service configuration list or a periodically generated architecture snapshot, and designs a fault injection scene based on the model. Due to the dynamic registration and discovery of micro-service instances, the elastic scaling and the traffic changes, such a static model is difficult to accurately reflect the real-time changes of the calling topology and the communication frequency between the services in the online services. Selecting the fault injection point based on outdated or incomplete dependency information makes it difficult for the simulation experiment to cover the key links and core bottlenecks in the current real running environment, reducing the pertinence and effectiveness of the drill.

[0003] When observing the fault injection results, the existing technology mainly relies on monitoring the performance indicators of a single service instance or tracing a preset single calling link. This method can only capture the local and direct manifestations of the fault, and cannot depict how an initial fault triggers a cascading effect in a complex service calling network from the overall system level. It also lacks quantitative description of the fault propagation range, path order and influence diffusion dynamics. The discreteness and limitation of the observation data make it difficult to accurately evaluate the actual risk level of different faults on the global business based on the drill results, and also cannot provide accurate and systematic basis for the optimization of defense strategies.

[0004] The dynamics of the micro-service environment requires that the planning of fault simulation must be closely related to the real-time dependency state of the system. At the same time, it is necessary to break through the isolated observation of fault phenomena and establish a method that can quantify the fault propagation behavior from the globally collected signals and predict the potential risk pattern recognition and evolution trend of the online running link. SUMMARY

[0005] The purpose of the present application is to provide a full-link intelligent fault simulation and evaluation defense method in a micro-service scenario to solve the problems raised in the background art.

[0006] To achieve the above purpose, the present application provides a full-link intelligent fault simulation and evaluation defense method in a micro-service scenario, which comprises:

[0007] Based on the service registration center and the inter-service communication traffic, a real-time dependency graph containing hierarchical structure and calling frequency is constructed;

[0008] According to the real-time dependency graph, the historical fault events are graphically matched and backtracked to generate a graph-aware fault injection point list and an influence propagation path set.

[0009] In a non-production isolated environment, according to the graph-aware fault injection point list and the influence propagation path set, a programmable agent is driven to perform multi-dimensional fault injection on the microservice instances, and full-quantity system response signals during the injection process are synchronously collected;

[0010] Multi-dimensional deviation calculation is performed on the full-quantity system response signals and preset baseline signals to form a fault simulation observation record containing signal deviation intensity and propagation rate;

[0011] Based on the fault simulation observation record, a graph neural network model with time sequence dependence understanding ability is trained, and the trained graph neural network model is used to perform fault mode recognition and evolution prediction on online real-time link data, and output a fault risk assessment report.

[0012] Preferably, the real-time dependency graph containing hierarchical structure and call frequency is constructed based on the service registry center and inter-service communication traffic, including:

[0013] The service instance online and offline events of the service registry center are continuously monitored, the network address, service identifier and metadata label of the service instance are extracted, and a service node basic information set is formed;

[0014] The inter-service communication traffic is bypassed to capture, the application layer protocol is parsed, the caller service identifier, the called service identifier, the call timestamp, the response code and the call time consumption are extracted, and an inter-service call record stream is formed;

[0015] The inter-service call record stream is subjected to time window aggregation and statistical calculation to obtain the call frequency, average time consumption and error rate between any two service identifiers, which are used as the weight attribute of the dependency edge;

[0016] The service node basic information set and the dependency edge with the weight attribute are graphically assembled to generate a real-time dependency graph, wherein the nodes of the real-time dependency graph correspond to service instances, the edges represent call dependencies, and the weight attribute of the edge contains call frequency, average time consumption and error rate.

[0017] Preferably, according to the real-time dependency graph, a graph-aware fault injection point list and an influence propagation path set are generated by graph matching and backtracking analysis of historical fault events, including:

[0018] The historical fault event reports are extracted from the fault management platform, and the fault service identifier, fault occurrence time, fault phenomenon description and manually annotated root cause in the report are parsed;

[0019] mapping the fault service identity to a corresponding node of the real-time dependency graph map, performing reverse and forward graph traversal from the corresponding node as a starting point in the direction of the dependency edge, and identifying all upstream and downstream service nodes that exist in the call association within the fault time window;

[0020] performing influence degree analysis on the associated service nodes obtained through traversal in combination with historical monitoring indicators, marking the nodes that appear performance degradation or error rate rise to form a fault influence range set;

[0021] taking the fault service identity as a root node and the nodes in the fault influence range set as leaf nodes, finding all propagation paths in the real-time dependency graph map, performing propagation probability estimation on each path in combination with call frequency and historical error rate, and generating a graph-aware fault injection point list and an influence propagation path set with accompanying probability.

[0022] Preferably, in the non-production isolated environment, a programmable agent is driven to perform multi-dimensional fault injection on the microservice instances according to the graph-aware fault injection point list and the influence propagation path set, and full-system response signals in the injection process are synchronously collected, including:

[0023] In the non-production isolated environment, a group of microservice instance replicas are deployed in accordance with the topology consistent with the production environment, and a programmable agent is installed on each replica;

[0024] Target service nodes are sequentially selected from the graph-aware fault injection point list, and fault injection instructions are issued to the programmable agent of the service instance replica corresponding to the target service node according to a pre-defined fault dimension library, the fault dimensions including network delay, high error rate return, resource exhaustion, and process abnormal exit;

[0025] During the fault injection duration, system response signals of all service instances involved in the influence propagation path set from the injection point are collected by the programmable agent and independent monitoring probes, including interface response time, system resource utilization, log error information, and health status of distributed tracking links;

[0026] All collected system response signals of service instances are aligned and integrated by timestamp and service identity to form a full-system response signal data package corresponding to a single fault injection event.

[0027] Preferably, the full-system response signal is subjected to multi-dimensional deviation calculation with a pre-set baseline signal to form a fault simulation observation record containing signal deviation intensity and propagation rate, including:

[0028] Collecting system response signals of each service instance in the same length of time when the non-production isolated environment is in steady state, and taking the signals after smoothing as preset baseline signals;

[0029] For each signal in the full-amount system response signal data packet, calculate the statistical deviation of the signal and the corresponding baseline signal in the same period, including mean difference, standard deviation change rate and distribution difference;

[0030] Analyze the diffusion process of signal deviation along the set of influence propagation paths, calculate the time difference of deviation propagation from the fault injection point to each affected service node, and calculate the average propagation rate combined with the calling relationship on the path;

[0031] The deviation intensity calculation results of each signal are associated with its corresponding propagation path and propagation rate, and a structured fault simulation observation record is formed, each record being associated with a specific fault injection event.

[0032] Preferably, based on the fault simulation observation record, a graph neural network model with time sequence dependence understanding ability is trained, including:

[0033] The structure of the real-time dependency graph is taken as the initial graph structure of the graph neural network model, and the signal deviation intensity and propagation rate in the fault simulation observation record are converted into time sequence feature vectors of corresponding nodes and edges in the graph;

[0034] A graph neural network model including a space-time convolution layer and an attention mechanism is constructed, and the input of the graph neural network model is a graph with time sequence feature vectors, which simultaneously captures feature dependence in topological space and time dimension through the space-time convolution layer;

[0035] The historical accumulated fault simulation observation record is used as a training sample set, and the prediction of node and edge state changes in a future time segment is taken as a training target, and the graph neural network model including the space-time convolution layer and the attention mechanism is supervised trained;

[0036] When the prediction accuracy of the model on the validation set reaches a predetermined threshold, the training is completed, and a trained graph neural network model is obtained.

[0037] Preferably, the trained graph neural network model is used to identify fault patterns and predict evolution of online real-time link data, and output a fault risk assessment report, including:

[0038] The tracking data of microservice call chain and node performance indicators are collected from the online production environment in real time, converted into real-time time sequence feature vectors consistent with the model input format, and mapped to corresponding nodes and edges of the real-time dependency graph;

[0039] inputting the atlas with real-time time sequence feature vectors into the trained graph neural network model, and outputting state prediction values of all nodes and edges in the graph in future time slices, the state prediction values including abnormal probability and performance index prediction;

[0040] comparing the state prediction values output by the model with current measured values, identifying nodes and edges with abnormal probability exceeding a threshold or performance index deviation being too large, and marking them as potential fault points;

[0041] performing correlation analysis on the potential fault points in combination with the real-time dependency atlas, outlining a fault propagation subgraph, and predicting the evolution trend and final impact range of the fault according to the state change sequence predicted by the model, and integrating to form a fault risk assessment report including potential fault points, propagation paths, estimated impact, and risk level.

[0042] Preferably, the method further comprises:

[0043] subscribing to distributed tracking data streams and index data streams in a message queue in real time, the distributed tracking data streams including tracking identifiers, service identifiers, start times, end times, and state codes, and the index data streams including service identifiers, central processing unit usage rates, memory usage rates, request rates, and error rates;

[0044] cleaning and time window aligning the data, aggregating each index of each service identifier within a fixed time window into a multi-dimensional index vector, and aggregating tracking data of the service corresponding to the service identifier into a tracking feature vector;

[0045] splicing and normalizing the multi-dimensional index vector and the tracking feature vector to form a real-time time sequence feature vector corresponding to the service node corresponding to the service identifier;

[0046] According to the service identifier, the real-time time sequence feature vector is assigned to the corresponding service node in the real-time dependency atlas as an attribute; for the dependent edge, the call frequency and average delay in the corresponding time window are extracted according to the service identifiers of the calling parties to form the real-time time sequence feature vector of the edge.

[0047] Preferably, the method further comprises:

[0048] According to the predicted fault propagation path in the fault risk assessment report, a targeted defense rule plan is dynamically generated, the defense rule plan including traffic pre-disposition for a specific path, elastic scaling pre-instruction for an overloaded node, and resource reservation strategy for a key service;

[0049] the defense rule plan is preloaded to the policy execution engine before the predicted failure occurrence time window, but is not immediately effective;

[0050] When the online monitoring system detects the initial failure symptoms consistent with the prediction, the policy execution engine is automatically triggered to make the corresponding defense rule plan immediately effective, and perform the predefined defense actions.

[0051] Preferably, when the online monitoring system detects the initial failure symptoms consistent with the prediction, the policy execution engine is automatically triggered to make the corresponding defense rule plan immediately effective, and perform the predefined defense actions, including:

[0052] continuously comparing the real-time indicators collected by the online monitoring system with the predicted initial failure symptom indicator mode in the failure risk assessment report;

[0053] When the matching degree of the real-time indicators with the predicted initial failure symptom indicator mode exceeds the set threshold in consecutive detection periods, it is determined that the predicted failure has begun to occur;

[0054] According to the matched specific symptom mode, the defense rule plan with the highest correlation is retrieved from the preloaded defense rule plan, and an immediate effective instruction is sent to the policy execution engine;

[0055] The policy execution engine performs the defense actions defined in the plan according to the immediate effective instruction, including switching part of the traffic flowing to the predicted failure source node to the backup instance, increasing the resource quota of the predicted affected nodes in advance, and inserting specific fusing or degradation rules at the link level.

[0056] Compared with the prior art, the present application has the following beneficial effects:

[0057] Based on the service registry center and real-time communication traffic, a real-time dependency graph containing hierarchy and call frequency is constructed, and historical failure events are matched and backtracked according to the graph. This technology makes the generation of failure injection point list and influence propagation path directly derived from the combination of the current topology state of the system and the historical failure mode. The generated failure simulation scenario can dynamically reflect the real-time key dependencies and call hotspots between services, so that the targeting of the failure injection experiment is enhanced, and the most active and core interaction relationship of the link and system at present is consistent.

[0058] In the fault simulation process, the full-quantity system response signal is synchronously collected, and multi-dimensional deviation calculation is performed on the baseline to form an observation record containing deviation intensity and propagation rate. A graph neural network model with time-dependent understanding ability is trained using such records, so that the model can learn the intensity variation law and spatio-temporal sequence of fault signal diffusion in the service network topology from the data. The trained model is used to analyze online real-time link data, which can identify similar early-stage fault propagation signal patterns in historical simulation observations and predict the diffusion trend and potential impact range of abnormal signals. This method can present the evolution trend of the fault from the global and correlation dimensions of the system, and based on the prediction information, the targeted defense actions can be started earlier. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The working principle diagram of the microservice scene full-link intelligent fault simulation and defense evaluation method described in the application;

[0060] Figure 2 The flowchart for constructing a real-time dependency graph;

[0061] Figure 3 The flowchart for non-production environment fault injection and signal collection;

[0062] Figure 4 The feature capture effect comparison chart of the space-time convolution layer in the microservice fault simulation GNN model;

[0063] Figure 5 The effect comparison chart of the microservice fault defense strategy. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0065] Please refer to Figure 1This invention provides a method for end-to-end intelligent fault simulation, assessment, and defense in a microservice scenario. The method includes: First, constructing a real-time dependency graph containing service hierarchy and call frequency information based on the service registry and inter-service communication traffic. Based on this real-time dependency graph, performing graph matching and backtracking analysis on historical fault events to generate a graph-aware list of fault injection points and their corresponding impact propagation paths. Then, in a non-production isolated environment, driving a programmable agent deployed on the microservice instance to perform multi-dimensional fault injection based on the aforementioned list and path set, and simultaneously collecting all system response signals generated during the injection process. Performing multi-dimensional deviation calculations on these collected full system response signals and a preset baseline signal to form a fault simulation observation record containing information such as signal deviation strength and propagation rate. Based on the accumulated fault simulation observation record, training a graph neural network model with temporal dependency understanding capabilities. Using the trained graph neural network model, performing fault mode recognition and evolution prediction on online real-time link data, and finally outputting a fault risk assessment report to provide a basis for defense decisions.

[0066] Example 1: See Figure 2 The system continuously monitors service instance online / offline events in the service registry, extracting the network address, service identifier, and metadata tags of each service instance to form a set of basic service node information. It also captures inter-service communication traffic, parses application layer protocols, and extracts the caller's service identifier, the callee's service identifier, the call timestamp, the response code, and the call duration, forming an inter-service call record stream. This stream is then aggregated and statistically calculated using time windows to obtain the call frequency, average duration, and error rate between any two service identifiers, which are used as the weight attributes of dependency edges. Finally, the set of basic service node information and the weighted dependency edges are assembled into a graph structure to generate a real-time dependency graph. In this graph, nodes correspond to service instances, edges represent call dependencies, and the weight attributes of the edges include call frequency, average duration, and error rate.

[0067] Extract historical fault event reports from the fault management platform, analyze the fault service identifier, fault occurrence time, fault phenomenon description and manually annotated root cause in the report. Map the fault service identifier to the corresponding node in the real-time dependency graph. Starting from the corresponding node, perform reverse and forward graph traversal along the direction of the dependency edge to identify all upstream and downstream service nodes that exist in the call association within the fault time window. Combine historical monitoring indicators to analyze the impact of the associated service nodes obtained by traversal, mark the nodes with performance degradation or error rate increase to form a fault impact range set. Take the fault service identifier as the root cause node and the nodes in the fault impact range set as the leaf nodes to find all propagation paths in the real-time dependency graph. Combine the call frequency and historical error rate to estimate the propagation probability of each path to generate a list of graph-aware fault injection points and an impact propagation path set with probability.

[0068] In specific implementation, the service registry center continuously publishes the online and offline events of service instances. The system listens to these events and extracts the network address, service identifier and metadata tag of the service instance contained in the event. Each extraction forms a record, and the collection of all records constitutes the service node basic information set. In specific implementation, the system captures the inter-service communication network traffic within the microservice cluster through a bypass method, and performs application layer protocol analysis on the captured network data packets. The caller service identifier, the called service identifier, the call timestamp, the response code and the call duration are extracted from the parsed request and response messages. These information are continuously generated in the form of stream records, constituting the inter-service call record stream.

[0069] When processing the inter-service call record stream, the system sets a rolling time window. Within a time window, all call records with the same caller service identifier and called service identifier are aggregated and calculated. The aggregation calculation includes counting the total number of calls as the call frequency, calculating the arithmetic mean of all call durations as the average duration, and calculating the proportion of the number of calls with failed response codes to the total number of calls as the error rate. Within a time window, the call frequency, average duration and error rate between any two service identifiers constitute the weight attribute of the dependency edge between the two service identifiers. It can be understood that all service identifiers in the service node basic information set are instantiated as graph nodes, and all dependency relationships with weight attributes calculated are instantiated as directed edges connecting corresponding nodes. Through graph structure assembly operation, a dynamically updated real-time dependency graph is generated. The nodes in the real-time dependency graph represent a specific service instance, the directed edges represent the call dependency from the caller to the called, and the weight attribute of the edge is a data structure containing the call frequency, average duration and error rate.

[0070] In some embodiments, when analyzing historical fault events according to the real-time dependency graph, historical fault event reports are extracted from a database interface of the fault management platform, each report text is parsed to extract fault service identification, fault occurrence time, fault phenomenon description field, and root cause field annotated by an operation and maintenance personnel. The fault service identification in the historical fault event report is matched with the node service identification in the real-time dependency graph to locate the corresponding fault node in the real-time dependency graph. Starting from the fault node, reverse depth-first traversal is performed in the real-time dependency graph along the direction of the dependency edge to identify all upstream service nodes that call the fault node within a time window before and after the fault occurrence time, and forward breadth-first traversal is performed along the direction of the dependency edge to identify all downstream service nodes called by the fault node within the same time window. All nodes obtained by reverse traversal and forward traversal constitute a node set that has a calling association with the fault event.

[0071] In some embodiments, the influence degree of the node set is analyzed in combination with historical monitoring indicators, and the historical monitoring indicators are obtained by querying a time series database, and the query condition is the performance indicator time series data of each service identification in the node set within the fault occurrence time window. The performance indicator time series data of each associated service node is analyzed, and if the percentile value of the response time indicator of the node has a significant increase within the fault window compared with before the window, or the time series of the error rate indicator has a sharp peak within the fault window, the node is marked as an affected node. All marked affected nodes constitute a fault influence range set. The fault service identification in the historical fault event report is taken as a root cause node, and each node in the fault influence range set is taken as a leaf node, and all directed paths from the root cause node to the leaf node are found in the real-time dependency graph, and each directed path represents a fault propagation path. Optionally, for each identified fault propagation path, a propagation probability estimate is calculated in combination with the historical call frequency weight and the historical error rate weight of each edge on the path. One calculation method is to take the product of the historical error rate weights of all edges on the path, and then multiply the value normalized by the call frequency weight from the root cause node to the leaf node. The fault injection point list is a list of all unique root cause node service identifications extracted from the historical fault event analysis, and each root cause node corresponds to a set of influence propagation paths with probabilities, which includes all paths from the root cause node to each affected leaf node and the estimated propagation probability.

[0072]

[0073] wherein the symbol represents the fault propagation probability estimate from the root cause node to a specific leaf node, and the symbol denotes the set of all edges on the propagation path connecting the root cause node to the leaf node, symbol denotes an edge historical error rate weight in the fault time window, symbol denotes the total historical call frequency from the root cause node to the leaf node along the propagation path in the fault time window, which is estimated by the minimum call frequency of each edge on the path, symbol denotes the total historical call frequency of all calls from the root cause node in the fault time window.

[0074] Embodiment 2: refer to Figure 3 In a non-production isolated environment, a set of microservice instance replicas are deployed according to the consistent topology of the production environment, and programmable agents are installed on each replica. From the list of fault injection points perceived by the graph, the target service node is selected in turn, and according to the pre-defined fault dimension library, the programmable agent of the target service node corresponding to the service instance replica is issued a fault injection instruction. The fault dimensions include network delay, high error rate return, resource exhaustion and process abnormal exit. During the duration of the fault injection, through the programmable agent and the independent monitoring probe, the system response signals of all service instances involved along the impact propagation path set are collected from the injection point, including interface response time, system resource utilization, log error information, and distributed tracking link health status. All collected signals are aligned and integrated according to timestamp and service identification to form a full-amount system response signal data package corresponding to a single fault injection event.

[0075] When the non-production isolated environment is in a steady state, the system response signals of each service instance in the same time length are collected, and after smoothing processing, they are used as the pre-set baseline signals. For each signal in the full-amount system response signal data package, the statistical deviation of the corresponding baseline signal in the same period is calculated, including mean difference, standard deviation change rate and distribution difference. The diffusion process of signal deviation along the impact propagation path set is analyzed, the time difference of deviation propagation from the fault injection point to each affected service node is calculated, and the average propagation rate is calculated combined with the call relationship on the path. The deviation intensity calculation results of each signal are associated with its corresponding propagation path and propagation rate for storage, forming a structured fault simulation observation record, each record associated with a specific fault injection event.

[0076] In a specific implementation, the non-production isolated environment is a simulation environment that is network isolated from the online production environment but has consistent topology, in which a complete copy of a set of microservice instances is deployed, and a programmable agent is installed on each microservice instance copy, which receives instructions from the control center and performs interference operations on the microservice instance. According to the graph-aware fault injection point list, the control center selects a target service node in turn, for example, a node identified as "PaymentService-v1.2", and for this target service node, the control center selects a specific fault type and parameter from a predefined fault dimension library, and the fault dimensions defined in the fault dimension library include network delay, high error rate return, resource exhaustion, and process abnormal exit. In a specific implementation, the "network delay" dimension is selected and the delay time is set to 500 ms, the control center generates a fault injection instruction containing the target microservice instance identifier "PaymentService-v1.2", the fault type "network delay", and the parameter "500ms", and sends it to the programmable agent installed on the microservice instance copy through the control channel. The programmable agent parses the instruction and injects the specified delay at the network interface layer of the microservice instance.

[0077] During the execution and continuous effect of the fault injection instruction, the system starts multi-source data collection, collects internal indicators through programmable agents installed inside all related microservice instance copies on the fault injection point and its downstream path, and collects external observation data through independent monitoring probes deployed in the non-production isolated environment. The collected system response signals cover multiple levels, including the response time of each microservice instance interface, the central processing unit usage and memory usage of each microservice instance container, the error entries recorded in each microservice instance application log, and the health status and span data of the distributed tracking link throughout multiple microservice instances. It can be understood that all signal data collected are attached with accurate timestamps and service identifiers that generate the signal, and the system classifies all signals according to the unique identifier of the fault injection event, and time-aligns and integrates all signal data generated at different timestamps with the same service identifier under the same event, and finally packages to form a full-amount system response signal data package strictly corresponding to a single fault injection event.

[0078] In some embodiments, in order to perform the deviation calculation, a baseline signal needs to be established in advance. When the non-production isolated environment is not performing any fault injection and is in a steady state operation, the system collects the system response signals of all microservice instance copies on the same set of monitoring indicators for a long enough time, and performs smoothing processing on the time series data of each type of signal, for example, using a moving average algorithm to eliminate short-term fluctuations. The processed stable signal sequence is taken as the preset baseline signal of the microservice instance on the indicator. For the full amount of system response signal data packets generated by each fault injection experiment, the system performs multi-dimensional statistical deviation calculation on each signal in the data packet. The difference between the experimental signal and the corresponding baseline signal in the same time interval during the fault injection period is calculated. In a specific implementation, for the response time signal, the difference between the mean value of the experimental signal sequence and the mean value of the baseline signal sequence is calculated as the mean difference; the ratio of the standard deviation of the experimental signal sequence to the standard deviation of the baseline signal sequence is calculated as the standard deviation change rate.

[0079] In some embodiments, in addition to the mean difference and the standard deviation change rate, the system also calculates the distribution difference degree between the experimental signal and the baseline signal. The distribution difference degree is used to quantify the difference between the overall probability distribution of the two signal sequences. Optionally, one method of calculating the distribution difference degree is to use a statistical test based on the cumulative distribution function, such as the D statistic in the Kolmogorov-Smirnov test. When calculating the distribution difference degree, the statistical test based on the cumulative distribution function is used, and the Kolmogorov-Smirnov test is specifically implemented. For the experimental signal sequence and the corresponding baseline signal sequence collected in the fault injection event, first, the empirical cumulative distribution functions of the two are constructed respectively; then, the difference between the two empirical cumulative distribution functions at each data point in the entire data value range is calculated, and the maximum value of the absolute values of all the differences is found. The maximum value is the D statistic in the Kolmogorov-Smirnov test, which is the quantitative result of the distribution difference degree. For the response time experimental signal sequence collected from the service identifier "A" in the fault injection event , and the corresponding response time baseline signal sequence obtained from the baseline , the empirical cumulative distribution functions are calculated, and the distribution difference degree is calculated as follows:

[0080]

[0081] wherein the symbol represents the distribution difference degree, the symbol represents the supremum, and the symbol represents the value of the response time variable, and the symbol represents the empirical cumulative distribution function of the experimental signal sequence at The value at the location, the symbol Represents the baseline signal sequence The empirical cumulative distribution function in The system analyzes the propagation process of signal deviation in the microservice dependency network. Based on the set of influence propagation paths, it identifies the moment when the deviation is first detected at the fault injection point and the moment when the deviation is first detected on each affected service node. The time difference between these two moments is calculated as the propagation delay. Based on the call relationships and hop counts of the edges in the influence propagation path set, the average propagation rate of the deviation from the source node to the target node is calculated. Finally, the system associates and stores the deviation intensity calculation results for each signal, including the mean difference, standard deviation rate of change, and distribution variability, with the corresponding service identifier, the fault injection event identifier, the propagation path, and the calculated average propagation rate, forming a structured fault simulation observation record. Each fault simulation observation record is uniquely associated with a specific fault injection event.

[0082] Example 3: Using the structure of a real-time dependent graph as the initial graph structure for the graph neural network model, the signal deviation intensity and propagation rate in the fault simulation observation records are converted into temporal feature vectors of corresponding nodes and edges in the graph. A graph neural network model containing spatiotemporal convolutional layers and an attention mechanism is constructed. The input of this graph neural network model is a graph with temporal feature vectors, and the spatiotemporal convolutional layers simultaneously capture feature dependencies in both the topological space and the temporal dimension. Using historically accumulated fault simulation observation records as the training sample set, and with the training objective of predicting the changes in the state of nodes and edges in future time segments, the graph neural network model containing spatiotemporal convolutional layers and an attention mechanism is subjected to supervised training. When the prediction accuracy of the model on the validation set reaches a predetermined threshold, the training is completed, and the trained graph neural network model is obtained.

[0083] In practical implementation, the initial graph structure for constructing the graph neural network model is directly derived from the real-time dependency graph. The set of nodes and edges, as well as the directionality of the edges, in the real-time dependency graph are mapped as is to form the graph structure foundation of the graph neural network model. The signal deviation intensity and propagation rate information stored in the fault simulation observation records need to be converted. For each service node in the real-time dependency graph, all fault simulation observation records with that node as the observation object are aggregated, and the signal deviation intensity values ​​of that node at each time point are extracted and arranged in chronological order to form a time-series sequence of the node's signal deviation intensity. Simultaneously, for each dependency edge in the real-time dependency graph, all fault simulation observation records containing the propagation path of that edge are aggregated, and the propagation rate values ​​corresponding to that edge are extracted to form a time-series sequence of the edge's propagation rate. These time-series sequences are then converted into fixed-dimensional feature vectors through an embedding layer. The node's signal deviation intensity time-series sequence is converted into a node's time-series feature vector, and the edge's propagation rate time-series sequence is converted into an edge's time-series feature vector. After processing, a graph structure data with time-series feature vectors is obtained, which serves as the input to the graph neural network model.

[0084] The constructed graph neural network model includes a spatiotemporal convolutional layer and an attention mechanism. The spatiotemporal convolutional layer consists of a combination of spatial graph convolution operations and temporal one-dimensional convolution operations. The spatial graph convolution operations are performed on the topological structure defined by the real-time dependent graph, aggregating the temporal feature vector information of each node's neighboring nodes and their connecting edges. The temporal one-dimensional convolution operations are performed on the independent temporal feature vector sequences of each node and each edge, capturing their evolution patterns over time. In practice, the output of the spatiotemporal convolutional layer is connected to an attention mechanism. The attention mechanism calculates the association weight between any two nodes in the graph that are connected by a path. The association weight is determined by the similarity between the feature vectors of these two nodes after processing by the spatiotemporal convolutional layer. The attention mechanism uses the calculated association weights to perform weighted propagation and aggregation of information in the graph, enabling the model to focus on other node information most relevant to the current node's state.

[0085] The training sample set is constructed by using the historical accumulated failure simulation observation records, and each training sample is composed of a graph sequence in a historical time period and a graph state in a corresponding future time period. The graph sequence in the historical time period is composed of graph snapshots with time sequence feature vectors in a plurality of continuous time slices in the time period, and the graph state in the future time period is a target to be predicted, that is, the state changes of nodes and edges in the graph after one or more future time slices, which can be represented by the predicted values of the performance indicators of the nodes and the predicted values of the call delays of the edges. In specific implementation, the target of model training is to enable the graph neural network model to accurately predict the states of nodes and edges in the future time slice according to the input continuous historical graph sequence, which is a supervised learning task. The model calculates the predicted values by forward propagation, compares the predicted values with the true values extracted from the failure simulation observation records, uses the mean square error as the loss function, and optimizes the parameters in the graph neural network model, including the convolution kernel weights of the space-time convolution layer and the weight matrix in the attention mechanism, by the back propagation algorithm.

[0086] In some embodiments, the accumulated failure simulation observation records are divided into a training set, a validation set and a test set in chronological order during the training process. The model is trained for multiple rounds of iteration on the training set, and the prediction performance of the model is evaluated using the validation set after each round of iteration. The evaluation index of the prediction performance of the model is the prediction accuracy, and the calculation of the prediction accuracy involves comparing the predicted state and the true state output by the model. Optionally, for the prediction of node state, an error tolerance threshold can be set, and when the absolute error between the predicted value and the true value of the node performance indicator is less than the threshold, the state prediction of the node is considered to be accurate; for the prediction accuracy of the whole graph state in a time slice, the proportion of the number of accurately predicted nodes and edges to the total number of nodes and edges in the graph is the prediction accuracy. When the prediction accuracy of the model on the validation set reaches a predetermined threshold, for example, 90%, the model training is determined to be completed, and the trained graph neural network model is obtained. It can be understood that the symbol of the completion of the model training is that the prediction accuracy of the model on the unseen validation set data meets the predetermined requirement, at which time the parameters of the model are saved, and the trained graph neural network model is obtained. In some embodiments, the calculation formula of the attention correlation weight in the attention mechanism is:

[0087]

[0088] wherein the symbol represents the attention correlation weight of the node to the node , the symbol represents the natural exponential function, the symbol represents the activation function, such as LeakyReLU, and the symbol represents the feature vector of the node after the space-time convolution layer processing. representing a node feature vector after processing by a spatio-temporal convolution layer, symbol representing a node feature vector after processing by a spatio-temporal convolution layer, symbol representing a trainable parameter weight matrix, symbol representing all other nodes in the graph that are path-connected to the node representing a set of all other nodes that are path-connected to the node representing a transposition operation on a vector.

[0089] Referring to Figure 4 , it is a comparison chart of feature capturing effect of spatio-temporal convolution layer in microservice fault simulation GNN model. The chart corresponds to the "spatio-temporal convolution layer design" stage of the microservice fault simulation GNN model, and is used to show the two-dimensional feature capturing ability of the spatio-temporal convolution layer: the spatial convolution focuses on the "topological association between service nodes", and the temporal convolution focuses on the "temporal evolution of fault signals"; embody the complementarity of the two types of features: the spatial feature increases with the increase of the dimension (number of adjacent nodes), and the temporal feature presents fluctuation and decline with the increase of the dimension (time step), reflecting the difference in feature mode of different dimensions; provide basis for the convolution layer parameter optimization of the GNN model. Through the chart, it can be directly judged that the capturing effect of the spatial convolution on the "node topological association" increases with the increase of the number of adjacent nodes, and is suitable for identifying the propagation path of the fault in the service link; the capturing of the time convolution on the "temporal pattern" is fluctuant, and the attention mechanism needs to be further focused on the signal features of the key time segments.

[0090] Example 4: Real-time collection of microservice call chain tracking data and node performance indicators from online production environment, conversion into real-time time series feature vectors consistent with the input format of the model, and mapping to the corresponding nodes and edges of the real-time dependency graph. Specifically, real-time subscription of distributed tracking data stream and indicator data stream in the message queue, the distributed tracking data stream includes tracking identifier, service identifier, start time, end time and status code, the indicator data stream includes service identifier, central processing unit usage rate, memory usage rate, request rate and error rate. Clean and time window align the data, aggregate each indicator of each service identifier into a multi-dimensional indicator vector within a fixed time window, and aggregate the tracking data of the service corresponding to the service identifier into a tracking feature vector. Concatenate and normalize the multi-dimensional indicator vector and the tracking feature vector to form the real-time time series feature vector corresponding to the service node corresponding to the service identifier. According to the service identifier, assign the real-time time series feature vector as an attribute to the corresponding service node in the real-time dependency graph; for the dependent edge, according to the calling service identifiers, extract the calling frequency and average delay in the corresponding time window to form the real-time time series feature vector of the edge.

[0091] The atlas with real-time time sequence feature vectors is input into the trained graph neural network model, and the model outputs the state prediction values of all nodes and edges in the graph in the future several time slices, including abnormal probability and performance index estimation. Compare the state prediction values output by the model with the current measured values, identify nodes and edges with abnormal probability exceeding the threshold or performance index deviation too large, and mark them as potential fault points. In combination with the real-time dependency atlas, the correlation analysis of the potential fault points is carried out, the fault propagation subgraph is outlined, and according to the state change sequence predicted by the model, the evolution trend and the final impact range of the fault are estimated, and the fault risk assessment report containing potential fault points, propagation path, estimated impact and risk level is integrated.

[0092] In a specific implementation, the monitoring data of the online production environment is collected through a real-time data pipeline. The system subscribes to the distributed tracking data stream and the index data stream in the message queue. Each record in the distributed tracking data stream contains the tracking identifier, service identifier, start time, end time and status code fields. Each record in the index data stream contains the service identifier, central processing unit usage rate, memory usage rate, request rate and error rate fields, as well as the corresponding timestamp. In a specific implementation, the collected raw data is cleaned. The cleaning operation includes filtering out records with missing service identifier fields, correcting values that are obviously out of the reasonable range, and aligning data from different time sources to a unified time reference. The system sets a fixed aggregation time window, for example, 5 seconds, and performs aggregation calculation on all index data records of each service identifier within a time window. The central processing unit usage rate, memory usage rate, request rate and error rate indicators at multiple time points are calculated to obtain their average values within the window, forming a multi-dimensional index vector containing these four dimensions. See Table 1.

[0093] Table 1: Service node real-time time sequence feature vector aggregation table

[0094] Service identification Central processor usage rate (%) Memory usage rate (%) Request rate (req / s) Error rate (%) Tracking average time consumption (ms) Tracking error proportion (%) Order service 45.2 68.7 1250 0.1 152.3 0.05 Payment service 32.1 45.3 890 0.8 210.5 0.12 Inventory service 28.7 39.8 560 0.0 87.6 0.00

[0095] Meanwhile, for each service identifier, all distributed tracing data of the service in the same time window is aggregated, the average value of the time consumption in the tracing data is calculated to form a tracing average time consumption feature, and the proportion of the number of traces with a failure status code to the total number of traces is calculated to form a tracing error proportion feature, both of which together constitute a tracing feature vector. The multi-dimensional index vector corresponding to each service identifier is spliced with the tracing feature vector to form a comprehensive real-time feature vector, and then the comprehensive real-time feature vector is normalized to make the values of each dimension in the same scale range. The normalized vector obtained finally is the real-time time sequence feature vector of the service node corresponding to the service identifier. According to the service identifier, the real-time time sequence feature vector is assigned as a node attribute to the corresponding service node in the real-time dependency graph. For the dependency edge in the real-time dependency graph, according to the main calling service identifier and the called service identifier recorded on the edge, the total number of calls from the main calling service to the called service in the same time window is extracted from the cleaned calling records as the calling frequency, and the average response time of these calls is extracted as the average delay. The calling frequency and the average delay are normalized to form the real-time time sequence feature vector of the dependency edge.

[0096] The graph with the real-time time sequence feature vector is input into the trained graph neural network model. The model takes the fixed time window as the step size and receives the historical graph sequence of the recent time windows as the input. The graph neural network model performs forward calculation and outputs the predicted graph state for the future time windows. The predicted graph state contains the state prediction values of all nodes and edges in the graph in each future time window. The state prediction value of a node includes the abnormal probability and the estimated values of performance indicators such as the central processing unit usage rate and the request rate. The state prediction value of an edge includes the abnormal probability and the estimated value of the average delay. In some embodiments, the abnormal probability output by the model is a value between 0 and 1, indicating the likelihood of the model judging that the node or edge is in an abnormal state in the corresponding future time window. Subsequently, the system compares the state prediction values output by the model for the nearest future time window with the actual measured values of the current time window. For a node, the predicted values of its performance indicators are compared with the measured values. For an edge, the predicted value of its average delay is compared with the measured value. It can be understood that when the abnormal probability prediction value of a node exceeds a pre-set threshold value, or the deviation between the predicted value and the measured value of its key performance indicators exceeds another threshold value, the node is marked as a potential fault point. The judgment of an edge follows similar logic.

[0097] In some embodiments, after identifying potential failure points, the system conducts correlation analysis in combination with the topology of the real-time dependency graph. Starting from each potential failure point, the system traverses the real-time dependency graph in the direction of dependency edges for a limited depth, finds other nodes that are directly or indirectly connected to it and are also marked as potential failure points or indicator abnormalities, extracts these correlated nodes and edges between them from the entire real-time dependency graph, and constructs a failure propagation subgraph. According to the state prediction value sequence output by the graph neural network model, the system analyzes the abnormal probability of each node and edge in the failure propagation subgraph and the trend of the performance indicators in the future several time windows, estimates the diffusion direction and range of the failure impact. Optionally, one method to estimate the final impact range is to set an abnormal probability threshold chain. When the abnormal probability prediction value of a node exceeds the increasing threshold value in the threshold chain in consecutive time windows, it is determined that it will be affected. The set of all affected nodes is merged with the failure propagation subgraph to outline the evolution trend and final impact range of the failure. Finally, the system integrates the list of potential failure points, the structure of the failure propagation subgraph, the estimated impact range, and the risk level of each node to generate a structured failure risk assessment report. The risk level in the failure risk assessment report The risk level score can be calculated by the following formula:

[0098]

[0099] wherein the symbol represents the risk level score, the symbol and are preset weight coefficients, the symbol represents the abnormal probability output by the graph neural network model, and the symbol represents the normalized deviation absolute value of the predicted value and the current measured value of the key performance indicator. It can be understood that the report is presented in a standard format, including text description and visual charts, for use by the operation and maintenance decision system.

[0100] Embodiment 5: According to the predicted failure propagation path in the failure risk assessment report, a targeted defense rule plan is dynamically generated, which includes traffic pre-allocation for a specific path, elastic scaling pre-instruction for overloaded nodes, and resource reservation strategy for key services. The defense rule plan is preloaded into the policy execution engine before the predicted failure occurs in the time window, but does not take effect immediately.

[0101] When the online monitoring system detects the initial failure signs consistent with the prediction, the strategy execution engine is automatically triggered to make the corresponding defense rule plan take effect immediately, and perform the predefined defense actions. The specific process includes: continuously comparing the real-time indicators collected by the online monitoring system with the predicted initial failure sign indicator mode in the failure risk assessment report; when the matching degree of the real-time indicators with the predicted initial failure sign indicator mode exceeds the set threshold in consecutive detection periods, it is determined that the predicted failure has started; according to the specific sign mode matched, the highest associated plan is retrieved from the preloaded defense rule plan, and an immediate effect instruction is sent to the strategy execution engine; the strategy execution engine executes the defense actions defined in the plan according to the immediate effect instruction, including switching part of the flow to the backup instance, increasing the resource quota in advance for the predicted affected nodes, and inserting a fuse or degradation rule at the link level.

[0102] In specific implementation, the failure risk assessment report output contains the predicted failure propagation path, potential failure point and estimated impact range. The system analyzes the predicted failure propagation path information in the report, and the path information is represented in a graph structure, including the sequence of affected nodes and their associated relationships. According to the analyzed path information, the system dynamically generates targeted defense rule plans, and the generation logic of the plan is based on the roles and dependency relationships of the nodes in the path. For example, for the downstream nodes overloaded due to traffic impact on the predicted failure propagation path, elastic scaling pre-instructions are generated for the services to which the nodes belong, and the instruction content is to increase the number of instances by a preset percentage within a specific time window; for the key upstream nodes that are predicted to become the root cause of the failure, resource reservation strategies are generated for the nodes, and the strategy content is to allocate additional computing resource quota to the nodes within the predicted time period; for the specific call links through which the failure flow is predicted to flow, traffic pre-dispatch rules are generated for the path, and the rule content is to route part of the traffic to backup service instances or degraded service interfaces in proportion.

[0103] Before the predicted failure occurrence time window, the system preloads the generated defense rule plan into the rule library of the strategy execution engine, which is responsible for managing the storage, matching and execution of all defense rules. The preloading operation loads the metadata and executable logic of the defense rule plan into the strategy execution engine, but the state of the rule is marked as "to be activated", meaning that the rule has been identified and stored by the strategy execution engine, but will not immediately affect the online traffic or resource configuration. In some embodiments, the preloaded defense rule plan contains a predefined trigger condition field, and the content of the trigger condition field is directly related to the initial failure sign indicator mode described in the failure risk assessment report, for example, "when the 95th percentile of response time of service A exceeds 300 milliseconds for 3 consecutive detection periods and the error rate rises to 1%".

[0104] When the online monitoring system is running continuously, the system continuously compares the real-time indicators collected by the online monitoring system with the predicted initial failure symptom indicator pattern in the failure risk assessment report. The initial failure symptom indicator pattern is described in the report in a structured data pattern, such as a set of indicator threshold values and duration conditions for a specific service identifier. The comparison process occurs at each preset detection period, and the matching degree between the real-time indicator sequence and the predicted initial failure symptom indicator pattern is calculated. In some embodiments, the matching degree is calculated as follows:

[0105]

[0106] wherein the symbol represents the matching degree score, the symbol represents the number of independent indicator conditions contained in the predicted initial failure symptom indicator pattern, the symbol represents the th indicator condition, the symbol is an indicator function, which is 1 when the corresponding real-time indicator collected by the online monitoring system satisfies the indicator condition in the continuous detection period, and 0 otherwise, and the symbol is the preset weight of the th indicator condition in the pattern, and the sum of all weights is 1. It can be understood that when the matching degree of the real-time indicator with the predicted initial failure symptom indicator pattern exceeds a set threshold value in a continuous number of detection periods, for example, the matching degree is greater than 0.85 for three consecutive periods, the system determines that the predicted failure has begun to occur.

[0107] According to the specific symptom pattern matched, the system retrieves the pre-case with the highest correlation from the defense rule pre-case preloaded by the policy execution engine. The retrieval is based on the semantic similarity or exact match between the trigger condition field of the pre-case and the currently matched symptom pattern. An immediate effect instruction for the pre-case is sent to the policy execution engine, and the immediate effect instruction carries the unique identifier of the pre-case. According to the immediate effect instruction, the policy execution engine changes the state of the corresponding pre-case from “to be activated” to “has been activated”, and immediately executes all defense actions defined in the pre-case. The defense actions include switching part of the traffic flowing to the predicted failure source node to the standby instance, increasing the resource quota of the predicted affected node in advance, and inserting a fuse or degradation rule at the link level. For example, when the symptom of payment service delay increase is matched, the policy execution engine executes the action in the associated pre-case: switch 30% of the payment request traffic from the predicted overloaded “PaymentService-A” instance group to the standby instance group “PaymentService-B”, and at the same time, send an instruction to the resource management platform to increase the central processing unit resource quota of the “OrderService” instance group that is affected by the connection in advance by 20%.

[0108] Referring to Figure 5 This is a comparison chart of the effect of the microservice fault defense strategy. The chart corresponds to the effect verification stage of the microservice fault defense strategy, and is used to quantify the effectiveness of the defense strategy: by comparing the response time of "no defense" and "defense", the protection effect of the defense on service performance is intuitively reflected; the synergistic effect of the defense action is shown: the combination strategy of traffic switching (shifting fault pressure) and resource expansion (enhancing node carrying capacity) reduces the response delay; the timeliness of defense triggering is verified: after 30 seconds of defense triggering, the deterioration trend of response time is quickly contained. Through the chart, it can be judged that the defense strategy can effectively control the response time under fault from 250 ms to within 150 ms, and the performance improvement is significant; the synergistic strategy of traffic switching and resource expansion is feasible, and can be used as the core defense means of microservice full-link fault.

[0109] It should be noted that in this text, relational terms such as first and second are used only to distinguish one entity or action from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or actions. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0110] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for end-to-end intelligent fault simulation, assessment, and defense in microservice scenarios, characterized in that: The process includes the following: Based on the communication traffic between the service registry and services, a real-time dependency graph containing hierarchical structure and call frequency is constructed. Based on the real-time dependency graph, graph matching and backtracking analysis are performed on historical fault events to generate a graph-aware list of fault injection points and a set of impact propagation paths; In a non-production isolation environment, based on the list of fault injection points and the set of impact propagation paths perceived by the graph, the programmable agent is driven to perform multi-dimensional fault injection on microservice instances, and the full system response signals are collected synchronously during the injection process. The full system response signal is compared with the preset baseline signal in multiple dimensions to calculate the deviation, forming a fault simulation observation record that includes the signal deviation intensity and propagation rate. Based on fault simulation observation records, a graph neural network model with time-series dependency understanding is trained. The trained graph neural network model is used to perform fault mode recognition and evolution prediction on real-time online link data, and output a fault risk assessment report. The method utilizes a trained graph neural network model to perform fault mode identification and evolution prediction on real-time online link data, and outputs a fault risk assessment report, including: The system collects real-time tracking data and node performance metrics of the microservice call chain from the online production environment, converts them into real-time time-series feature vectors consistent with the model input format, and maps them to the corresponding nodes and edges of the real-time dependency graph. The graph with real-time temporal feature vectors is input into the trained graph neural network model. The model outputs the state prediction values ​​of all nodes and edges in the graph in the future time segments. The state prediction values ​​include anomaly probability and performance index estimates. The predicted state values ​​output by the model are compared with the current measured values ​​to identify nodes and edges whose abnormal probability exceeds the threshold or whose performance indicators deviate too much, and these are marked as potential fault points. By combining the real-time dependency graph, correlation analysis is performed on potential fault points to outline the fault propagation subgraph. Based on the state change sequence predicted by the model, the evolution trend and final impact range of the fault are estimated, and a fault risk assessment report containing potential fault points, propagation paths, estimated impacts and risk levels is formed.

2. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 1, characterized in that, The construction of a real-time dependency graph, including hierarchical structure and call frequency, based on the communication traffic between the service registry and services includes: Continuously monitor the online / offline events of service instances in the service registry, extract the network address, service identifier, and metadata tags of the service instances, and form a set of basic information about the service nodes. Bypass captures inter-service communication traffic, parses application layer protocols, and extracts the caller service identifier, callee service identifier, call timestamp, response code, and call duration to form an inter-service call record stream. Time window aggregation and statistical calculation are performed on the service call record stream to obtain the call frequency, average time consumption and error rate between any two service identifiers, which are used as the weight attributes of the dependency edge. The service node basic information set and the dependency edges with weight attributes are assembled into a graph structure to generate a real-time dependency graph. The nodes of the real-time dependency graph correspond to service instances, the edges represent call dependencies, and the weight attributes of the edges include call frequency, average time consumption, and error rate.

3. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 1, characterized in that, Based on the real-time dependency graph, historical fault events are matched and backtracked to generate a graph-aware list of fault injection points and a set of impact propagation paths, including: Extract historical fault event reports from the fault management platform and analyze the fault service identifier, fault occurrence time, fault phenomenon description, and root cause manually annotated in the reports; The fault service identifier is mapped to the corresponding node of the real-time dependency graph. Starting from the corresponding node, the graph is traversed in both the forward and reverse directions along the dependency edge to identify all upstream and downstream service nodes that have call associations with it within the fault time window. By combining historical monitoring indicators, the impact of the associated service nodes obtained through the traversal is analyzed, and nodes that have experienced performance degradation or increased error rate are marked to form a set of fault impact ranges. Using the fault service identifier as the root cause node and the nodes in the fault impact range set as leaf nodes, all propagation paths are searched in the real-time dependency graph. The propagation probability of each path is estimated by combining the call frequency and historical error rate, and a graph-aware list of fault injection points and a set of propagation paths with accompanying probabilities are generated.

4. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 1, characterized in that, In a non-production isolated environment, based on the fault injection point list and impact propagation path set perceived by the graph, a programmable agent is driven to perform multi-dimensional fault injection on microservice instances, synchronously collecting all system response signals during the injection process, including: In a non-production isolated environment, a set of replicas of microservice instances are deployed according to the same topology as the production environment, and a programmable agent is installed on each replica; Select target service nodes sequentially from the list of fault injection points perceived by the graph, and issue fault injection instructions to the programmable agent of the service instance replica corresponding to the target service node according to the predefined fault dimension library. The fault dimensions include network latency, high error rate return, resource exhaustion and abnormal process exit. During the fault injection, the programmable agent and independent monitoring probes collect system response signals from the injection point along the set of impact propagation paths, including interface response time, system resource utilization, log error information, and the health status of the distributed tracing link. The system response signals of all collected service instances are aligned and integrated according to timestamps and service identifiers to form a full system response signal data packet corresponding to a single fault injection event.

5. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 4, characterized in that, The step of performing multi-dimensional deviation calculations between the full system response signal and a preset baseline signal to form a fault simulation observation record that includes signal deviation intensity and propagation rate includes: When the system is running in a steady state in a non-production isolation environment, the system response signals of each service instance are collected within the same time period and smoothed out to serve as a preset baseline signal. For each signal in the full system response signal data packet, calculate its statistical deviation from the corresponding baseline signal within the same time period. The statistical deviation includes the mean difference, the rate of change of standard deviation, and the degree of distribution difference. Analyze the diffusion process of the signal deviation along the set of influence propagation paths, calculate the time difference of the deviation propagating from the fault injection point to each affected service node, and calculate the average propagation rate by combining the call relationship on the path; The deviation intensity calculation results of each signal are associated with its corresponding propagation path and propagation rate and stored to form a structured fault simulation observation record. Each record is associated with a specific fault injection event.

6. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 1, characterized in that, The process of training a graph neural network model with time-dependent understanding capabilities based on fault simulation observation records includes: Using the structure of the real-time dependent graph as the initial graph structure of the graph neural network model, the signal deviation intensity and propagation rate in the fault simulation observation record are converted into the temporal feature vectors of the corresponding nodes and edges in the graph. A graph neural network model containing spatiotemporal convolutional layers and an attention mechanism is constructed. The input of the graph neural network model is a graph with temporal feature vectors. The spatiotemporal convolutional layers simultaneously capture feature dependencies in the topological space and time dimension. Using historically accumulated fault simulation observation records as a training sample set, and with the training objective of predicting the changes in node and edge states in future time segments, the graph neural network model containing spatiotemporal convolutional layers and attention mechanisms is trained under supervision. When the model's prediction accuracy on the validation set reaches a predetermined threshold, training is complete, and a trained graph neural network model is obtained.

7. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 6, characterized in that, The process of collecting real-time trace data and node performance metrics of the microservice call chain from the online production environment, converting them into real-time time-series feature vectors consistent with the model input format, and mapping them to the corresponding nodes and edges of the real-time dependency graph includes: The system subscribes to distributed tracing data streams and metric data streams in a real-time message queue. The distributed tracing data streams include tracing identifiers, service identifiers, start time, end time, and status codes. The metric data streams include service identifiers, CPU utilization, memory utilization, request rate, and error rate. The data is cleaned and aligned with the time window. The various indicators of each service identifier within a fixed time window are aggregated into a multi-dimensional indicator vector. The tracking data of the service corresponding to the service identifier is aggregated into a tracking feature vector. The multi-dimensional indicator vector and the tracking feature vector are concatenated and normalized to form the real-time time-series feature vector corresponding to the service node corresponding to the service identifier; Based on the service identifier, the real-time time-series feature vector is used as an attribute and assigned to the corresponding service node in the real-time dependency graph; for dependency edges, the call frequency and average latency within the corresponding time window are extracted based on the service identifiers of both calling parties to form the real-time time-series feature vector of the edge.

8. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 1, characterized in that, The method further includes: Based on the fault propagation path predicted in the fault risk assessment report, targeted defense rule plans are dynamically generated. The defense rule plans include traffic pre-allocation for specific paths, elastic scaling pre-instructions for overloaded nodes, and resource reservation strategies for critical services. Before the predicted failure time window, the defense rule plan is preloaded into the policy execution engine, but does not take effect immediately; When the online monitoring system detects initial fault symptoms that match the prediction, it automatically triggers the policy execution engine, making the corresponding defense rule plan take effect immediately and executing the predefined defense actions.

9. The end-to-end intelligent fault simulation and assessment defense method in a microservice scenario according to claim 8, characterized in that, When the online monitoring system detects initial fault symptoms consistent with the prediction, it automatically triggers the policy execution engine, causing the corresponding defense rule plan to take effect immediately and execute predefined defense actions, including: Continuously compare the real-time indicators collected by the online monitoring system with the initial fault symptom indicator patterns predicted in the fault risk assessment report. When the real-time indicator matches the predicted initial fault symptom indicator pattern more than a set threshold within several consecutive detection cycles, it is determined that the predicted fault has begun to occur. Based on the specific symptom pattern matched, the most relevant plan is retrieved from the pre-loaded defense rule plans, and an immediate effective command is sent to the policy execution engine. Based on the immediate effective instructions, the policy execution engine executes the defensive actions defined in the contingency plan, including switching some traffic destined for the predicted fault source node to a backup instance, increasing resource quotas for nodes predicted to be affected in advance, and inserting specific circuit breaker or degradation rules at the link level.

Images