Distributed software self-repairing maintenance method and system based on deep reinforcement learning

Abstract

The application discloses a distributed software self-recovery maintenance method and system based on deep reinforcement learning, and relates to the technical field of computer system fault tolerance and self-recovery operation and maintenance. The method comprises the following steps: S1, collecting self-recovery operation data in real time and performing data preprocessing; S2, performing trusted evaluation on the consistency of the calling chain observation based on the upstream and downstream Span time sequence relationship of the cross-service calling chain, screening effective calling chain samples, and updating a service dependency topology structure diagram; S3, performing propagation situation strength evaluation on the calling edges in the service dependency topology structure diagram, and identifying key propagation edges and root cause candidate nodes; S4, constructing a reinforcement learning state input and a self-recovery action space, outputting a self-recovery action sequence through a policy network, performing action arrangement, and continuously optimizing the policy network. The problems that the distributed microservice abnormal propagation chain is difficult to accurately trace and self-recover due to the dynamic evolution of the calling topology and the incomplete cross-node time sequence observation are solved.

Description

Technical Field

[0001] This invention relates to the field of fault tolerance and self-healing operation and maintenance technology for computer systems, specifically to a distributed software self-healing maintenance method and system based on deep reinforcement learning. Background Technology

[0002] With the widespread application of cloud-native architecture and microservice technology, modern software systems are increasingly characterized by large service scale, complex call chains, and significant dynamic changes in operational status. System failures are no longer limited to the failure of a single node but can propagate and spread across multiple components along service dependencies, leading to cascading timeouts, performance degradation, and service unavailability. Simultaneously, monitoring data in distributed environments is characterized by multi-source heterogeneity, difficulties in time-series alignment, and continuous topology evolution. This makes traditional operation and maintenance methods based on static rules or single-point alarms insufficient to meet the needs for rapid perception and continuous maintenance of abnormal propagation trends. These technical challenges are particularly prominent in internet security service fields such as network and perimeter security software and basic network and information security software. Therefore, intelligent operation assurance and self-healing maintenance technologies for complex distributed software systems have gradually become an important research direction in this field.

[0003] For example, invention patent CN120295910A discloses a distributed Android application automated testing method based on reinforcement learning. Multiple agents execute the following steps in parallel: a) Independent testing phase: Each agent interacts with the application under test within a preset round duration, performing the following operations: generating a state representation based on the current interface structure; selecting and executing actions based on a reinforcement learning policy, recording state transitions and reward values; maintaining a global state set to track the states discovered by all agents; b) Experience aggregation phase: at the end of the preset round duration, aggregating the learning experiences of all agents to generate a global policy. The learning experience includes at least a Q-table and action execution frequency; c) Policy synchronization phase: distributing the global policy to all agents as the initial policy for the next time period; repeating steps a)-c) until the test terminates. Through distributed architecture design, a state similarity-driven reward function, and a periodic experience aggregation mechanism, efficient exploration and test case generation are achieved.

[0004] For example, invention patent CN114153640A discloses a system fault tolerance strategy method based on deep reinforcement learning, specifically involving a system fault tolerance strategy method based on deep reinforcement learning. This invention applies deep reinforcement learning technology to the fault tolerance problem of distributed stream processing systems, enabling the data backup allocation problem in fault tolerance to be expressed as a resource allocation problem. It constructs a neural network model with the relationships between tasks as vertices and edges, and through multi-level training, effectively improves the current system fault tolerance problem handling performance. Simultaneously, it provides a feasible method based on deep reinforcement learning for fault tolerance problems in distributed stream processing systems. The technical solution provided by this invention can appropriately allocate backup resources and evaluate system fault tolerance overhead, thereby reducing the complexity of current methods that primarily rely on upstream backup, improving system fault tolerance efficiency, and reducing the fault tolerance overhead of existing methods.

[0005] However, in microservices and distributed systems, common failures propagate across multiple nodes along call dependencies. For example, upstream latency jitter or resource contention can cause downstream queues to accumulate, further triggering timeouts, circuit breakers, and cascading failures. In current operational practices, some systems primarily rely on single-point metric threshold alarms or static dependencies to locate anomalies. When the call topology changes dynamically, link sampling is incomplete, or cross-node time alignment errors are large, alarms often only expose the first or most significant surface-level nodes that exceed limits, making it difficult to reconstruct the source service and propagation path of the anomaly. This results in self-healing actions being repeatedly executed on non-root cause nodes, failing to stop the propagation chain.

[0006] Therefore, in order to address the above problems, there is an urgent need for a distributed software self-healing maintenance method and system based on deep reinforcement learning. Summary of the Invention

[0007] Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a distributed software self-healing maintenance method and system based on deep reinforcement learning, which solves the problem of difficulty in accurately tracing the source and self-healing blocking of abnormal propagation chains in distributed microservices caused by the dynamic evolution of call topology and incomplete cross-node temporal observation.

[0008] Technical solution To achieve the above objectives, this invention provides the following technical solution: a distributed software self-healing maintenance method based on deep reinforcement learning, comprising the following steps: S1, real-time collection of self-healing operation data of microservice instances and call chains, and data preprocessing of the self-healing operation data to form an operation feature set; S2, based on the upstream and downstream span temporal relationship of cross-service call chains, a reliable assessment of the consistency of call chain observations is performed, effective call chain samples are selected, and the service dependency topology diagram is dynamically updated; S3, combining the operation features of the service dependency topology diagram and window self-healing operation data, the intensity assessment of the call edge propagation situation is performed, key propagation edges and root cause candidate nodes are identified, and a set of key propagation paths is generated; S4, the operation feature set, service dependency topology diagram, and propagation situation results are used to construct reinforcement learning state inputs, and a self-healing action space is constructed. The current self-healing action sequence is output through the policy network, and action orchestration is performed, and the policy network is continuously optimized.

[0009] Furthermore, the specific process of collecting self-healing runtime data of microservice instances and call chains in real time, and performing data preprocessing on the self-healing runtime data to form a runtime feature set is as follows: For all microservice instances and call chains, self-healing runtime data is collected in real time. The self-healing runtime data includes: call edge identifier, service dependency relationship, request response latency, request return status code, request timeout flag, number of downstream waiting tasks, Span start timestamp, Span end timestamp, call edge latency, and Span exception flag; a unified clock reference alignment process is performed on the self-healing runtime data to identify and mark abnormal Span records with causal violations; based on the request return status code, request timeout flag, and Span exception flag, consistency verification is performed on abnormal request samples and abnormal call samples to remove dirty data records; for the number of downstream waiting tasks, request response latency, and call edge latency, a neighborhood interpolation method is used. The formula is completed, and sliding robust statistics are used to suppress abnormal spikes and monitoring noise. Based on the sliding time window, statistical window indicators are calculated, including: the request response latency and call edge latency sample sets within the window are sorted and the latency value corresponding to the 95th percentile position is taken to obtain the request response P95 latency and call edge P95 latency; the number of requests within the window is divided by the window duration to obtain the request arrival rate; the number of requests that time out within the window is counted to obtain the request timeout rate; the number of requests that return abnormal status codes within the window is counted to obtain the request error rate; the number of abnormal Spans on the call edge within the window is counted to obtain the call edge error rate; normalization processing is performed on the self-healing operation data and window indicators; a self-healing maintenance database is established to store the original and preprocessed self-healing operation data and window indicators.

[0010] Furthermore, based on the upstream and downstream span time-series relationship of the cross-service call chain, the specific process for reliable evaluation of the call chain observation consistency is as follows: During the cross-service call process, when the calling microservice instance initiates a remote request to the called microservice instance, the call span generated by the calling party is marked as the upstream call span, and the call span generated by the called party is marked as the downstream call span; the difference between the start timestamp of the downstream call span and the end timestamp of the upstream call span is calculated to obtain the causal residual of the call chain; based on the sliding time window, the median absolute deviation of the causal residual is calculated as the causal residual fluctuation intensity value; and so on. At that time, the proportion of the number of abnormal Spans that violated causality within the statistical window to the total number of Spans within the window is used to obtain the causality violation rate; the median of the historical causality residual fluctuation intensity value and the causality violation rate are taken to obtain the residual fluctuation benchmark term and the violation rate benchmark term, respectively; the causality residual fluctuation intensity value is divided by the sum of the residual fluctuation benchmark term and the smallest positive number, and the negative value is used to perform natural exponential operation to obtain the residual stability decay term; the causality violation rate is divided by the sum of the violation rate benchmark term and the smallest positive number, and the negative value is used to perform natural exponential operation to obtain the causality consistency decay term; the residual stability decay term and the causality consistency decay term are multiplied together to obtain the time-series alignment reliability value of the cross-service call chain.

[0011] Furthermore, the specific process of screening valid call chain samples and dynamically updating the service dependency topology graph is as follows: The time-series alignment confidence value is compared with the confidence threshold. When the time-series alignment confidence value is greater than or equal to the confidence threshold, the time-series observation of the call chain within the current window is determined to be reliable, and valid labels are generated for the corresponding self-healing operation data and window metrics. When the time-series alignment confidence value is less than the confidence threshold, the time-series observation of the call chain within the current window is determined to be unreliable, and unreliable removal processing is performed on the self-healing operation data and window metrics of the corresponding call chain within the window. Based on the call edge identifier, the caller microservice instances and callee microservice instances within the window are counted as topology nodes, and the actual cross-service call records within the window are used as directed call edges between nodes. A service call relationship graph within the current window is constructed, and the newly added and failed call edges are updated consistently in conjunction with the service dependency relationship to obtain a service dependency topology graph that changes with the time window.

[0012] Furthermore, combining the operational characteristics of the service dependency topology graph and the window self-healing operation data, the specific process for assessing the intensity of the call edge propagation trend is as follows: For each call edge in the service dependency topology graph, the effective call edge P95 latency, call edge error rate, and the number of downstream waiting tasks corresponding to the callee are read. The difference between adjacent sampling times is calculated and divided by the sampling time interval to obtain the edge latency growth rate, edge error growth rate, and queuing growth rate, which are taken as three growth rate components. The median absolute deviation of the three growth rate components in the corresponding historical sample set is calculated to obtain the corresponding robust fluctuation scale, and the median of the corresponding historical growth rate components is taken to obtain the corresponding historical center benchmark. For each of the three growth rate components, the historical center benchmark is subtracted from its own value and divided by the sum of the robust fluctuation scale and the smallest positive number to obtain the corresponding robust standardized value. The robust standardized values ​​corresponding to the three growth rate components are subjected to natural exponential operation and summed. The natural logarithm of the summation result is taken to obtain the propagation surge intensity value.

[0013] Furthermore, the specific process of identifying key propagation edges and root cause candidate nodes, and generating a set of key propagation paths, is as follows: Based on the propagation surge intensity values ​​of all call edges within the current sliding time window, a set of call edge propagation intensity values ​​is constructed, and a consistent mapping of the propagation surge intensity values ​​is performed based on the service dependency topology graph, so that each call edge corresponds to a unique propagation surge intensity label; the propagation surge intensity values ​​within the window are sorted in descending order, and the top K call edges are used to construct a set of key call edges; the caller microservice instances in the set of key call edges are counted, and when the same caller node has at least one robustly normalized value of a growth rate component continuously exceeding the growth threshold within the most recent L windows, the following criteria are applied: When the calling node is identified as a root cause candidate service node, a root cause candidate service node set is generated. Starting from the root cause candidate service node, the calling edges are traversed along the service dependency topology graph, and a threshold is determined for the propagation surge intensity value of each calling edge. When the propagation surge intensity value of a calling edge is greater than or equal to the intensity threshold, the calling edge is marked as a valid propagation edge, and the propagation link continues to be extended along the downstream nodes of the calling edge. When all adjacent calling edges in the calling edge sequence formed from the root cause candidate node are valid propagation edges, and the sequence length is not less than the minimum propagation hop count threshold, the calling edge sequence is determined as a critical propagation path, and the sequence is summarized to form a critical propagation path set.

[0014] Furthermore, the specific process of constructing the reinforcement learning state input by building the runtime feature set, service dependency topology graph, and propagation status results, and constructing the self-healing action space is as follows: Based on the self-healing runtime data and window metrics, service dependency topology graph, time-series alignment reliability value, call edge propagation surge intensity value set, root cause candidate service node set, and key propagation path set, a reinforcement learning state vector within the current sliding time window is constructed; the self-healing action space is defined, which includes executable self-healing maintenance actions for key call edges and root cause candidate service nodes, specifically including: performing rate limiting control, circuit breaker isolation, and retry suppression actions on call edges on key propagation paths; performing instance expansion, service restart, and resource quota adjustment actions on root cause candidate service nodes; and performing augmented sampling actions on call chain observations.

[0015] Furthermore, the specific process of outputting the current self-healing action sequence and executing action orchestration through the policy network is as follows: using the reinforcement learning state vector as input, the self-healing policy network model is trained using the proximal policy optimization algorithm (PPO), and the parameters of the self-healing policy network model are updated by maximizing the cumulative reward function; the current reinforcement learning state vector is input into the self-healing policy network model, and the optimal self-healing action sequence is output; the optimal self-healing action sequence is sent to the execution layer for action orchestration and implementation, and the execution layer determines the target of the action based on the service dependency topology diagram.

[0016] Furthermore, the specific process of continuous optimization of the policy network is as follows: After the action is executed, the request response P95 latency, the call edge P95 latency, the call edge error rate, and the number of downstream waiting tasks are continuously monitored, and the propagation surge intensity value of the call edge is updated. The propagation and diffusion risk value is obtained by aggregating the propagation surge intensity values ​​of all call edges, and the decrease rate of the propagation and diffusion risk value is calculated. The decrease rate of the request response P95 latency, the call edge P95 latency, the call edge error rate, and the number of downstream waiting tasks are combined with the decrease rate as reinforcement learning reward signals and fed back to the self-healing policy network model to drive the continuous optimization of model parameters.

[0017] The second aspect of this invention provides a distributed software self-healing maintenance system based on deep reinforcement learning, comprising: a self-healing operation data acquisition and processing module, used to collect self-healing operation data of microservice instances and call chains in real time, and perform data preprocessing on the self-healing operation data to form an operation feature set; a call chain temporal reliability assessment module, used to perform reliability assessment on the consistency of call chain observations based on the upstream and downstream span temporal relationships of cross-service call chains, screen effective call chain samples, and dynamically update the service dependency topology diagram; an anomaly propagation chain situation modeling module, used to combine the service dependency topology diagram and the operation features of window self-healing operation data to perform strength assessment on the propagation situation of call edges, identify key propagation edges and root cause candidate nodes, and generate a set of key propagation paths; and a reinforcement learning self-healing decision optimization module, used to construct the operation feature set, service dependency topology diagram, and propagation situation results as reinforcement learning state inputs, construct a self-healing action space, output the current self-healing action sequence through a policy network and execute action orchestration, and continuously optimize the policy network.

[0018] Beneficial effects The present invention has the following beneficial effects: (1) This invention uses a call chain observation consistency reliability assessment mechanism based on upstream and downstream Span timing relationship to make reliable judgments on cross-node clock deviation and link sampling missing by utilizing causal residual fluctuation and causal violation rate, thereby achieving the screening and effective labeling of untrusted call chain samples, thus significantly improving the availability of distributed operation data and the reliability of subsequent modeling input.

[0019] (2) This invention constructs a propagation surge intensity assessment method for call edge, transforms call edge latency, error rate and queuing characteristics into growth rate components and performs robust standardization fusion, which can effectively characterize the diffusion intensity of anomalies along service dependency topology, thereby identifying key propagation edges, root cause candidate service nodes and key propagation path sets, and achieving accurate location and situational reconstruction of anomaly sources and propagation links.

[0020] (3) In this invention, the running feature set, dynamic topology diagram and propagation situation results are jointly constructed as reinforcement learning state input, and the action space covering rate limiting, circuit breaking, expansion and restart is defined. The self-healing policy network is trained by the near-end policy optimization algorithm PPO to realize adaptive mapping from complex propagation situation to optimal self-healing action sequence, avoiding the problem of action rigidity and false repair under traditional static rules.

[0021] (4) This invention continuously monitors the propagation situation and changes in operational indicators after the action is executed, constructs a reinforcement learning reward signal based on the decrease in propagation and diffusion risk and the decline in key performance, drives the policy network to continuously update, realizes a closed-loop self-learning optimization process of "observation, decision-making, execution, and feedback", thereby effectively suppressing the cascading diffusion of anomalies, shortening the fault recovery time and improving the long-term operational stability of the system.

[0022] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0023] Figure 1 The flowchart shows a distributed software self-healing maintenance method based on deep reinforcement learning. Figure 2 This is a structural diagram of a distributed software self-healing maintenance system based on deep reinforcement learning. Figure 3 A distribution map of the propagation surge intensity values; Figure 4 This is a flowchart of a closed-loop process for distributed software self-healing maintenance based on deep reinforcement learning. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. As those skilled in the art will understand, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figures 1-4 This invention provides a technical solution: a distributed software self-healing maintenance method based on deep reinforcement learning, such as... Figure 1 As shown, the process includes the following steps: S1, real-time collection of self-healing operation data of microservice instances and call chains, and data preprocessing of the self-healing operation data to form an operation feature set; S2, based on the upstream and downstream span temporal relationship of cross-service call chains, a reliable assessment of the consistency of call chain observations is performed, effective call chain samples are selected, and the service dependency topology diagram is dynamically updated; S3, combining the operation features of the service dependency topology diagram and window self-healing operation data, the intensity assessment of the call edge propagation situation is performed, key propagation edges and root cause candidate nodes are identified, and a set of key propagation paths is generated; S4, the operation feature set, service dependency topology diagram, and propagation situation results are used to construct reinforcement learning state inputs, and a self-healing action space is constructed. The current self-healing action sequence is output through the policy network, and action orchestration is performed, and the policy network is continuously optimized.

[0026] Specifically, the process of collecting self-healing runtime data of microservice instances and call chains in real time, and performing data preprocessing on the self-healing runtime data to form a runtime feature set is as follows: In a microservice system, the call chain is triggered frequently and repeatedly with business traffic, forming a continuous call sample sequence. For all microservice instances and call chains, self-healing runtime data is collected in real time. The self-healing runtime data includes: call edge identifier, service dependency relationship, request response latency, request return status code, request timeout flag, number of downstream waiting tasks, Span start timestamp, Span end timestamp, call edge latency, and Span exception flag. Among them, the request response latency is obtained by the gateway or server recording the timestamps when the request enters and returns and taking the difference. The request return status code is directly obtained from the HTTP / gRPC response header field. The request timeout flag is automatically set by the service governance component when the request duration exceeds the timeout configuration threshold. The number of downstream waiting tasks is obtained in real time from the thread pool queue count of the called microservice instance. The Span start timestamp and Span end timestamp are automatically generated by the distributed tracing component when the request enters and ends. The call edge latency is calculated by subtracting the Span start timestamp from the Span end timestamp, which is used to characterize the end-to-end call time between the caller and the callee. The Span exception flag is generated by the tracing system when it detects an error return, an abnormal interruption, or a missing Span. The call edge identifier is generated by a unique key composed of the caller service name, the callee service name, and the interface path. The service dependency relationship is dynamically summarized from the actual set of call edges that appear in the window. A unified clock reference alignment process is performed on the self-healing operation data to identify and mark abnormal Span records with causal violations. This unified clock reference alignment can be achieved by synchronizing the system time of each node via NTP and collecting node clock offset and jitter metrics within each sliding time window for timing quality verification. When a node clock offset or jitter exceeds a preset limit, the corresponding Span is marked as having a time reference anomaly and is removed as untrusted, ensuring that Span timestamps generated by different microservice instances are under the same time reference. Anomalous Span records with causal violations refer to timing anomalies where the downstream Span start timestamp is earlier than the upstream Span end timestamp. Consistency verification is performed on abnormal request samples and abnormal call samples based on request return status codes, request timeout flags, and Span anomaly flags, removing dirty data records. Request return status codes indicate whether the call returned successfully, request timeout flags indicate whether the request exceeded a preset timeout threshold, and Span anomaly flags indicate erroneous or missing Spans detected in the tracing process. Cross-verification of these three factors avoids data deviations caused by false alarms from a single monitoring source.To address downstream waiting task counts, request response latency, and call edge latency, neighborhood interpolation is used to fill in the gaps, and sliding robust statistics are employed to suppress abnormal spikes and monitoring noise. The neighborhood interpolation method is preferably linear interpolation based on adjacent sampling points or nearest neighbor completion to correct short-term monitoring gaps. Sliding robust statistics preferably use a median and median absolute deviation (MAD) mechanism to suppress inaccurate spikes caused by instantaneous traffic surges or sampling jitter. A sliding time window is used to calculate statistical window metrics. The sliding time window is preferably a fixed-length window, ideally between 5 and 120 seconds, to balance rapid anomaly propagation detection with the stability of statistical quantiles. The window is updated in a rolling manner, with a step size preferably 1 / 5 of the window length. The sliding time window length is determined based on the typical RTT of the microservice link and the time scale of fault propagation. If the window is too short, the quantiles are easily affected by instantaneous jitter; if the window is too long, it will reduce the response sensitivity to anomaly bursts. Furthermore, since microservice call chain metrics are continuous time-series monitoring data, the fixed duration can be consistent with the real-time alarm cycle, tracing refresh cycle, and self-healing action execution cycle, thus achieving an optimal balance between real-time performance and statistical robustness. Window metrics include: sorting the request-response latency and call-edge latency samples within the window, and taking the latency value corresponding to the 95th percentile position to obtain the request-response P95 latency and call-edge P95 latency; where the request-response P95 latency characterizes the tail latency level at the service instance level, and the call-edge P95 latency characterizes the tail transmission latency level at the call chain level, both derived from the statistical quantile calculation of observable latency samples within the window. The request arrival rate is obtained by dividing the number of requests within the window by the window duration; where the number of requests is directly counted from the total number of requests entering within the window, and the window duration is a preset statistical period used to quantify the load intensity per unit time. The request timeout rate is calculated as the ratio of the number of requests that timed out within a given window to the total number of requests within that window. The request error rate is calculated as the ratio of the number of requests that returned an abnormal status code within a given window to the total number of requests within that window. The call edge error rate is calculated as the ratio of the number of abnormal Spans on the call edges within a given window to the total number of call edge Spans. Among these, the request timeout rate, request error rate, and call edge error rate are all dimensionless proportional indicators used to characterize the frequency of abnormal events occurring within the window. The self-healing operation data and window indicators are subjected to minimum-maximum scale unified normalization processing to ensure that indicators of different dimensions are within a unified numerical range in subsequent propagation modeling and reinforcement learning state construction. A self-healing maintenance database is established to store the original and preprocessed self-healing operation data and window indicators for long-term preservation of monitoring data, cleaning results, and statistical characteristics of each window. This supports anomaly propagation chain situation modeling, reinforcement learning strategy training, and backtracking analysis of self-healing effects, thereby ensuring the sustainable implementation and closed-loop optimization of the overall process.

[0027] In this implementation plan, key operational data of microservice instances and call chains are collected in real time. Based on unified clock alignment, anomaly consistency verification, missing data completion, and robust statistical noise suppression, highly reliable preprocessing of cross-service call observation data is achieved. At the same time, a fixed-duration sliding window is used to perform stable statistics and normalization mapping on indicators such as tail latency, error rate, and load intensity. A unified operational feature set that can directly support anomaly propagation situation modeling and deep reinforcement learning self-healing decision-making is constructed. Long-term traceable storage is achieved through a self-healing maintenance database, thereby effectively improving the accuracy of distributed system fault propagation perception, the robustness of feature input, and the feasibility of self-healing maintenance closed-loop optimization.

[0028] Specifically, the process of reliably assessing the consistency of call chain observations based on the upstream and downstream span timing relationships of cross-service call chains is as follows: During a cross-service call, when the calling microservice instance initiates a remote request to the called microservice instance, the call span generated by the calling party is marked as the upstream call span, and the call span generated by the called party is marked as the downstream call span. The upstream call span corresponds to the entry tracing segment of the remote call initiated by the calling party, and the downstream call span corresponds to the processing tracing segment after the called party receives the request. Both have unique trace identifiers in the distributed tracing system and can form an upstream-downstream timing association. To ensure the comparability of span timestamps generated by different microservice nodes, NTP periodic synchronization is used during preprocessing to align the system time of each node and record the node clock offset and jitter status within each sliding time window. This supports reliable calculation of the cross-node span timing relationships, thereby avoiding residual misjudgments caused by inconsistent clock references. The causal residual of the call chain is obtained by calculating the difference between the start timestamp of the downstream Span call and the end timestamp of the upstream Span call. This causal residual characterizes the timing offset caused by clock synchronization errors, link sampling jitter, or asynchronous scheduling in cross-service calls. A negative causal residual indicates that the downstream observation ends earlier than the upstream one, a typical causal violation. Furthermore, the causal residual is constructed based on the direct parent-child Span call relationship within the same Trace. The start timestamp of the downstream Span and the end timestamp of the upstream Span should satisfy a non-negativity constraint in a normal causal chain. Therefore, the causal residual can be used to capture timing consistency violations caused by out-of-order Span calls, time drift, or missing link traces. Based on a sliding time window, the median absolute deviation (MAD) of the causal residual is calculated as the causal residual fluctuation intensity value. The MAD is robustly statistically analyzed by comparing the deviation of all causal residual samples within the window from the median, effectively suppressing the interference of a small number of abnormal spike residuals on the overall fluctuation assessment. The reason for using MAD instead of mean-variance is that even if systematic clock skew occurs at different nodes, causing an overall shift in the residual sequence, this type of near-constant skew will not significantly increase the dispersion of the residuals within the window. Therefore, MAD is insensitive to systematic skew but highly sensitive to non-stationary fluctuations such as drift jitter and out-of-order spikes, thus reliably characterizing the true fluctuation intensity of the residuals even in the presence of clock skew. Simultaneously, the proportion of anomalous spans that violate causality within the window to the total number of spans within the window is used to obtain the causality violation rate. The number of anomalous spans is directly counted from call records within the window where the residual is less than zero or where timestamps are reversed. The causality violation rate is used to characterize the frequency of timing anomalies in the call chain within the current window.By simultaneously introducing the causal violation rate, misjudgments caused by relying solely on residual amplitude can be further avoided. When systematic shifts exist but do not trigger causal reversals, the violation rate remains low, thus avoiding erroneous reduction in reliability. However, when Span out-of-order events occur frequently, the violation rate rises rapidly, effectively characterizing the degree of disruption to observation consistency. The median of historical causal residual fluctuation intensity values ​​and causal violation rates are taken to obtain the residual fluctuation benchmark term and the violation rate benchmark term. The historical benchmark term is derived from the long-term accumulated call window statistics in the self-healing maintenance database, used to provide a stable reference scale for comparison between different service call chains. The number of historical samples is preferably between 100 and 1000 sliding window samples to ensure that the median statistics have sufficient historical coverage and suppress benchmark drift caused by short-term operating condition fluctuations. When the number of historical samples is too small, the benchmark term is easily affected by occasional abnormal windows and becomes unstable. When the number of historical samples is too large, it will introduce too long-term operating states, causing the benchmark to lag behind the current system load and topology changes. Therefore, adopting the above-mentioned optimal range can achieve a balance between benchmark stability and dynamic adaptability. The residual stability decay term is obtained by dividing the causal residual fluctuation intensity value by the sum of the residual fluctuation baseline term and the smallest positive number, and then performing a natural exponential operation on the negative value. The smallest positive number is used to avoid a zero denominator, and the exponential decay form is used to quickly reduce the stability contribution when residual fluctuation increases significantly. The causal consistency decay term is obtained by dividing the causal violation rate by the sum of the violation rate baseline term and the smallest positive number, and then performing a natural exponential operation on the negative value. The causal consistency decay term is used to reduce the observation consistency contribution when causal violations occur frequently within the window, thereby preventing untrusted call chain samples from entering the subsequent propagation modeling process. Multiplying the residual stability decay term and the causal consistency decay term yields the temporal alignment confidence value across service call chains. To address the inconsistency in call chain timing caused by cross-node clock skew, incomplete link sampling, and out-of-order Span events in distributed microservice environments, this paper characterizes the stability and consistency of call chain observations from two dimensions: the intensity of causal residual fluctuations and the proportion of causal violation events. An exponential decay mechanism rapidly reduces the reliability as residual fluctuations or violation rates increase, generating a comprehensive reliability index that reflects the reliability of call chain timing. This index is used to screen valid call chain samples and ensure the reliability of subsequent anomaly propagation modeling and self-healing decision inputs. Because it integrates both "residual fluctuation intensity" and "causal violation frequency" dimensions, and amplifies the impact of anomalies through historical benchmark normalization and exponential decay, the reliability value remains sensitive to drift jitter, out-of-order Span events, and observation inconsistencies even in the presence of systemic clock skew, ensuring the reliability of subsequent anomaly propagation situation modeling and reinforcement learning decision inputs. A timing alignment reliability value closer to 1 indicates a more reliable call chain timing, while a value closer to 0 indicates a significant bias in call chain observations, which can be used for subsequent valid sample screening and topology updates.

[0029] The specific formula for the time alignment confidence value is as follows: ; In the formula, The temporal alignment confidence value is used to characterize the degree of temporal alignment confidence of the cross-service call chain within the current sliding window, in order to determine whether the call chain observation data can be used as a valid input for subsequent propagation modeling and self-healing decisions. This represents the set of historical samples used to build a long-term stable benchmark. It represents the causal residual of the call chain, used to characterize the causal residual deviation between upstream and downstream Spans during a call, thereby reflecting the timing inconsistency risk caused by cross-node clock drift or link out-of-order. This represents the causal residual fluctuation intensity value, used to robustly measure the fluctuation intensity of the causal residual sequence within a window to identify trends of declining temporal stability in the call chain; The residual fluctuation benchmark term serves as a historical benchmark scale for the intensity of residual fluctuation, used to map the current fluctuation to the degree of relative deviation and achieve normalized comparison. This represents the causal violation rate, used to reflect the proportion of occurrences of causal violation anomalies (Span) within a window, in order to quantify the frequency and consistency quality of violation events in the call chain observation. This represents the benchmark term for the violation rate, which serves as a historical benchmark term for the causal violation rate and is used to characterize the degree of abnormality of the current violation level relative to the normal state. This represents a very small positive number that ensures the denominator is not zero and enhances the stability of exponential decay calculations. The preferred value is [value missing]. .

[0030] In this implementation scheme, by utilizing the temporal correlation of upstream and downstream Spans across service call chains, a temporal alignment reliability assessment mechanism combining causal residual fluctuation intensity and causal violation rate is constructed. Furthermore, by introducing historical benchmark scale and exponential decay form, robust quantitative judgment of call chain observation consistency is achieved. This enables the effective identification of untrusted call chain samples under complex distributed operating conditions such as clock skew, sampling gaps, and Span out-of-order conditions, ensuring the reliability of subsequent propagation situation modeling and reinforcement learning self-healing decision inputs, and improving the stability and feasibility of anomaly localization and self-healing maintenance closed loop.

[0031] Specifically, the process of selecting valid call chain samples and dynamically updating the service dependency topology graph is as follows: The temporal alignment confidence value of the call chain within the window is calculated, and then compared with a confidence threshold. When the temporal alignment confidence value is greater than or equal to the confidence threshold, the temporal observation of the call chain within the current window is determined to be reliable, and a valid label is generated for the corresponding self-healing operation data and window metrics. The valid label indicates that the call chain sample has reliable temporal consistency and can serve as a valid input for subsequent anomaly propagation situation assessment and reinforcement learning state construction. When the temporal alignment confidence value is less than the confidence threshold, the temporal observation of the call chain within the current window is determined to be unreliable, and unreliable removal processing is performed on the self-healing operation data and window metrics of the corresponding call chain within the window. Unreliable removal processing refers to removing the call chain sample from the current window statistical set and propagation modeling input set to avoid false propagation paths caused by clock drift or Span out-of-order entering the subsequent decision-making process, thereby ensuring the accuracy of propagation chain inference. Based on call edge identifiers, the calling and called microservice instances within a statistical window are used as topology nodes, and the actual cross-service call records within the window are used as directed call edges between nodes to construct a service call relationship graph within the current window. The calling and called microservice instances correspond to the upstream and downstream service nodes of the call chain, respectively, and the direction of the directed call edges points from the caller to the callee, depicting the propagation direction of anomalies in the dependency relationship. Furthermore, the newly added and failed call edges are consistently updated in conjunction with the service dependency relationship, resulting in a service dependency topology graph that changes over time. Consistent updates refer to adding a new call edge that was not previously recorded within the window and marking it as a new dependency; conversely, if a call edge does not appear in multiple consecutive windows, it is marked as a failed dependency and removed from the topology. This allows the topology graph to dynamically adapt to the actual operating conditions of microservice elastic scaling, instance migration, and call relationship evolution, ensuring that subsequent propagation chain situation modeling and root cause localization are implemented based on the latest and most valid service dependency structure.

[0032] In this implementation, by performing time-series alignment and reliability determination on call chain samples within a sliding time window, untrusted observation data is automatically eliminated and valid samples are screened, thereby avoiding interference from pseudo-propagation links caused by clock drift and Span out-of-order. At the same time, a service call relationship graph is dynamically constructed based on the actual call edge records within the window, and new and failed dependencies are updated consistently, so that the service dependency topology can be adaptively updated in real time with the scaling up and down of microservices and the evolution of call relationships. This provides the latest and most reliable topological structure support for subsequent anomaly propagation situation modeling, root cause candidate localization, and reinforcement learning self-healing decision-making, improving the accuracy and feasibility of the distributed self-healing maintenance process.

[0033] Specifically, combining the operational characteristics of the service dependency topology diagram and the window self-healing operation data, the specific process for assessing the intensity of call edge propagation is as follows: For each call edge in the service dependency topology diagram, the effective call edge P95 latency, call edge error rate, and the number of downstream waiting tasks corresponding to the callee are read. The call edge P95 latency is derived from the tail-partial statistics of the call latency sample set of that call edge within the sliding time window, used to characterize the high latency risk level of the call chain; the call edge error rate is derived from the proportion of the number of abnormal spans within the window to the total number of spans of that call edge, used to characterize the frequency of abnormal returns on that call edge; the number of downstream waiting tasks is obtained in real-time from the thread pool queuing length of the microservice instance of the callee, used to characterize the queuing backlog of downstream services. The difference between adjacent sampling times is calculated and divided by the sampling time interval to obtain the edge latency growth rate, edge error growth rate, and queuing growth rate, as three growth rate components. Adjacent sampling times refer to two consecutive sliding window update times, and the sampling time interval is the window step size. Difference normalization can characterize the growth trend of the indicators within a unit of time, thus reflecting the dynamic evolution speed of abnormal propagation. The median absolute deviation (MAD) of each of the three growth rate components within their respective historical sample sets is calculated to obtain the corresponding robust fluctuation scales. The median of each historical growth rate component is then used to obtain the corresponding historical center benchmark. The historical sample set is derived from multi-window growth rate sequences accumulated in a self-healing maintenance database. The median represents the typical center level under normal operating conditions, and the MAD represents the normal fluctuation range and suppresses the interference of a few abnormal spikes on scale estimation, thus ensuring comparable robust benchmarks for growth rate components across different call edges. For each of the three growth rate components, its own value is subtracted from the historical center benchmark, and then divided by the sum of the robust fluctuation scale and a minimum positive number to obtain the corresponding robust standardized value. The minimum positive number is used to avoid numerical instability when the scale is zero, and the robust standardized value uniformly characterizes the degree of deviation of the current growth rate from the historical normal level; a larger value indicates a more significant abnormal surge. The robust standardized values ​​corresponding to the three growth rate components are then subjected to natural exponential operation and summed. The natural logarithm of the sum is taken to obtain the propagation surge intensity value. Among them, the exponential operation is used to highlight the amplification effect of the significant surge of any component on the propagation situation, and the natural logarithm is used to compress the magnitude and maintain the interpretability of the intensity value, thereby forming a call-side propagation surge intensity index that can comprehensively characterize the three propagation factors of delayed propagation, error propagation and queuing accumulation, which is used for subsequent key propagation side identification and propagation path generation.To address the propagation characteristics of anomalies spreading along call edges in distributed call chains, which are typically accompanied by simultaneous increases in latency, errors, and queue backlog, three dynamic components—edge latency growth rate, edge error growth rate, and queue growth rate—are constructed. A comparable and standardized characterization across call edges is achieved using a historical center benchmark and a robust fluctuation scale. Furthermore, exponential aggregation and logarithmic compression are employed to highlight the amplifying effect of any component surge on the propagation trend, thereby generating a quantitative intensity index that comprehensively reflects the degree of anomaly propagation enhancement at call edges. This index is used for identifying key propagation edges and inferring propagation paths.

[0034] The specific formula for the propagation surge intensity value is as follows: ; In the formula, The propagation surge intensity value represents the unified characterization of the three types of abnormal propagation trends of latency deterioration, error propagation and queue accumulation on the call edge within the current window. This quantifies the strength of the spread of the anomaly along the service dependency topology on the call edge, so that when any propagation component experiences a significant surge, the propagation surge intensity value can increase synchronously to highlight the key propagation edge. The robust normalized value of the edge latency growth rate component is used to reflect the propagation trend of the continuously increasing edge latency within the window. Robust normalized values ​​representing the edge error growth rate component are used to characterize the extent to which abnormal status codes, timeouts, or Span exceptions propagate and amplify on the calling edge. The robust normalized value representing the queuing growth rate component is used to characterize the accelerating trend of the number of downstream waiting tasks on the callee.

[0035] In this embodiment, Table 1 is a data table of propagation surge intensity values. The table details the robust normalized values ​​of the edge delay growth rate component, the edge error growth rate component, the queuing growth rate component, and the propagation surge intensity value for the five calling edges. Specifically, the robust normalized values ​​for the edge delay growth rate component of calling edge 1 are 0.22, 0.10, 0.15, and 1.2565; for calling edge 2, the robust normalized values ​​are 0.65, 0.30, 0.55, and 1.6092; and for calling edge 3, the robust normalized values ​​are 1.40 and 1.40 respectively. The normalization value is 0.95, the robust normalization value of the queue growth rate component is 1.10, and the propagation surge strength value is 2.2664; the robust normalization value of the edge delay growth rate component corresponding to call edge 4 is 2.10, the robust normalization value of the edge error growth rate component is 1.85, the robust normalization value of the queue growth rate component is 2.30, and the propagation surge strength value is 3.1987; the robust normalization value of the edge delay growth rate component corresponding to call edge 5 is 0.90, the robust normalization value of the edge error growth rate component is 0.60, the robust normalization value of the queue growth rate component is 0.75, and the propagation surge strength value is 1.8561.

[0036] Table 1 Data on the Intensity Values ​​of Transmission Surge like Figure 3 The figure shows the distribution of propagation surge intensity values. In the figure, the horizontal axis represents different service call edge numbers, and the vertical axis represents the propagation surge intensity value of the corresponding call edge, used to characterize the strength of anomaly propagation along the call edge. The larger the bar, the more significant the anomaly propagation trend is under the combined effects of latency surge, error propagation, or queuing backlog propagation factors. (Based on Table 1 and...) Figure 3 It can be seen that: Call edge 4 has the highest propagation surge intensity value, indicating that the anomaly spreads most prominently on this call edge, making it a key propagation edge; call edge 3 is second, also showing a strong propagation enhancement trend; call edge 1 has the lowest propagation surge intensity value, indicating that its propagation is relatively weak; call edges 2 and 5 are at a medium level, showing a certain degree of propagation surge but not reaching the risk level of the strongest propagation edge. Therefore, overall, it can intuitively support the identification of key propagation edges and provide quantitative basis for subsequent root cause candidate node location and key propagation path generation.

[0037] In this implementation scheme, key operational characteristics such as tail latency, error rate, and downstream queuing load of the calling edge are extracted under the constraints of service-dependent topology. Three dynamic propagation components, namely edge latency growth rate, edge error growth rate, and queuing growth rate, are constructed to characterize the temporal evolution of the anomaly propagation trend. At the same time, a historical center benchmark and robust fluctuation scale are introduced for standardized evaluation, and significant surge factors are highlighted through exponential aggregation. This forms a propagation surge intensity index that can comprehensively reflect the degree of propagation enhancement of the calling edge, providing a reliable quantitative basis for key propagation edge identification, propagation path inference, and subsequent self-healing decision-making, thereby improving the accuracy and feasibility of anomaly propagation modeling.

[0038] Specifically, the process of identifying key propagation edges and root cause candidate nodes, and generating a set of key propagation paths, is as follows: Based on the propagation surge intensity values ​​of all call edges within the current sliding time window, a set of call edge propagation intensity values ​​is constructed. A consistent mapping is then performed on the propagation surge intensity values ​​based on the service dependency topology graph, ensuring that each call edge corresponds to a unique propagation surge intensity label. The call edge propagation intensity set is used to summarize the propagation momentum intensity of all call edges within the current window. The consistent mapping refers to binding the propagation surge intensity values ​​to the corresponding directed edges in the topology graph according to the call edge identifier, thereby ensuring that the propagation intensity can be uniquely located on the topology graph and traversed subsequently. The propagation surge intensity values ​​within the window are sorted in descending order, and the top K call edges are used to construct a set of key call edges. Here, K is a preset parameter for the number of key edges, preferably 1% to 10% of the total number of topology call edges, used to focus on high-risk call edges with the most significant propagation surges, avoiding redundant calculations caused by full graph traversal. In the set of key call edge instances, when the robust normalized value of at least one growth rate component continuously exceeds the growth threshold within the most recent L windows, the caller node is determined to be a root cause candidate service node, and a root cause candidate service node set is generated. The most recent L windows are used to characterize the continuous trend of propagation surge, preferably 3 to 10 windows to avoid misjudgment caused by instantaneous jitter in a single window. The growth threshold is used to determine the significance of the growth rate deviating from the historical center. The robust normalized value continuously exceeding the threshold indicates that the caller node has a stable surge drive in at least one dimension of delay propagation, error propagation, or queue accumulation, thus possessing root cause source characteristics. The aforementioned "continuous surge in caller nodes" judgment rule can significantly reduce the situation of mistakenly identifying downstream nodes as the root cause: In typical cascading propagation scenarios, the increased latency and queuing of downstream services are often induced by abnormal request injections from upstream, which is a passive result of pressure. The real source of propagation is usually manifested as a continuous surge in the growth rate of the call edge output when it acts as a caller. Therefore, priority is given to counting the caller nodes of key call edges, and their growth rate component is required to continuously exceed the threshold within multiple windows. This constrains the generation of root cause candidates from the perspective of the temporal persistence and directionality of propagation, thereby avoiding the misjudgment of the victim node as the root cause node based solely on the downstream high latency phenomenon in a single window.Starting from the root cause candidate service node, the call edges are traversed along the service dependency topology graph. A threshold is applied to the propagation surge intensity value of each call edge. When the propagation surge intensity value of a call edge is greater than or equal to the intensity threshold, the call edge is marked as a valid propagation edge, and the propagation link continues to extend downstream nodes. The intensity threshold is preferably determined based on the quantile statistics of the propagation surge intensity value within a window, used to screen call edges with significant propagation trends. Valid propagation edges characterize the main diffusion channels of anomaly propagation in the topology. When all adjacent call edges in the call edge sequence formed from the root cause candidate node are valid propagation edges, and the sequence length is not less than the minimum propagation hop count threshold, the call edge sequence is determined as a critical propagation path, and a set of critical propagation paths is formed. The minimum propagation hop count threshold is preferably 2 to 5 hops to avoid misjudging locally isolated anomaly edges as propagation links. The set of critical propagation paths characterizes the main propagation direction and range of the anomaly in the service dependency topology, thus providing clear targets and propagation blocking locations for subsequent self-healing action orchestration. Through the path generation mechanism of "starting from the root cause candidate node, expanding the effective propagation edge, and constraining the minimum number of hops", it is possible to retain only the continuous propagation sequence from the upstream driving node to the downstream, without directly promoting the downstream local isolated high-latency edge to the root cause path. From the perspective of topological propagation coherence, it further suppresses misjudgment and ensures that the self-healing action is preferentially applied to the propagation source node and its dominant diffusion channel, thereby reducing the risk of repeated handling of non-root cause downstream nodes and repeated alarm triggering.

[0039] In this implementation scheme, by performing topological consistency mapping and descending order filtering on the surge intensity value of the call edge propagation, the key call edge with the most significant propagation trend can be quickly identified, and further, root cause candidate service nodes can be generated based on the multi-window continuous surge feature. At the same time, threshold traversal and hop count constraints are performed on the high-intensity propagation edge along the service dependency topology to form a set of key propagation paths. This enables accurate characterization of the abnormal propagation link and root cause location in complex distributed call relationships, providing a reliable basis for the targeted blocking and priority orchestration of subsequent self-healing actions, and improving the accuracy and feasibility of abnormal propagation identification.

[0040] Specifically, the process of constructing the reinforcement learning state input by integrating the runtime feature set, service dependency topology graph, and propagation status results, and constructing the self-healing action space, is as follows: Based on the self-healing runtime data and window metrics, service dependency topology graph, temporal alignment confidence value, call edge propagation surge intensity value set, root cause candidate service node set, and key propagation path set, a reinforcement learning state vector within the current sliding time window is constructed. The reinforcement learning state vector is composed of observable runtime feature metrics such as latency, error rate, and queuing load within the window, combined with the call edge connection relationship, temporal alignment confidence level, and propagation surge intensity distribution in the topology. It is used to comprehensively characterize the current abnormal propagation status and service dependency structure, enabling the policy network to simultaneously perceive the abnormal source, diffusion path, and propagation intensity in a single state input. The construction of the reinforcement learning state vector preferably employs a computable structured encoding method: for each call edge, the call edge P95 latency, call edge error rate, number of downstream waiting tasks, and propagation surge intensity value are used as edge feature components; for each service node, the number of times it participates in key propagation edges as a caller or callee, root cause candidate labels, and node in-degree and out-degree statistics are used as node feature components; further, the set of key propagation paths is mapped to a sequence of path edge labels on the topology graph, and these are added to the state representation in the form of 0-1 indicator variables, thereby enabling the set-class path information to be transformed into numerical features that can be directly input into the policy network. Furthermore, the structural information of the service dependency topology graph is preferably incorporated into the state vector through sparse encoding of the graph adjacency matrix. Specifically, topology nodes and call edges can be constructed into directed graph adjacency relationships, and the node features can be embedded in a low dimension using a graph convolutional network (GCN) to obtain a topology embedding vector representing the call dependency structure. This vector is then concatenated with the window running features to form the final reinforcement learning state input, thus ensuring that the topology and propagation situation information have a unified and achievable encoding entry point. A self-healing action space is defined, which includes executable self-healing maintenance actions for key call edges and root cause candidate service nodes. The self-healing action space is a set of controllable operations output by the policy network. All actions correspond to standard control instructions that can be directly implemented by existing microservice governance and operation and maintenance platforms, thereby ensuring the engineering feasibility of reinforcement learning decisions. Specifically, it includes: performing rate limiting control, circuit breaking isolation, and retry suppression actions on call edges on key propagation paths. Among them, rate limiting control is used to reduce the request rate of abnormal traffic continuing to spread downstream, circuit breaking isolation is used to temporarily cut off high-risk call edges when the error rate continues to rise, and retry suppression is used to avoid request storms and cascading amplification caused by automatic retries.The system performs instance scaling, service restart, and resource quota adjustment actions on root cause candidate service nodes. Instance scaling increases processing capacity by adding service replicas, service restart eliminates abnormal operational states such as deadlocks and memory leaks, and resource quota adjustment dynamically allocates CPU and memory resources to alleviate resource contention bottlenecks, thereby achieving rapid recovery and propagation blocking at the root cause node. Specifically, the resource quota adjustment action dynamically corrects the container resource limits corresponding to the root cause candidate service node. This is preferably performed through the resource configuration interface of the container orchestration platform. For example, in a Kubernetes environment, this is achieved by updating the `resources.requests` and `resources.limits` fields of the Deployment or Pod to adjust the number of CPU cores and memory quotas online, ensuring that the action has a clear engineering implementation entry point and can be directly invoked by the operations and maintenance execution layer. Additionally, the system performs enhanced sampling actions on call chain observations. Enhanced sampling actions are used to increase the link tracing sampling rate or expand the observation span coverage when the propagation situation is significant, to supplement the observation data of key propagation edges and improve the accuracy of subsequent state assessments and policy updates. Among them, the enhanced sampling action is preferably implemented through the sampling control interface of the distributed link tracing system. For example, the link sampling rate can be increased by issuing a command through the service mesh governance component, so that the call edge on the critical propagation path generates a higher density of Span observations in the subsequent window, thereby forming a practical observation enhancement entry point and avoiding propagation misjudgment caused by incomplete sampling.

[0041] In this implementation plan, by unifying window operation characteristics, service dependency topology, time sequence reliability, and propagation status results into a reinforcement learning state vector, the policy network can comprehensively perceive the source of anomalies and the propagation path. At the same time, an executable self-healing action space covering key call edge control and root cause node repair is defined to realize engineering-feasible maintenance operations such as rate limiting isolation, resource adjustment, and instance recovery. This provides clear state inputs and action constraints for deep reinforcement learning, and improves the pertinence, real-time performance, and stable implementation capability of self-healing decisions.

[0042] Specifically, the process of outputting the current self-healing action sequence and executing action orchestration through the policy network is as follows: Using the reinforcement learning state vector as input, the self-healing policy network model is trained using the Proximal Policy Optimization (PPO) algorithm, and the parameters of the self-healing policy network model are updated by maximizing the cumulative reward function. Here, PPO is a reinforcement learning training algorithm suitable for continuous decision-making scenarios. It ensures stable convergence of the training process by limiting the policy update amplitude. The cumulative reward function is used to quantify the comprehensive contribution of self-healing actions to reducing propagation risk and improving anomaly recovery, thereby driving the policy network to learn the optimal self-healing decision rule. The current reinforcement learning state vector is input into the self-healing policy network model, outputting the optimal self-healing action sequence. This optimal self-healing action sequence is a combination of actions generated by the policy network under the current window's anomaly propagation situation, including rate limiting and isolation actions for key propagation edges and resource recovery actions for root cause candidate nodes. The action sequence can be arranged in priority order to accommodate the coordinated execution of multiple actions. The optimal self-healing action sequence is then distributed to the execution layer for action orchestration and implementation. The execution layer determines the target of the actions based on the service dependency topology diagram. The execution layer is preferably integrated into the service governance and container orchestration platform. Actions are issued through the service governance platform API or container orchestration interface. Specifically, actions can be automatically issued and implemented by calling the rate limiting rule configuration interface, circuit breaker control interface, instance scaling interface, and resource quota adjustment interface. The service dependency topology diagram is used to locate the specific call edge and microservice node to which the action is applied, ensuring that self-healing actions can accurately act on the critical positions of the anomaly propagation link and avoid misoperation on irrelevant nodes. To further ensure the operational security of self-healing actions, an action security constraint layer can be optionally set up, adopting an approval and confirmation mechanism for high-risk actions such as circuit breaker isolation and service restart, improving the controllability of the production environment without affecting the automatic decision-making of the policy network. The determination of the target of the action is preferably based on an explicit mapping between the set of critical propagation paths and the set of root cause candidate service nodes: when the policy network outputs actions such as rate limiting, circuit breaking isolation, and retry suppression on the call edge, the execution layer preferably selects the first call edge with the largest propagation surge intensity value in the set of critical propagation paths as the first target, and applies the action downstream in sequence according to the topological order of the critical propagation paths, thereby prioritizing the blocking of the dominant diffusion channel of abnormal propagation; when the policy network outputs actions such as instance expansion, service restart, and resource quota adjustment, the execution layer preferably selects the node with the most significant propagation surge trend in the set of root cause candidate service nodes as the target, so as to achieve priority recovery of the abnormal source node.To avoid multiple actions being repeatedly applied to the same object within the same window or causing control conflicts, the execution layer preferably introduces action orchestration constraints: when multiple actions simultaneously point to the same call edge, the circuit breaker isolation action is executed first, and the rate limiting action is postponed to avoid repeated suppression causing complete service interruption; when multiple actions simultaneously point to the same root cause candidate node, the instance expansion action is executed first, and the service restart action is triggered only after the expansion fails to avoid state oscillation caused by frequent restarts; when actions simultaneously cover multiple call edges and multiple nodes, the affected objects are preferably sorted in descending order of propagation surge intensity value, and a maximum of a preset number of critical actions are limited to be executed within each window to ensure the controllability of self-healing and system stability. Through the above orchestration mechanism of "prioritizing the blocking of critical paths, prioritizing the recovery of root cause nodes, and resolving action conflicts," the execution layer can map the action sequence output by the policy network to a definite affected edge and a definite affected node on the topology, thereby ensuring that self-healing actions have clear implementation objects, execution order, and conflict handling rules, realizing a feasible self-healing approach for abnormal propagation chains.

[0043] In this implementation plan, PPO policy network training is driven by reinforcement learning state vectors to achieve automatic learning and stable optimization of self-healing actions under abnormal propagation conditions, and the optimal action sequence can be output in the current window. At the same time, the precise location and execution orchestration of the action objects are completed by relying on the service dependency topology diagram, so that maintenance operations such as rate limiting isolation, capacity expansion and restart can be directly implemented on the existing governance platform, thereby improving the real-time performance, pertinence and engineering feasibility of the distributed system's self-healing decision.

[0044] Specifically, the process of continuous optimization of the policy network is as follows: after the action is executed, continuously monitor the request response P95 latency, the call edge P95 latency, the call edge error rate, and the number of downstream waiting tasks, and update the propagation surge intensity value of the call edge. Among them, the request response P95 latency is used to characterize the overall tail response recovery of the service instance, the call edge P95 latency is used to characterize the high latency risk fallback of the critical call link, the call edge error rate is used to characterize the change in the frequency of abnormal call return, and the number of downstream waiting tasks is used to reflect the degree of relief of the queue backlog of the callee. All of the above indicators are real-time observable operational feedback quantities within the window. The propagation risk value is obtained by aggregating the propagation surge intensity values ​​of all call edges. The decrease in the propagation risk value is then calculated. Preferably, the propagation risk value is obtained by performing log-sum-exp aggregation on the propagation surge intensity values ​​of all call edges within the window. This involves summing the natural exponents of each propagation surge intensity value and taking the natural logarithm. This method highlights the dominant contribution of key high-risk propagation edges while maintaining the interpretability of the overall risk magnitude, thus forming a propagation risk indicator characterizing the global spread of anomalies in the service-dependent topology. The decrease in risk value is obtained by calculating the difference, which quantifies the effectiveness of self-healing actions in weakening propagation links and blocking propagation. This serves as the main decision feedback indicator for optimizing the self-healing strategy, reflecting whether the overall spread intensity of anomalies in the service-dependent topology has been effectively suppressed. The algorithm simultaneously combines the decline magnitudes of request-response P95 latency, call-side P95 latency, call-side error rate, and the number of downstream waiting tasks. The decline magnitude of each indicator is obtained by the difference between the window statistics before and after the action execution. When tail latency decreases, error rate decreases, and queuing load decreases, the corresponding reward signal increases. This serves as an auxiliary decision feedback indicator to characterize the comprehensive effect of self-healing actions on service performance recovery and load reduction while blocking propagation, thus avoiding strategies that only pursue reduced propagation intensity while ignoring service availability and tail performance. To ensure that multiple decline magnitudes are considered during reward fusion... To ensure scale consistency and comparability, it is preferable to perform uniform normalization processing on each decline amplitude component: Specifically, the decline amplitude of request response P95 latency is divided by its historical window median baseline value, the decline amplitude of call edge P95 latency is divided by the historical window median baseline value of the corresponding call edge, the decline amplitude of call edge error rate is divided by the historical window median baseline value, and the decline amplitude of downstream waiting task number is divided by the historical window median baseline value. This converts different dimensional indicators into a standardized degree of improvement relative to their own historical normal level, avoiding the incomparability of reward components due to differences in numerical scale.Furthermore, the normalized decline amplitude is preferably limited to a preset range using truncation constraints to suppress reward explosion caused by abnormal spikes, ensuring the stability and learnability of the reward signal. Based on this, the standardized improvement component and the decline amplitude of propagation risk together constitute the reward feedback, with the decline in propagation risk serving as the primary decision feedback component and the performance decline improvement serving as the auxiliary feedback component. This achieves synergistic optimization of risk suppression and performance recovery without introducing artificial empirical weights. This reward signal is fed back to the self-healing policy network model as a reinforcement learning reward signal, guiding the policy network to tend to select self-healing actions that can quickly restore performance and suppress propagation. This drives continuous optimization of model parameters. The reward signal serves as the training feedback input for the PPO algorithm, used to iteratively update the policy network parameters in a continuous self-healing closed loop. This allows the policy to gradually adapt to the optimal self-healing maintenance decisions under different anomaly types and topological evolution conditions, achieving online self-learning and continuous improvement of the self-healing policy.

[0045] like Figure 4 The diagram shows a closed-loop flowchart for distributed software self-healing maintenance based on deep reinforcement learning. The flowchart fully depicts the end-to-end closed-loop logic of the distributed software self-healing maintenance method based on deep reinforcement learning in this invention. First, self-healing operation data is collected in real-time at the microservice call chain level. Cross-node time drift, out-of-order anomaly spans, and monitoring noise are eliminated through unified clock alignment and data cleaning, forming stable operational statistical characteristics within a sliding time window. Then, based on the upstream and downstream span temporal relationships, the causal residuals of the call chain are calculated, and observation consistency is evaluated to generate call chain temporal reliability. Valid samples are then screened using a threshold judgment to prevent untrusted call chains from entering the propagation modeling process. Based on reliable samples, the service dependency topology is dynamically updated, and the surge intensity of call edge propagation is further calculated. Top-K key propagation edges, root cause candidate nodes, and key propagation path sets are identified, thereby characterizing the diffusion trend of anomalies in the topology. Finally, the operational characteristics, propagation status, and topology are jointly constructed as the state input for reinforcement learning. The policy network outputs the optimal self-healing action sequence and completes the action orchestration and execution. After the action is implemented, the propagation risk reduction and indicator recovery effect are continuously monitored, the reward signal is calculated, and the policy parameters are updated through the PPO algorithm to achieve continuous closed-loop iteration of abnormal propagation blocking and self-healing decision optimization.

[0046] In this implementation scheme, after the self-healing action is executed, operational feedback such as tail latency, error rate, and queuing load are continuously monitored. A global propagation and diffusion risk index is formed based on the aggregation of the surge intensity of call edge propagation, which enables the main decision evaluation of the anomaly propagation suppression effect. At the same time, the performance drop-off magnitude is combined as auxiliary feedback to take into account both propagation blocking and service recovery quality. This constructs a PPO policy network driven by a consistent reinforcement learning reward signal to continuously iterate and optimize the self-healing strategy, enabling the self-healing strategy to adapt to topology evolution and anomaly type changes online, thereby improving the stability, pertinence, and implementable closed-loop optimization capability of the distributed system's self-healing maintenance.

[0047] Reference Figure 2 As shown, the second aspect of this invention provides a distributed software self-healing maintenance system based on deep reinforcement learning, applied to the aforementioned distributed software self-healing maintenance method based on deep reinforcement learning. The system includes: a self-healing operation data acquisition and processing module, used to collect self-healing operation data of microservice instances and call chains in real time, and perform data preprocessing on the self-healing operation data to form an operation feature set; a call chain temporal reliability assessment module, used to perform reliability assessment on the consistency of call chain observations based on the upstream and downstream span temporal relationships of cross-service call chains, filter valid call chain samples, and dynamically update the service dependency topology diagram; an anomaly propagation chain situation modeling module, used to combine the service dependency topology diagram and the operation features of the window self-healing operation data to perform strength assessment on the propagation situation of call edges, identify key propagation edges and root cause candidate nodes, and generate a set of key propagation paths; and a reinforcement learning self-healing decision optimization module, used to construct the operation feature set, service dependency topology diagram, and propagation situation results as reinforcement learning state inputs, construct a self-healing action space, output the current self-healing action sequence through a policy network and execute action orchestration, and continuously optimize the policy network.

[0048] In this implementation scheme, reliable operational feature inputs are constructed by real-time collection and robust preprocessing of microservice call chain operation data. Effective call chain samples and dynamic topology updates are then screened based on Span temporal consistency assessment, providing reliable support for distributed observation data. Furthermore, by combining anomaly propagation situation strength modeling, key propagation edges, root cause candidate nodes, and propagation paths are identified, providing structured state awareness and precise target objects for deep reinforcement learning. Finally, a self-healing action sequence that can be implemented is output through the PPO policy network and continuously optimized under closed-loop reward feedback, enabling the system to achieve stable, targeted, and implementable automated self-healing maintenance decisions in complex topology evolution and cascading fault propagation scenarios.

[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0050] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. As those skilled in the art will understand, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A distributed software self-healing maintenance method based on deep reinforcement learning, characterized in that, Includes the following steps: S1 collects self-healing operation data of microservice instances and call chains in real time, and performs data preprocessing on the self-healing operation data to form an operation feature set; S2, based on the upstream and downstream span timing relationship of cross-service call chains, performs a reliable assessment of the consistency of call chain observations, filters valid call chain samples, and dynamically updates the service dependency topology diagram. S3 combines the operational characteristics of the service dependency topology diagram and the window self-healing operation data to perform an intensity assessment of the propagation status of the call edge, identify key propagation edges and root cause candidate nodes, and generate a set of key propagation paths. S4 constructs the running feature set, service dependency topology graph and propagation situation results as reinforcement learning state input, constructs a self-healing action space, outputs the current self-healing action sequence through the policy network and performs action orchestration, and continuously optimizes the policy network.

2. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of collecting self-healing operation data of microservice instances and call chains in real time, and performing data preprocessing on the self-healing operation data to form an operation feature set is as follows: For all microservice instances and call chains, self-healing operation data is collected in real time. The self-healing operation data includes: call edge identifier, service dependency relationship, request response latency, request return status code, request timeout flag, number of downstream waiting tasks, Span start timestamp, Span end timestamp, call edge latency, and Span exception flag. The self-healing operation data undergoes unified clock reference alignment processing to identify and mark abnormal Span records with causal violations. Based on the request return status code, request timeout flag, and Span anomaly flag, consistency checks are performed on abnormal request samples and abnormal call samples to remove dirty data records. For downstream waiting task counts, request response latency, and call edge latency, neighborhood interpolation is used to fill in the gaps, and sliding robust statistics are employed to suppress abnormal spikes and monitoring noise. Based on a sliding time window, statistical window metrics are calculated, including: the 95th percentile position of the sorted request response latency and call edge latency sample sets within the window. The corresponding latency values ​​are used to obtain the request-response P95 latency and the calling edge P95 latency; the number of requests within the window is divided by the window duration to obtain the request arrival rate; the proportion of requests that time out within the window to the total number of requests within the window is used to obtain the request timeout rate; the proportion of requests that return abnormal status codes within the window to the total number of requests within the window is used to obtain the request error rate; the proportion of abnormal Spans on the calling edge within the window to the total number of Spans on the calling edge is used to obtain the calling edge error rate; normalization processing is performed on the self-healing operation data and window metrics; a self-healing maintenance database is established to store the original and preprocessed self-healing operation data and window metrics.

3. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process for reliably assessing the consistency of call chain observations based on the upstream and downstream Span timing relationships across service call chains is as follows: During cross-service calls, when the calling microservice instance initiates a remote request to the called microservice instance, the call span generated by the calling party is marked as the upstream call span, and the call span generated by the called party is marked as the downstream call span; the difference between the start timestamp of the downstream call span and the end timestamp of the upstream call span is calculated to obtain the causal residual of the call chain. Based on the sliding time window, the median absolute deviation of the causal residuals is calculated as the causal residual fluctuation intensity value; at the same time, the proportion of the number of abnormal spans that violate causality within the window to the total number of spans within the window is calculated to obtain the causal violation rate; the median of the historical causal residual fluctuation intensity value and the causal violation rate are respectively taken to obtain the residual fluctuation benchmark term and the violation rate benchmark term. The residual stability decay term is obtained by dividing the causal residual fluctuation intensity value by the sum of the residual fluctuation benchmark term and the smallest positive number, and then performing a natural exponential operation on the negative value. The causal consistency decay term is obtained by dividing the causal violation rate by the sum of the violation rate benchmark term and the smallest positive number, and then performing a natural exponential operation on the negative value. The causal stability decay term and the causal consistency decay term are multiplied together to obtain the time-aligned reliability value of the cross-service call chain.

4. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of filtering valid call chain samples and dynamically updating the service dependency topology diagram is as follows: The time-aligned confidence value is compared with the confidence threshold: when the time-aligned confidence value is greater than or equal to the confidence threshold, the time-series observation of the call chain within the current window is determined to be reliable, and a valid label is generated for the corresponding self-healing running data and window metrics. When the time alignment confidence value is less than the confidence threshold, the timing observation of the call chain within the current window is determined to be unreliable, and unreliable removal processing is performed on the self-healing running data and window metrics of the corresponding call chain within the window. Based on the call edge identifier, the caller microservice instance and the callee microservice instance in the statistical window are used as topology nodes, and the cross-service call records that actually occur in the window are used as directed call edges between nodes to construct the service call relationship graph in the current window. The newly added call edges and the failed call edges are updated in a consistent manner in combination with the service dependency relationship to obtain the service dependency topology graph that changes with the time window.

5. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process for assessing the strength of the call edge propagation situation by combining the operational characteristics of the service dependency topology diagram and window self-healing operation data is as follows: For each call edge in the service dependency topology graph, read the valid call edge P95 latency, call edge error rate, and the number of downstream waiting tasks corresponding to the callee. Calculate the difference between adjacent sampling times and divide it by the sampling time interval to obtain the edge latency growth rate, edge error growth rate, and queuing growth rate, which are the three growth rate components. Calculate the median absolute deviation of the three growth rate components in the corresponding historical sample set to obtain the corresponding robust fluctuation scale. Take the median of the corresponding historical growth rate components to obtain the corresponding historical center benchmark. For each of the three growth rate components, subtract the historical center benchmark from its own value and divide by the sum of the robust fluctuation scale and the minimum positive number to obtain the corresponding robust standardized value. Perform natural exponential operations on the robust standardized values ​​corresponding to the three growth rate components and sum them. Take the natural logarithm of the summation result to obtain the propagation surge intensity value.

6. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of identifying key propagation edges and root cause candidate nodes, and generating a set of key propagation paths is as follows: Based on the propagation surge intensity values ​​of all call edges within the current sliding time window, a set of call edge propagation intensity values ​​is constructed, and a consistent mapping of the propagation surge intensity values ​​is performed based on the service dependency topology graph, so that each call edge corresponds to a unique propagation surge intensity label; the propagation surge intensity values ​​within the window are sorted in descending order, and the top K call edges are selected to construct a set of key call edges; In the set of key call edges, if the robust normalized value of at least one growth rate component of the same caller node continuously exceeds the growth threshold within the most recent L windows, the caller node is determined to be a root cause candidate service node, and a set of root cause candidate service nodes is generated. Starting from the root cause candidate service node, traverse the call edges along the service dependency topology graph and determine the threshold value of the propagation surge intensity of each call edge. When the propagation surge intensity value of a call edge is greater than or equal to the intensity threshold, the call edge is marked as a valid propagation edge, and the propagation link continues to be extended along the downstream nodes of the call edge. When all adjacent call edges in the call edge sequence formed from the root cause candidate node are valid propagation edges, and the sequence length is not less than the minimum propagation hop count threshold, the call edge sequence is determined as a critical propagation path, and the sequence is summarized to form a set of critical propagation paths.

7. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of constructing the running feature set, service dependency topology graph, and propagation situation results as reinforcement learning state inputs and constructing the self-healing action space is as follows: Based on self-healing operation data and window metrics, service dependency topology graph, temporal alignment reliability value, call edge propagation surge intensity value set, root cause candidate service node set, and key propagation path set, construct the reinforcement learning state vector within the current sliding time window; Define a self-healing action space, which includes executable self-healing maintenance actions for critical call edges and root cause candidate service nodes. Specifically, it includes: performing rate limiting, circuit breaking isolation, and retry suppression actions on call edges on critical propagation paths. Perform instance expansion, service restart, and resource quota adjustment actions on root cause candidate service nodes; and perform enhanced sampling actions on call chain observations.

8. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of outputting the current self-healing action sequence through the policy network and executing action orchestration is as follows: Using the reinforcement learning state vector as input, a self-healing policy network model is trained using the Proximal Policy Optimization (PPO) algorithm, and the parameters of the self-healing policy network model are updated by maximizing the cumulative reward function. The current reinforcement learning state vector is input into the self-healing policy network model, and the optimal self-healing action sequence is output. The optimal self-healing action sequence is then sent to the execution layer for action orchestration and implementation. The execution layer determines the target of the action based on the service dependency topology diagram.

9. The distributed software self-healing maintenance method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of continuously optimizing the policy network is as follows: After the action is executed, the request response P95 latency, the call edge P95 latency, the call edge error rate, and the number of downstream waiting tasks are continuously monitored. The propagation surge intensity value of the call edge is updated. The propagation and diffusion risk value is obtained by aggregating the propagation surge intensity values ​​of all call edges. The decrease rate of the propagation and diffusion risk value is calculated. The decrease rate of the request response P95 latency, the call edge P95 latency, the call edge error rate, and the number of downstream waiting tasks are combined with the decrease rate as reinforcement learning reward signals and fed back to the self-healing strategy network model to drive the continuous optimization of model parameters.

10. A distributed software self-healing maintenance system based on deep reinforcement learning, characterized in that, include: The self-healing runtime data acquisition and processing module is used to collect self-healing runtime data of microservice instances and call chains in real time, and to perform data preprocessing on the self-healing runtime data to form a runtime feature set. The call chain timing reliability assessment module is used to perform reliability assessment on the observation consistency of the call chain based on the upstream and downstream Span timing relationship of the cross-service call chain, filter valid call chain samples, and dynamically update the service dependency topology diagram. The anomaly propagation chain situation modeling module is used to combine the operational characteristics of the service dependency topology diagram and window self-healing operation data to perform strength assessment of the propagation situation of the call edge, identify key propagation edges and root cause candidate nodes, and generate a set of key propagation paths. The reinforcement learning self-healing decision optimization module is used to construct the reinforcement learning state input from the running feature set, service dependency topology graph and propagation situation results, construct the self-healing action space, output the current self-healing action sequence through the policy network and execute action orchestration, and continuously optimize the policy network.

Images