Fault processing method and device for distributed micro-service system, equipment and medium

Abstract

The invention discloses a fault processing method and device for a distributed micro-service system, equipment and a medium, relates to the technical field of communication, and aims to solve the problems of low real-time performance and poor precision of fault processing. Transient fault data is injected into the target micro-service group through a chaos injection unit, real-time abnormal data acquisition is performed through a non-intrusive probe, and a transient snapshot is generated; according to the transient snapshots, single-service micro-explosion segments are obtained, the single-service micro-explosion segments are aggregated based on the cross-service overlapping degree, a transient family is generated, index slopes, resource contention traces, log emergence frequencies and topological slices in the transient family are coded into multi-view time sequence vectors, the multi-view time sequence vectors are written into an index database, and the multi-view time sequence vectors are sent to a meta reinforcement learning training queue; in an isolation environment, according to the meta reinforcement learning training queue, performing meta reinforcement learning model training on the high-frequency abrupt change fault scene, and determining a meta strategy according to an instant reward function; and adjusting a chaos injection unit according to the meta-strategy, and carrying out fault processing operation on the distributed micro-service system.

Description

Fault handling methods, devices, equipment and media for distributed microservice systems Technical Field

[0001] This application belongs to the field of communication technology, and specifically relates to a fault handling method, apparatus, device and medium for a distributed microservice system. Background Technology

[0002] Existing fault handling methods for distributed microservice systems include static chaotic injection, fault detection, and manual tuning. However, static chaotic injection (i.e., fault injection) is weakly coupled with real-time business conditions, and transient vulnerabilities are not fully exposed. Furthermore, fault detection mainly uses a single reinforcement learning or static threshold model, which requires periodic offline training to cope with workload distribution drift. These methods cannot meet the adaptive adjustment requirements of distributed microservice systems for scaling up and down, path changes, etc., resulting in low real-time performance and poor accuracy in fault handling. Summary of the Invention

[0003] This application provides a fault handling method, apparatus, device, and medium for a distributed microservice system, addressing the issues of low real-time performance and poor accuracy in fault handling.

[0004] In a first aspect, embodiments of this application provide a fault handling method for a distributed microservice system, including:

[0005] Transient fault data based on micro-explosive scenarios is injected into the target microservice group through a chaos injection unit, and real-time abnormal data is collected through non-intrusive probes to generate transient snapshots containing metrics, logs and topology information.

[0006] Based on the transient snapshot, single-service micro-explosion segments are obtained, and single-service micro-explosion segments are aggregated based on cross-service overlap to generate transient families. The indices slope, resource contention traces, log burst frequency, and topology slices within the transient families are encoded into multi-view time-series vectors, written into the index library, and pushed to the meta-reinforcement learning training queue.

[0007] In an isolated environment, a meta-reinforcement learning model is trained for high-frequency sudden fault scenarios based on the meta-reinforcement learning training queue, and a meta-policy is determined based on the immediate reward function.

[0008] The chaos injection unit is adjusted according to the meta-strategy to perform fault handling operations on the distributed microservice system.

[0009] Optionally, the target microservice group is obtained based on the gray-scale release period, automatic scaling records, and business peak curves.

[0010] Optionally, the timestamps of the target microservice group are aligned to the nanosecond level using a shared logical clock.

[0011] Optionally, the method further includes:

[0012] If a call topology hole is determined to exist based on the multi-view time-series vector, the chaotic injection unit is instructed to expand the uncovered blind zone and update the transient fault data.

[0013] Optionally, the instant reward function is determined based on the recovery time, false alarm rate, and resource overhead.

[0014] Optionally, the method further includes:

[0015] In an online environment, log-metric dual-channel features are constructed on the real-time monitoring stream according to a pre-set analysis window to obtain an anomaly impact range score. Based on the anomaly impact range score, the low-confidence analysis window is sent to the meta-reinforcement learning training queue.

[0016] Optionally, the method further includes:

[0017] The anomaly impact range score is parsed into a root cause confidence matrix. Combined with the call chain depth, high-frequency fault nodes and blank coverage nodes are identified. An exploration bias vector is generated based on the high-frequency fault nodes and blank coverage nodes. The meta-reinforcement learning model and the chaotic injection unit are adjusted based on the exploration bias vector.

[0018] Optionally, the method further includes:

[0019] Based on the blind zone weight function, the chaotic injection unit performs directional transient injection on the blank coverage node and improves the sampling resolution. It detects index intervals that continuously cross the historical percentile threshold online and freezes all data. It uses the injected meta-information to generate an overlapping cluster label set and optimizes the meta-strategy based on the overlapping cluster label set.

[0020] Secondly, embodiments of this application also provide a fault handling apparatus for a distributed microservice system, comprising:

[0021] The first module is used to inject transient fault data based on micro-explosion scenarios into the target microservice group through the chaos injection unit, and to collect real-time abnormal data through non-intrusive probes to generate transient snapshots containing metrics, logs and topology information.

[0022] The second module is used to obtain single-service micro-explosion segments based on the transient snapshot, aggregate the single-service micro-explosion segments based on cross-service overlap, generate transient families, encode the indicator slope, resource contention traces, log burst frequency and topology slices in the transient families into multi-view time series vectors, write them into the index library, and push them to the meta-reinforcement learning training queue.

[0023] The third module is used to train a meta-reinforcement learning model for high-frequency mutation fault scenarios in an isolated environment based on the meta-reinforcement learning training queue, and to determine the meta-policy based on the immediate reward function.

[0024] The fourth module is used to adjust the chaos injection unit according to the meta-strategy to perform fault handling operations on the distributed microservice system.

[0025] Optionally, the target microservice group is obtained based on the gray-scale release period, automatic scaling records, and business peak curves.

[0026] Optionally, the timestamps of the target microservice group are aligned to the nanosecond level using a shared logical clock.

[0027] Optionally, the first module is further configured to, when it is determined from the multi-view time-series vector that a call topology hole exists, instruct the chaotic injection unit to expand the uncovered blind zone and update the transient fault data.

[0028] Optionally, the instant reward function is determined based on the recovery time, false alarm rate, and resource overhead.

[0029] Optionally, the third module is further configured to construct log-indicator dual-channel features of the real-time monitoring stream in an online environment according to a pre-set analysis window, obtain an anomaly impact range score, and send the low-confidence analysis window to the meta-reinforcement learning training queue according to the anomaly impact range score.

[0030] Optionally, the third module is further configured to parse the anomaly impact range score into a root cause confidence matrix, combine it with the call chain depth to determine high-frequency fault nodes and blank coverage nodes, generate an exploration bias vector based on the high-frequency fault nodes and blank coverage nodes, and adjust the meta-reinforcement learning model and the chaotic injection unit based on the exploration bias vector.

[0031] Optionally, the third module is further configured to perform directional transient injection and improve sampling resolution at the blank coverage node through the chaotic injection unit according to the blind zone weight function, detect index intervals that continuously cross historical percentile thresholds online and freeze all data, generate an overlapping cluster label set using injected meta-information, and optimize the meta-strategy based on the overlapping cluster label set.

[0032] Thirdly, embodiments of this application also provide a communication device, including: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the fault handling method of the distributed microservice system described above.

[0033] Fourthly, embodiments of this application also provide a readable storage medium on which a program is stored, and when the program is executed by a processor, it implements the steps in the fault handling method of the distributed microservice system described above.

[0034] Fifthly, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the fault handling method of the distributed microservice system described above.

[0035] In this embodiment, the chaotic injection unit's operation of injecting transient fault data based on micro-explosive scenarios into the target microservice group can be adjusted according to a meta-policy. This meta-policy is determined during the training of the meta-reinforcement learning model based on an immediate reward function, ensuring close coupling between fault injection and real-time business dynamics, fully exposing potential transient faults, and thus guaranteeing the accuracy of subsequent fault handling. Since the immediate reward function can change in real time according to actual conditions, the technical solution provided in this embodiment does not require offline training, thereby ensuring the real-time nature of fault handling. Furthermore, this embodiment uses a meta-reinforcement learning model for learning, avoiding the problem of poor fault handling adaptability in scenarios such as scaling up and down and path changes in distributed microservice systems caused by using a single reinforcement learning or static threshold model. Attached Figure Description

[0036] Figure 1 is a flowchart of one of the fault handling methods for a distributed microservice system provided in an embodiment of this application;

[0037] Figure 2 is a second flowchart of a fault handling method for a distributed microservice system provided in an embodiment of this application.

[0038] Figure 3 is a flowchart of the third method for handling faults in a distributed microservice system provided in the embodiments of this application.

[0039] Figure 4 is a flowchart of the fault handling method of the distributed microservice system provided in the embodiment of this application;

[0040] Figure 5 is the fifth flowchart of the fault handling method for a distributed microservice system provided in the embodiments of this application;

[0041] Figure 6 is a structural diagram of the fault handling device for a distributed microservice system provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0044] It is worth noting that the technologies described in this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in this application are often used interchangeably, and the described technologies can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies. However, the following description describes New Radio (NR) systems for illustrative purposes, and NR terminology is used in most of the following description. These technologies can also be applied to applications beyond NR systems, such as 6th generation (6G) radio systems. th Generation 6G communication system.

[0045] Referring to Figure 1, which is a flowchart of a fault handling method for a distributed microservice system provided in an embodiment of this application, the method includes the following steps:

[0046] Step 101: Inject transient fault data based on micro-explosion scenarios into the target microservice group through the chaos injection unit, and collect real-time abnormal data through non-intrusive probes to generate transient snapshots containing metrics, logs and topology information.

[0047] In this embodiment, a target microservice group may include multiple microservices, and each microservice may include multiple container units (pods); transient fault data may include CPU preemption pulses and random service mesh latency, etc.; non-intrusive probes may include Extended Berkeley PacketFilter-Kernel Probe (eBPF-kprobe) probes and sidecar-trace probes.

[0048] In this embodiment, the target microservice group is obtained based on the gray-scale release period, automatic scaling records, and business peak curve, enabling the technical solution provided in this embodiment to achieve dynamic point selection.

[0049] In this embodiment, the timestamps of the target microservice group are aligned at the nanosecond level using a shared logical clock. This ensures that cross-core and cross-container event sampling is based on a unified time base, thereby obtaining finer-grained fluctuation profiles and dependency chain characteristics within the same sampling window.

[0050] Compared to the observation errors caused by existing technologies that use fixed injection templates and millisecond-level Network Time Protocol (NTP) alignment, this method exhibits higher consistency and reproducibility in terms of data acquisition accuracy and perturbation controllability, providing an accurate and continuous transient feature stream for subsequent algorithms.

[0051] In this embodiment, step 101 can be implemented through the following steps:

[0052] S11. Based on the grayscale release period Automatic scaling records and business peak curve Calculate the injection priority for each microservice group. , and select The largest microservice group is designated as the target microservice group.

[0053] In this embodiment, It can be obtained through the following formula (1):

[0054] ; (1)

[0055] in, To increase traffic; Dependency depth; To expand and shrink capacity activity; This represents the number of instantaneous requests during peak business hours. Baseline request count; The depth of the call chain; For the kth instance in The number of times the volume expands or contracts within the container; For the number of instances; The non-zero weights obtained from offline calibration.

[0056] Step S11 ensures that the target microservice group not only handles sudden traffic surges but is also located at the deepest dependency chain, thus maximizing the probability of exposing transient risks.

[0057] S12. Generate transient fault data for the target microservice group based on micro-explosion scenarios. And write it to the memory buffer.

[0058] In this embodiment, transient fault data may include:

[0059] A CPU peak preemption pulse of 300ms-500ms, which can be injected at the container layer;

[0060] A random delay jitter sequence of 5ms-10ms can be synchronously superimposed on the service mesh channel.

[0061] In this embodiment, S12 can Each perturbation field and Perform a normalized mapping to keep the perturbation granularity consistent with the natural distribution of jitter in the production environment.

[0062] The transient fault data provided in this embodiment amplifies microscopic fluctuations through instantaneous multidimensional stimulation, making it easier for subsequent non-invasive probes to capture spike features.

[0063] S13, Invoke the chaos injection unit, and Serialization is written to a Custom Resource Definition (CRD) object in the Kubernetes container orchestration engine, denoted as: Based on the principle of scheduling by nearest node, the fault injection Pod is scheduled to a node on the same host machine as the core business request, so as to minimize the delay of fault activation. The chaos injection unit can be a chaos controller.

[0064] In this embodiment, the principle of scheduling by the nearest node can avoid cross-node network overhead and ensure that injected disturbances and service traffic are executed concurrently within the same Non-Uniform Memory Access (NUMA) domain.

[0065] S14. When the injection command is sent to the host machine (such as a microservice pod), the eBPF-kprobe probe and sidecar-trace probe are automatically launched.

[0066] In this embodiment, eBPF-kprobe is used to capture system call latency vectors. The sidecar-trace probe is used to record the boundary set of cross-service call nodes (spans). ,form The system provides dual-channel raw data streams. The trigger thresholds for both types of probes are anchored to the rising edge of the CPU preemption pulse, ensuring millisecond-level positioning accuracy for call chain spike events.

[0067] S15. Mount a logical clock stamp sequence to the host machine's shared memory. It also requires that the clock sequence be strictly monotonically increasing. Distributed clock skew was detected. At that time, a uniform time shift is performed on all timestamps using the following formula (2) to eliminate errors caused by cross-node clock drift. This process ensures that the time alignment accuracy of subsequent event slices is less than 1.

[0068] (2)

[0069] in, Let i be the i-th timestamp. In this embodiment, the timestamp may include, but is not limited to, log time, metric sampling time, and Span time.

[0070] S16. When any probe detects a sudden spike in an indicator or a doubling of the log abnormal word occurrence rate. At that time, extract the transient snapshot interval. The transient snapshot is constructed from all event vectors within the data using the following formula (3). :

[0071] ; (3)

[0072] in, For the p-th non-invasive probe at time The collected event vector; P is the number of non-invasive probe paths; is the number of sampling points for the p-th non-invasive probe; j is the sampling point number; Transient snapshot interval Indicator functions; This is to invoke the adjacency matrix of topological fragments.

[0073] The transient snapshot constructed using S16 not only retains all the original metrics within the time window but also embeds service call relationships, providing a unified feature base for meta-reinforcement learning.

[0074] Step 102: Based on the transient snapshot, obtain the single-service micro-explosion segment, aggregate the single-service micro-explosion segment based on the cross-service overlap, generate a transient family, encode the indicator slope, resource contention traces, log burst frequency and topology slice within the transient family into a multi-view time series vector, write it into the index library, and push it to the meta-reinforcement learning training queue.

[0075] In this embodiment, step 102 can be implemented through the following steps:

[0076] S21. Perform a two-level sort on all transient snapshots, including:

[0077] First, based on the clock stamp sequence Complete the first-level sorting of transient snapshots;

[0078] Secondly, based on the call chain ( Complete the two-level sorting of the transient snapshot.

[0079] After the secondary sorting is completed, S21 writes the sorting results into the monotonically increasing timeline. .

[0080] S21 ensures consistent event views among distributed nodes, providing a unified time base for window computation.

[0081] S22. Extract dynamic baselines and mark peak initiation points on single service sequences, and analyze the timeline. Each microservice Maintenance time sliding window The baseline amplitude threshold is calculated using the following formula (4). :

[0082] ; (4)

[0083] in, For microservices at any time The collected multidimensional indicator vector may include, but is not limited to, latency and error rate. for The mean vector; As a magnification factor, in this embodiment, For example, in this embodiment, The length can be 3 seconds.

[0084] When the latest transient snapshot amplitude In step S22, time t can be marked as the peak starting point of the highly dynamic distributed microservice system.

[0085] S23. Generate a single-service micro-burst segment centered on the peak origin, starting from the peak origin. Adaptive expansion forward and backward, if adjacent peak intervals Then it will be merged in sequentially, and the window boundaries will be updated synchronously. Ultimately forming microservice-level continuous segments It should be noted that, This is just a specific example. In actual use, other values ​​can also be set, which will not be elaborated here.

[0086] S23 can compress highly discrete spike sequences into "micro-explosive segments" that can be continuously tracked.

[0087] S24. Merge transient families based on cross-service overlap, and traverse all instances within the same time slice. The following formula (5) is used to calculate any two segments. and Cross-service overlap :

[0088] ; (5)

[0089] in, Indicates time overlap; Indicates topological / spatial overlap; This indicates request / resource overlap; and A set of time intervals; and For the request token set; and For the set of return codes; , , These are the weighting coefficients. .

[0090] In this embodiment, S24 can be At that time, the corresponding micro-explosion segments will be merged across services into transient families. It inherits the parent-child relationship of the call chain to achieve implicit dependency aggregation.

[0091] S25, For transient families Continuous tracking and dynamic extension for each Set the initial tracking window, such as 1 second. If a new micro-explosion occurs during the tracking period... And with overlap Then extend in real time The end time is until all fluctuations return to zero, ensuring the integrity of the cluster during high-frequency switching.

[0092] S26, Transient Family The four information streams—internal indicator curve slope sequence, resource contention traces, log keyword burst frequency, and call topology slice—are concatenated chronologically and encoded into a sequence-preserving multi-view vector. In this embodiment, By fusing numerical, textual, and graph structural features within the same time coordinate system, a unified input is provided for meta-reinforcement learning strategies.

[0093] S27, will Write the data to the index and push it to the Meta-Reinforcement Learning (Meta-RL) training queue.

[0094] Optionally, this embodiment may also include a step of instructing the chaotic injection unit to expand the uncovered blind zone and update the transient fault data when it is determined from the multi-view time-series vector that a call topology hole exists. This step completes the closed-loop operation from data acquisition to event aggregation to injection adjustment.

[0095] Step 103: In an isolated environment, train a meta-reinforcement learning model for high-frequency mutation fault scenarios based on the meta-reinforcement learning training queue, and determine the meta-policy based on the immediate reward function.

[0096] In this embodiment, step 103 may include the following steps:

[0097] S31. Construct a reproducible fault seed library, including:

[0098] Reading transient event sets Where N is the total number of events; Let i be the i-th transient event.

[0099] Based on the exception type and link depth The fault seed library is obtained by performing a two-level hierarchical structure using the following formula (6). :

[0100] ; (6)

[0101] in, A subset of events of exception type c and link depth d; The cardinality of the corresponding subset; The sampling probability is obtained based on inverse cardinality normalization and is used for balanced extraction in subsequent chaotic injection.

[0102] S32. Use online microservice mirrors of the same version. With real traffic replay Building the basic environment in an isolated cluster The link proxy component sends back flow metrics, logs, and tracking data to the meta-reinforcement learning agent in real time, maintaining data distribution consistency with production.

[0103] S33, Inject a single point of failure and start the first phase of learning. Mounting Chaos Injection Unit ,according to The indexes are sequentially injected with three types of single-point faults: network jitter, dependency timeout, and resource contention. The duration of each fault type does not exceed a first threshold, which can be 500ms, for example. Meta-reinforcement learning agent Collect the state-action-reward sequence in a pure single-point scenario and execute the first phase of policy update.

[0104] S34, superimposed dual faults and shortened injection interval, when In single-point scenarios, the policy loss is below the threshold. At that time, through The system automatically adds a second type of fault and compresses the injection interval to a second threshold, which, for example, can be 300ms. By re-estimating state values ​​and rapidly readjusting network parameters in overlapping anomalies, cross-scenario migration can be achieved.

[0105] S35. Entering the high-frequency mutation stage, after the dual fault learning converges, through... The system randomly switches between three types of fault combinations and further compresses the switching interval to a third threshold. In this embodiment, the third threshold is on the order of seconds, thus constructing a high-frequency sudden fault scenario. . The policy network needs to converge rapidly during the restart cycle to ensure that the positioning latency remains within a certain range. .

[0106] Optionally, in this embodiment, S35 can model active microservice instances within the same time slice as an Agent set in a high-frequency sudden failure scenario. (Dynamically changes with expansion and contraction of volume), among which, This represents the number of active instances. Each Agent is represented by a multi-view local slice. Location embedding With short-period implicit representation Composition of local observation A centralized training / decentralized execution paradigm is adopted to share the global state. Topology slicing, resource consumption, link depth, exploration bias and with a variable-length mask Handle instance creation, deletion, and inactivation.

[0107] The joint value is expressed as shown in the following formula (7), which is a value decomposition network (VDN) / monotonic value function factorisation (QMIX) gated hybrid that maintains monotonicity:

[0108] ; (7)

[0109] in, For the linear part of VDN; This refers to the non-linear component of QMIX; For global value function; This is the global state; For joint action; For gated functions; The independent decision-making value of a single microservice; For monitoring metrics of a single microservice; For this super network by Generate non-negative weights to ensure , Adaptive selection of weights.

[0110] To align parent-child dependencies and suppress ineffective collaborations, a joint loss of topological consistency and prior constraints is introduced as shown in Equation (8):

[0111] (8)

[0112] in, Indicates TD error; Indicates topological regularization; This indicates prior constraints on the strategy; For instant rewards; Discount factor; The next state; For the next action; 'a' represents the current state; 'a' represents the current action. Regularization of graph structure; For each microservice's embedding vector; Let be the adjacency matrix at time t; These are topological regularization weights; Let KL divergence be a metric. The current strategy for service i; Expert-based prior strategies; This is essential for strategic regularization.

[0113] During training, differential rewards and priority replay of overlapping labels are combined to improve credit allocation and convergence speed under overlapping anomalies; during execution, each agent is based solely on... With local strategy Low-latency decision-making, action coverage includes parameterized subspaces such as self-healing, flow limiting, and threshold parameter tuning; when the confidence of the analysis window is low, the corresponding joint trajectory is weighted and replayed and the target network is refreshed, maintaining a consistent rhythm with the exploration and rearrangement, thus enhancing the efficiency of team collaborative positioning.

[0114] S36. Continuously monitor recovery time during training. False alarm rate and the increase in resource consumption Discount by time using the following formula (9) Calculate instant rewards :

[0115] ; (9)

[0116] in, Positive weights and satisfying ; Controlling the exponential decay of recovery time; For monitoring system resources; The penalty coefficients for each resource are denoted by . The reward function incentivizes the policy to minimize recovery time and suppress additional resource consumption while ensuring a controllable false alarm rate.

[0117] S37. After each round of multi-stage training, the meta-policy will be... With fast adaptive weights Write to the policy repository and include the set of exception patterns with insufficient coverage. The message is written back to step 102 via the message queue. This allows step 102 to be prioritized for marking during the next round of event reconstruction, thereby driving the chaos injection unit to expand and fill the blind zone, forming a continuous self-evolving closed loop of seed expansion - policy update - blind zone feedback.

[0118] Step 104: Adjust the chaos injection unit according to the meta-strategy to perform fault handling operations on the distributed microservice system.

[0119] In this embodiment, the chaotic injection unit's operation of injecting transient fault data based on micro-explosive scenarios into the target microservice group can be adjusted according to a meta-policy. This meta-policy is determined during the training of the meta-reinforcement learning model based on an immediate reward function, ensuring close coupling between fault injection and real-time business dynamics, fully exposing potential transient faults, and thus guaranteeing the accuracy of subsequent fault handling. Since the immediate reward function can change in real time according to actual conditions, the technical solution provided in this embodiment does not require offline training, thereby ensuring the real-time nature of fault handling. Furthermore, this embodiment uses a meta-reinforcement learning model for learning, avoiding the problem of poor fault handling adaptability in scenarios such as scaling up and down and path changes in distributed microservice systems caused by using a single reinforcement learning or static threshold model.

[0120] Referring to Figure 2, which is a flowchart of a fault handling method for a distributed microservice system provided in an embodiment of this application, the fault handling method for the distributed microservice system is basically the same as that shown in Figure 1, except that step 103 may include the following:

[0121] Step 105: In the online environment, construct log-indicator dual-channel features for the real-time monitoring stream according to the pre-set analysis window, obtain the anomaly impact range score, and send the low-confidence analysis window to the meta-reinforcement learning training queue according to the anomaly impact range score.

[0122] In this embodiment, step 105 may include the following steps:

[0123] S51. Sort the policy action sequence, log fragments, and indicator spikes in ascending order according to the call chain timestamp, and simultaneously divide the three types of streams into analysis windows of fixed length. In each Internally reserved microservice ID Parent chain depth In addition, a local time offset within the window is included to ensure that feature extraction maintains a consistent baseline in both spatial and temporal dimensions. In this embodiment, the length of the analysis window can be 100ms.

[0124] S52. In the log channel, a weight dictionary is used. Extract sudden phrases and record the order of their first appearance. In the indicator channel, calculate the slope of each indicator curve. Peak amplitude With the rate of fall Finally, the two channel vectors are concatenated into a fixed-length multimodal feature as shown in formula (10). :

[0125] (10)

[0126] in, To be according to The first occurrence order vector of burst phrases obtained from the index.

[0127] S52 aligns numerical and textual features to the same scale through vectorization, avoiding weight imbalance between modalities.

[0128] S53. Before entering the coding layer, concatenate short-period convolutional blocks covering three scales. Local path extraction is performed. Convolutional kernels share weight matrices to reduce parameter redundancy, and instantaneous co-occurrence of log phrases and local trajectories of indicator spikes are captured through cross-scale residual connections.

[0129] For example, the lengths of the three-scale short cycles can be 20ms, 50ms, and 100ms.

[0130] S54. The convolutional output is fed into a parallel N-Gram self-attention module. First, word sequences with a sudden increase in the frequency of consecutive occurrences are detected within the log channel. Then, the spike features of the index are aligned with the same window to achieve lateral correlation. This improves the model's ability to explicitly express cross-modal transient dependencies within the same analysis window.

[0131] S55, Embed at the hidden layer append call chain position ,in, and These represent the offsets from the upstream and downstream nodes, respectively. The embedding vectors are mapped to the same-dimensional space through a fully connected layer, enabling the network to explicitly learn the hierarchical dependencies between microservices during the training phase.

[0132] S56, Decoder-side fusion short-cycle representation With position embedding The score for the scope of abnormal impact is output using the following formula (11). With priority self-healing action candidates :

[0133] ;

[0134] (11)

[0135] Where u is the rating vector. , These are trainable weights; For meta-strategy; For the action space. When When the threshold is adaptively generated from the historical best distribution, the current analysis window will be... The data is fed back to step 103 to trigger policy retraining, forming a micro-closed loop of encoding inference and policy optimization, enabling the system to maintain continuous learning capabilities against novel transient anomalies.

[0136] Referring to Figure 3, which is a flowchart of a fault handling method for a distributed microservice system provided in an embodiment of this application, the fault handling method for the distributed microservice system is basically the same as that shown in Figure 3, except that step 105 may include the following:

[0137] Step 106: Parse the anomaly impact range score into a root cause confidence matrix, combine it with the call chain depth to determine high-frequency fault nodes and blank coverage nodes, generate exploration bias vectors based on high-frequency fault nodes and blank coverage nodes, and adjust the meta-reinforcement learning model and chaotic injection unit based on the exploration bias vectors.

[0138] In this embodiment, step 106 may include the following steps:

[0139] S61. Analyze the abnormal impact range score into a root cause confidence matrix. ,in, Let M be the number of transient events and M be the total number of failed components. For the e-th event, take the component index with the highest confidence level. Combined with the call chain depth Vector of influence Assemble the ternary label using the following formula (12):

[0140] (12)

[0141] Will Write to root cause label cache This provides a structured basis for frequency aggregation.

[0142] S62, in the scrolling time window Internally, for cache The labels are counted to form a node frequency vector. Calculate the global average. with standard deviation ,when The time marker is used to identify high-frequency fault nodes, when Components that are active in the topology are marked as blank coverage areas. The node distribution is then compared with the action probability distribution of the current meta-policy on the corresponding nodes. Merge and generate a strategy deviation report. .

[0143] S63, put Mapped to exploration bias vector In this context, high-frequency nodes are assigned an attenuation coefficient, while blank nodes are assigned an amplification coefficient. New exploration probability. The exponent is generated by normalization using the following formula (13):

[0144] ; (13)

[0145] Will The feedback vector is pushed to the meta-reinforcement learning agent in step 103 to rearrange the exploration weights in real time, weaken the fully covered nodes and increase the sampling probability of missed nodes.

[0146] S64, via the chaos injection unit according to Update transient fault data: Increase injection strength for high-confidence root cause nodes And keep the interval unchanged; for new suspicious nodes, maintain the intensity Under the premise of not changing, the injection interval will be shortened to The two-way adjustment mechanism ensures that the critical path remains under continuous pressure while rapidly validating new model conclusions.

[0147] S65. After the listener injection schedule changes, the meta-reinforcement learning agent will adjust the internal convergence window duration parameter. Dynamic mapping to ,in This is the experience scaling factor. It keeps the policy learning pace locked in line with the latest anomaly triggering frequency, preventing action updates from lagging behind.

[0148] S66. Comparison of average recovery time after three consecutive rounds of training. Compared with the average false alarm rate .like and Both declined or the rate of decline exceeded historical thresholds. If the new strategy version is implemented, it will be fixed and synchronized to the production environment; otherwise, it will be rolled back to the previous version and the deviation event set will be updated. Returning to step 105, the training samples are expanded, thus completing the multi-level closed-loop self-evolution of root cause analysis, strategy tuning, injection verification, and performance evaluation.

[0149] Referring to Figure 4, which is a flowchart of a fault handling method for a distributed microservice system provided in an embodiment of this application, the fault handling method for the distributed microservice system is basically the same as that shown in Figure 3, except that step 106 may include the following:

[0150] Step 107: Based on the blind zone weight function, implement directional transient injection at blank coverage nodes through the chaotic injection unit and improve sampling resolution. Detect index intervals that continuously cross historical percentile thresholds online and freeze all data. Generate overlapping cluster label sets using injected meta-information and optimize the meta-strategy based on the overlapping cluster label sets.

[0151] In this embodiment, step 107 may include the following steps:

[0152] S71, Root Cause Heatmap Perform dimensionality merging to obtain the node amplitude increment matrix. and confidence matrix Then, in every 50ms time slice... Within, based on historical amplitude baselines The blind zone weight piecewise function is constructed using the following formula (14). ,like This triggers blind spot recording, creating a short-term overlap anomaly compensation list. .

[0153] (14)

[0154] in, Let m be the amplitude jump variable of node m in time slice t. The historical amplitude standard deviation of node m Multiply by magnification factor , Let m be the root cause confidence of node m in time slice t. Let be the fluctuation interval between node m and its nearest collaborating node in time slice t. The indicator trigger function (value 1 or 0); For Heaviside step function (cross-node collaborative constraint).

[0155] S72, Traverse sequentially through the chaotic injection unit. Apply duration to the corresponding node Sudden CPU load or network jitter, randomly superimposed with a set of cascading restarts of two nodes. All injection events are logged. , used for subsequent alignment.

[0156] S73. Within the injection window, increase the indicator sampling period to At the same time, a link-level unique identifier is appended to the log stream. The data acquisition side uses a high-precision clock synchronization module to ensure that all metrics, logs, and call stack snapshots are aligned within nanosecond-level errors.

[0157] S74. The online detection module performs a sliding window judgment on the indicator stream of each node. If three consecutive sampling points all exceed the historical 95th percentile, the interval is immediately frozen. Full metrics, logs, and call stack snapshots This provides spatiotemporal closure data for subsequent precision calibration.

[0158] S75, Based on injected metadata right Perform nanosecond-level alignment and sequentially label the overlapping cluster numbers. First Node Follow the set of nodes Output short-time overlapping fault label set It also includes the injection type and delay distribution to characterize the differences in propagation links between nodes.

[0159] S76, will The overlapping weights in the reward function are updated by writing them into the experience replay pool via the collaborative training bus. The meta-strategy automatically increases the exploration probability of higher-level nodes in overlapping clusters in subsequent iterations, achieving rapid incremental representation of time-decoupled characteristics.

[0160] S77. After completing three rounds of overlapping label training, evaluate the success rate of anomaly localization within <100ms. If the success rate increases significantly If the target is less than 5%, return to S61 to adjust the injection plan and expand coverage; otherwise, solidify the new strategy and synchronize it to the production environment.

[0161] Re-inject using the following solidification strategy (15):

[0162] ; (15)

[0163] in, This is the baseline success rate before this iteration. For the success rate after the third round, To avoid zero denominators, the system uses extremely small positive numbers. By introducing a piecewise threshold decision function, it achieves quantitative closed-loop and adaptive enhancement of the ultra-short-period overlapping positioning capability.

[0164] Referring to Figure 5, which is a flowchart of a fault handling method for a distributed microservice system provided in an embodiment of this application, as shown in Figure 4, the fault handling method for the distributed microservice system is basically the same as that shown in Figure 4, except that step 107 may include the following:

[0165] Step 108: Reverse traversal to determine suspected trigger points, solidify trigger identifiers through semantic matching, generate a heat matrix and visual source tracing information, package the source tracing information and repair action set and push them to the operation and maintenance work order system, and update the case library and meta-policy reward function after the repair is confirmed to be effective.

[0166] In this embodiment, step 108 may include the following steps:

[0167] S81, Read the overlapping cluster tag set According to the call chain identifier Backtrack all node metric streams within 200ms With log fragments Aligned to a unified nanosecond-level timeline Write to the time-series linked list The linked list index maintains the topological parent-child hierarchical order, providing a complete context for reverse traversal.

[0168] S82, Reverse scan from the downstream of the linked list, when the node Time slice Simultaneously satisfy the indicator jump When the log anomaly detector returns 1, it is temporarily recorded as a candidate trigger point, and the confidence score is calculated using the following formula (16). The first threshold-exceeding node was fixed as a suspected trigger point:

[0169] ; (16)

[0170] In the formula, Assign a semantic score to log anomalies; For nodes The hierarchical distance to the head of the chain; This represents the maximum depth of the link. This is a weighting factor.

[0171] S83. Extract log segments within 50ms before and after the suspected trigger point, and process them using the Transformer semantic module and the fault keyword library. Matching, similarity Reaching the threshold The trigger identifier is fixed and written to the trigger cache. .

[0172] S84, with key polymerization Statistical frequency With average delay Combined with node business weight The micro-thermal matrix is ​​constructed using the following formula (17). :

[0173] ; (17)

[0174] in, This represents the global average trigger latency. For the global node set; For adjustment coefficients. Matrix elements. Simultaneously measure frequency, response latency, and business weight, and then visualize the mapping.

[0175] S85, front-end engine reading and The system marks the starting point of spikes in the timeline view, highlights hot nodes in the topology view, and renders a word cloud of logs in the sidebar, enabling multi-dimensional on-screen comparison and interactive drill-down.

[0176] S86, Report Generator will Package the three-view screenshots and complete source tracing path into a WebAssembly, and attach the impact assessment matrix. With repair action set The message is pushed to the maintenance work order platform and synchronized to the alarm channel.

[0177] S87. When the maintenance team confirms that the repair is effective, the trigger point will be... Store in the case library A positive factor is added to the meta-policy reward function; if the repair is ineffective, the process automatically returns to S81 to trigger the next round of micro-analysis until the case is successfully solidified, thus achieving a closed loop of positioning-verification-knowledge accumulation.

[0178] Referring to Figure 6, Figure 6 is a structural diagram of a fault handling device for a distributed microservice system provided in an embodiment of this application. As shown in Figure 6, the fault handling device for the distributed microservice system includes:

[0179] The first module 601 is used to inject transient fault data based on micro-explosion scenarios into the target microservice group through the chaos injection unit, and to collect real-time abnormal data through non-intrusive probes to generate transient snapshots containing metrics, logs and topology information.

[0180] The second module 602 is used to obtain single-service micro-explosion segments based on the transient snapshot, aggregate the single-service micro-explosion segments based on cross-service overlap, generate transient families, encode the indicator slope, resource contention traces, log burst frequency and topology slices in the transient families into multi-view time series vectors, write them into the index library, and push them to the meta-reinforcement learning training queue.

[0181] The third module 603 is used to train a meta-reinforcement learning model for high-frequency mutation fault scenarios in an isolated environment based on the meta-reinforcement learning training queue, and to determine the meta-policy based on the immediate reward function.

[0182] The fourth module 604 is used to adjust the chaos injection unit according to the meta-strategy to perform fault handling operations on the distributed microservice system.

[0183] Optionally, the target microservice group is obtained based on the gray-scale release period, automatic scaling records, and business peak curves.

[0184] Optionally, the timestamps of the target microservice group are aligned to the nanosecond level using a shared logical clock.

[0185] Optionally, the first module 601 is further configured to, when it is determined from the multi-view time-series vector that a call topology hole exists, instruct the chaotic injection unit to expand the uncovered blind zone and update the transient fault data.

[0186] Optionally, the instant reward function is determined based on the recovery time, false alarm rate, and resource overhead.

[0187] Optionally, the third module 603 is further configured to construct log-indicator dual-channel features of the real-time monitoring stream in an online environment according to a pre-set analysis window, obtain an anomaly impact range score, and send the low-confidence analysis window to the meta-reinforcement learning training queue according to the anomaly impact range score.

[0188] Optionally, the third module 603 is further configured to parse the anomaly impact range score into a root cause confidence matrix, combine it with the call chain depth to determine high-frequency fault nodes and blank coverage nodes, generate an exploration bias vector based on the high-frequency fault nodes and blank coverage nodes, and adjust the meta-reinforcement learning model and the chaos injection unit based on the exploration bias vector.

[0189] Optionally, the third module 603 is further configured to perform directional transient injection and improve sampling resolution at the blank coverage node through the chaotic injection unit according to the blind zone weight function, detect index intervals that continuously cross historical percentile thresholds online and freeze all data, generate an overlapping cluster label set using injected meta-information, and optimize the meta-strategy based on the overlapping cluster label set.

[0190] The fault handling device for a distributed microservice system provided in this application embodiment can execute the fault handling method embodiment for the distributed microservice system described above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0191] In this embodiment, the chaotic injection unit's operation of injecting transient fault data based on micro-explosive scenarios into the target microservice group can be adjusted according to a meta-policy. This meta-policy is determined during the training of the meta-reinforcement learning model based on an immediate reward function, ensuring close coupling between fault injection and real-time business dynamics, fully exposing potential transient faults, and thus guaranteeing the accuracy of subsequent fault handling. Since the immediate reward function can change in real time according to actual conditions, the technical solution provided in this embodiment does not require offline training, thereby ensuring the real-time nature of fault handling. Furthermore, this embodiment uses a meta-reinforcement learning model for learning, avoiding the problem of poor fault handling adaptability in scenarios such as scaling up and down and path changes in distributed microservice systems caused by using a single reinforcement learning or static threshold model.

[0192] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] This application provides a communication device, including: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the fault handling method of the distributed microservice system as described above.

[0195] This application also provides a readable storage medium storing a program. When executed by a processor, this program implements the various processes of the above-described distributed microservice system fault handling method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0196] This application also provides a computer program product, including computer instructions. When executed by a processor, these computer instructions implement the various processes of the above-described distributed microservice system fault handling method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0197] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0199] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A fault handling method for a distributed microservice system, characterized in that, include: Transient fault data based on micro-explosion scenarios is injected into the target microservice group through a chaotic injection unit. Real-time abnormal data is collected through non-intrusive probes to generate transient snapshots containing metrics, logs, and topology information. Based on the transient snapshots, single-service micro-explosion segments are obtained. The single-service micro-explosion segments are aggregated based on cross-service overlap to generate transient families. The metric slope, resource contention traces, log burst frequency, and topology slices within the transient families are encoded into multi-view time-series vectors, written into an index library, and pushed to the meta-reinforcement learning training queue. In an isolated environment, a meta-reinforcement learning model is trained for high-frequency mutation fault scenarios based on the meta-reinforcement learning training queue, and a meta-policy is determined based on the immediate reward function; the chaos injection unit is adjusted according to the meta-policy to perform fault handling operations on the distributed microservice system.

2. The method according to claim 1, characterized in that, The target microservice group is obtained based on the gray-scale release period, automatic scaling records, and business peak curves.

3. The method according to claim 1, characterized in that, The timestamps of the target microservice group are aligned to the nanosecond level using a shared logical clock.

4. The method according to claim 1, characterized in that, Also includes: If a call topology hole is determined to exist based on the multi-view time-series vector, the chaotic injection unit is instructed to expand the uncovered blind zone and update the transient fault data.

5. The method according to claim 1, characterized in that, The instant reward function is determined based on recovery time, false alarm rate, and resource overhead.

6. The method according to claim 1, characterized in that, Also includes: In an online environment, log-metric dual-channel features are constructed on the real-time monitoring stream according to a pre-set analysis window to obtain an anomaly impact range score. Based on the anomaly impact range score, the low-confidence analysis window is sent to the meta-reinforcement learning training queue.

7. The method according to claim 6, characterized in that, Also includes: The anomaly impact range score is parsed into a root cause confidence matrix. Combined with the call chain depth, high-frequency fault nodes and blank coverage nodes are identified. An exploration bias vector is generated based on the high-frequency fault nodes and blank coverage nodes. The meta-reinforcement learning model and the chaotic injection unit are adjusted based on the exploration bias vector.

8. The method according to claim 7, characterized in that, Also includes: Based on the blind zone weight function, the chaotic injection unit performs directional transient injection on the blank coverage node and improves the sampling resolution. It detects index intervals that continuously cross the historical percentile threshold online and freezes all data. It uses the injected meta-information to generate an overlapping cluster label set and optimizes the meta-strategy based on the overlapping cluster label set.

9. A fault handling device for a distributed microservice system, characterized in that, include: The first module is used to inject transient fault data based on micro-explosion scenarios into the target microservice group through the chaos injection unit, and to collect real-time abnormal data through non-intrusive probes to generate transient snapshots containing metrics, logs and topology information. The second module is used to obtain single-service micro-explosion segments based on the transient snapshot, aggregate the single-service micro-explosion segments based on cross-service overlap, generate transient families, encode the indicator slope, resource contention traces, log burst frequency, and topology slices within the transient families into multi-view time-series vectors, write them into an index library, and push them to the meta-reinforcement learning training queue; the third module is used to train a meta-reinforcement learning model for high-frequency sudden failure scenarios in an isolated environment based on the meta-reinforcement learning training queue, and determine the meta-policy based on the immediate reward function; The fourth module is used to adjust the chaos injection unit according to the meta-strategy to perform fault handling operations on the distributed microservice system.

10. A communication device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program from the memory to implement the steps in the fault handling method for a distributed microservice system as described in any one of claims 1 to 8.

11. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps in the fault handling method for a distributed microservice system as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the fault handling method for a distributed microservice system as described in any one of claims 1 to 8.

Images