An IT operation and maintenance anomaly detection and self-healing decision method based on reinforcement learning

Abstract

The application relates to the technical field of intelligent operation and maintenance, and discloses an IT operation and maintenance anomaly detection and self-healing decision method based on reinforcement learning, which comprises the following steps: collecting real-time operation and maintenance data of a micro-service system, constructing a call chain directed acyclic graph based on distributed call chain Span data; detecting micro-service interface response time anomalies and triggering a self-healing process, generating an abnormal influence subgraph based on abnormal period Span data, identifying a delay contribution path and locating a candidate root cause node set; calculating the causal contribution degree of the candidate root cause node to a service level target indicator, and selecting the one with the maximum contribution degree as the identified root cause node; defining a self-healing action set, generating a causal action mask vector according to the downstream influence range of the identified root cause node and an operation and maintenance safety strategy; extracting four types of features, constructing a reinforcement learning state vector; constructing an instant reward function; using a value network to train and update the self-healing decision strategy; deploying the strategy network in a production environment, and selecting and executing a self-healing action according to the state.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology, specifically to an IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning. Background Technology

[0002] The core transaction system of large commercial banks is a critical infrastructure supporting the continuity of financial business. This system typically adopts a distributed microservice architecture and a relational database cluster for collaborative deployment. During the end-of-day batch reconciliation tasks executed at the end of each business day, due to the database optimizer's statistics not being updated in a timely manner or the wrong execution plan being selected, the execution time of complex aggregation queries in certain patterns suddenly increases from the normal range to tens of seconds. A large number of such slow queries occupy database connection resources for a long time, causing the available connections in the microservice connection pool to be exhausted, which in turn triggers a cascading unavailability failure of multiple critical microservices that depend on the same database instance, such as account query services and risk control services.

[0003] However, existing operation and maintenance methods rely on setting fixed thresholds for the number of database connections or slow queries to trigger alarms. After receiving the alarm, operation and maintenance personnel log in to the database terminal and manually execute the Structured Query Language termination command or restart the relevant service instance. This manual response mode has inherent defects such as delayed fault perception, reliance on personal experience for root cause location, strong blindness in recovery operations, and easy to cause data inconsistency or extended business interruption time. When facing slow query avalanche failures triggered by the deterioration of the execution plan in the end-of-day reconciliation scenario, it is difficult to achieve fast and accurate automated self-healing recovery while ensuring the security of business data. Therefore, we propose an IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a reinforcement learning-based method for IT operations and maintenance anomaly detection and self-healing decision-making, comprising the following steps: Step S1: Collect real-time operation and maintenance data of the microservice system, including distributed call chain tracing span data, system resource indicator data, and database running status data, and construct a directed acyclic graph of the distributed call chain based on the tracing span data; Step S2: Perform anomaly detection on the response time series of the microservice interface. When an anomaly is detected, trigger the self-healing decision process, obtain the tracking span data from before the anomaly occurred to the current time to generate an anomaly impact subgraph, identify the path with the largest delay contribution in the anomaly impact subgraph, and locate the tracking span nodes with significantly higher time consumption than the historical level as the candidate root cause node set. Step S3: For each candidate root cause node in the candidate root cause node set, calculate the causal contribution of the candidate root cause node to the service level target indicator, and select the candidate root cause node with the largest causal contribution as the identified root cause node. Step S4: Define a set of executable self-healing actions. Generate a causal action mask vector based on the downstream affected range of the identified root cause node in the anomaly impact subgraph and the preset operation and maintenance security strategy to limit the effective execution range of the self-healing actions. Step S5: Extract system resource features, abnormal topology features, causal contribution features, and historical action statistics features, and concatenate them to form the state vector of the reinforcement learning semi-Markov decision process. Step S6: Construct an immediate reward function that includes service level target recovery items, causal alignment items, and action risk cost items to evaluate the quality of self-healing actions; Step S7: Use a value network with an action masking mechanism to learn and update the state-action value function, and train to obtain a self-healing decision-making strategy network; Step S8: Deploy the trained policy network in the production environment, respond to abnormal trigger signals in real time, select the optimal self-healing action based on the current state vector and execute it, continuously monitor the service level target indicators after execution, and decide whether to make a new round of decisions based on the recovery situation until the service returns to normal.

[0005] Preferably, step S1 includes the following steps: Step S11: Deploy extended Berkeley package filter probes on each worker node of the container orchestration cluster to perform system call tracing on microservice container processes, capture the entry and exit events of Hypertext Transfer Protocol requests and Remote Procedure Call requests, and generate tracing span data that conforms to the Open Telemetry Standard. Step S12: Collect container CPU usage, memory usage, network throughput, database connection count, active transaction count, and row lock wait count using monitoring and alarm tools; collect slow query logs from the database using log collection tools. Step S13: Deploy a distributed tracing system collector to receive tracing span data, aggregate the tracing span data with a preset time window, and construct a distributed call chain directed acyclic graph, where nodes represent tracing span operations and include service name, operation name, time consumption, and status code attributes, directed edges represent direct call relationships between tracing spans, and each edge records the average call time and the number of calls.

[0006] Preferably, step S2 includes the following steps: Step S21: Use a variational autoencoder to perform reconstruction probability estimation on the P99 response time series of the microservice interface. When the reconstruction error of a preset number of consecutive sampling points exceeds the dynamic threshold, determine that the corresponding interface has an anomaly and trigger the self-healing decision process. Step S22: Obtain all tracking span data from the first preset time before the exception occurred to the current time. Using the entry tracking span corresponding to the exception interface as the endpoint, traverse in reverse in the directed acyclic graph of the distributed call chain to generate the exception impact subgraph. Step S23: In the anomaly impact subgraph, for each path from the entry tracking span to the leaf tracking span, calculate the cumulative path time. The cumulative path time is used to characterize the total sum of delays of each tracking span node on a single call path, in order to identify the call path that contributes the most to the overall response time. The formula for calculating the cumulative path time is as follows: In the formula, The first subgraph representing the abnormal influence A path from the entry point to the leaf point, tracing the span; Representing a path The tracking span nodes passed through; Indicates tracking span nodes The time taken; Representing a path The cumulative sum of the time spent tracking all nodes across the span; Step S24: Identify the tracking span nodes whose cumulative time is higher than the historical level for the same period as candidate root cause nodes, and form a set of candidate root cause nodes.

[0007] Preferably, step S3 includes the following steps: Step S31: Select candidate root cause nodes The tracking span time variable is denoted as The service level target indicator is marked as Using historical observation data from normal and fault periods, variables are identified through a structural causal model. With variables The causal direction between them; Step S32: Calculate candidate root cause nodes Service level target indicators When determining the causal contribution, the backdoor adjustment formula in operator calculus is used to estimate the average causal effect; the average causal effect characterizes the effect on the variable. Under the condition that intervention causes it to take abnormally high values, the variable Expected value and variables variables under the condition of normal median The difference between the expected values ​​is used to quantify candidate root cause nodes. The extent to which the anomaly affects service level objectives; the formula for calculating the average causal effect is as follows: In the formula, Indicates candidate root cause nodes The causal contribution; This represents the interferometric quantifier in operator calculus; Represents the expectation operator; Indicates tracking span Time-consuming observations under the current anomalous state; Indicates tracking span The median time spent during a normal historical period; Step S33: Using the natural intervention events recorded in the historical slow query log as instrumental variables, approximate the expected value under the intervention distribution, and select... The candidate root cause node with the largest value is selected as the root cause node.

[0008] Preferably, step S4 includes the following steps: Step S41: Define the set of self-healing actions, which is represented as follows: In the formula, Represents the space of executable self-healing actions; This indicates the termination of the Structured Query Language thread action corresponding to the root cause tracing span; This indicates an action to switch read request traffic to the read-only replica database endpoint; This indicates actions that dynamically adjust the timeout threshold and concurrent execution limit of the microservice circuit breaker; This indicates an action that triggers service degradation logic and returns a cached approximate result. This indicates an action to temporarily increase the maximum number of connections in the database connection pool and restart the service instance; The self-healing action set is an exhaustive enumeration and formal definition of the recovery operations that the system can take after detecting an anomaly. Its purpose is to provide a clear and discrete action selection space for the reinforcement learning policy network, so that the policy network can make decisions based on state characteristics among a limited number of candidate actions.

[0009] Step S42: Calculate the set of all downstream affected service nodes of the identified root cause node in the anomaly impact subgraph, and generate a causal action mask vector; the causal action mask vector is used to mask actions that cannot effectively block the fault propagation chain during the reinforcement learning decision-making process, limiting the action exploration space to the operational causal reachability range; causal action mask vector The Middle The rules for the values ​​of each component are as follows: In the formula, Representing state Next The mask component value corresponding to each action is 1, which indicates that the action is valid, and 0, which indicates that the action is invalid. Indicates the identification of root cause nodes The set of downstream affected service nodes in the anomaly impact subgraph; This represents the set of actions that are prohibited from being executed according to the preset operation and maintenance security policy; This represents the relationship between the historical work order knowledge mapping table and the identified root cause node. The set of high-frequency successful actions corresponding to the type; Step S43: Maintain the historical work order knowledge mapping table, record the high-frequency actions of each root cause type in historical successful recovery cases. If an action appears in the high-frequency action list, the mask component corresponding to the action is forcibly set to 1 according to the mask value rule.

[0010] Preferably, in step S5, the features for constructing the state vector include: System resource characteristics include: active database connection pool connections, waiting connections, maximum number of connections, container CPU utilization, memory utilization, average database row lock wait time, and message queue consumption latency. The dimensions of abnormal topology features include: P99 time for identifying the root cause node and tracing the span, year-on-year increase in time, out-degree and in-degree of the root cause node in the abnormal impact subgraph, critical path delay ratio, and number of affected downstream services. The causal contribution feature dimensions include: normalized causal contribution degree and contribution ranking position; The statistical features of historical actions include: the maximum cosine similarity between the current fault fingerprint and similar faults in historical work orders, the average recovery time of similar fault cases, and the proportion of high-frequency actions used.

[0011] Preferably, in step S6, the immediate reward function is used to evaluate the degree of improvement in the system state after the self-healing action is executed. It is calculated by a weighted combination of three parts: the recovery magnitude of the service level target, the degree of alignment of the causal path, and the risk cost of the action execution. The formula for calculating the immediate reward function is as follows: In the formula, express The instant reward value gained after performing a self-healing action at any time; This represents the percentage decrease in the response time of the entry interface after the action is performed, relative to the abnormal peak value. The value ranges from zero to one. Indicates causal alignment terms, when the selected action... The action applies to identifying the root cause node as a direct child node in the cause-effect graph, and the action... When the optimal action is consistent with the corresponding root cause type in the historical work order, the causal alignment term takes the value of a linear mapping of the causal contribution. Indicates action The corresponding risk costs are pre-set based on the degree of negative impact the action may have on the business; , , These are preset weight hyperparameters used to adjust the relative importance of the three components in the reward calculation.

[0012] Preferably, step S7 includes the following steps: Step S71: Construct a dual deep Q-network, including a shared feature extraction layer, a value stream branch, and a dominance stream branch; the state-action value function is used to estimate the value of actions in the state. Select action The expected cumulative reward can be represented by a combination of value stream and advantage stream, separating the value of the state itself from the relative merits of each action; the formula for calculating the output of the state-action value function is as follows: In the formula, Indicates the state Select action Expected cumulative return; Indicates only with state Related value functions; Indicates the state Take action below The superiority or inferiority functions; Indicates the size of the action space; This is an index variable used to iterate through all actions in the action space; Step S72: Apply the causal action mask vector to the output layer, set the Q value corresponding to the invalid action of the mask component to the preset minimum value, so that the policy ignores invalid actions during the selection or exploration process; Step S73: Offline pre-training stage, extract state transition samples from the historical operation and maintenance work order database, and use expert demonstration data to perform supervised pre-training on the network. The loss function is the mean square error between the Q value and the target Q value. Step S74: In the online fine-tuning stage, fault injection is simulated in the shadow cluster of the non-production environment. The policy network makes online decisions and executes actions. The newly generated transfer samples are stored in the priority experience replay buffer. The network parameters are updated by sampling according to the time difference error priority and using the adaptive moment estimation optimizer.

[0013] Preferably, step S8 includes the following steps: Step S81: Encapsulate the trained and converged policy network model into a container orchestration custom resource definition controller, and deploy it in the operation and maintenance plane namespace in the form of container group deployment; Step S82: After the anomaly detection module sends a trigger signal, it pulls data from the monitoring and alarm tool and the distributed tracing system in real time to construct the current state vector; Step S83: Input the current state vector into the policy network for forward computation and select the self-healing action corresponding to the maximum Q value; Step S84: Execute the selected self-healing action by calling the container orchestration application interface, database driver, or configuration center interface; Step S85: Continuously monitor the service level target indicators. If the indicators do not recover to the baseline level within the preset time window, re-enter the decision loop and store the failure record in the experience base. If the indicators recover to normal, write the fault cause-effect graph, state sequence and successful action into the work order knowledge base.

[0014] The present invention has the following beneficial effects: (1) This invention expands the reinforcement learning state space from simple numerical indicators to the causal structure level of fault propagation by introducing distributed causal topology tracking and Do-calculus causal contribution calculation. In step S3, the average causal effect quantifies the actual impact of candidate root cause node anomalies on service level target indicators, making the decision basis change from statistical correlation to interventional causal relationship. The causal action masking mechanism established in step S4 further constrains the self-healing action space to an effective subset that can directly block the fault propagation chain, fundamentally avoiding the risk of reinforcement learning agents blindly trying irrelevant components or high-risk operations during the online exploration phase. The above causal alignment design makes the self-healing decision process strictly follow the physical causal path of fault propagation, rather than relying on association rules that are susceptible to noise interference, thereby significantly reducing the probability of misoperation in complex microservice call topologies and ensuring the continuity of production business.

[0015] (2) This invention systematically embeds operation and maintenance semantic constraints at three levels: state representation construction, reward function shaping, and policy output constraints. The state vector constructed in step S5 integrates abnormal topology features, causal contribution features, and historical action statistical features, so that the network input not only includes real-time monitoring values, but also carries fault context information refined by domain knowledge. The causal alignment reward shaping item designed in step S6 uses the high-frequency actions that have been verified to be effective in historical work orders as positive guidance signals, so that the reward function, in addition to pursuing the single goal of restoring the service level target index, simultaneously considers the causal rationality and experience consistency of the operation path. The mask Dueling DQN architecture adopted in step S7 ensures that the network output layer strictly follows the operation and maintenance safety boundary. Through multi-level knowledge injection, the reinforcement learning model retains the ability to generalize and adapt to unknown fault modes, while avoiding the high trial and error costs brought about by exploring from scratch, and realizes the complementary advantages of experience-driven and data-driven approaches.

[0016] (3) This invention uses a semi-Markov decision process to model the fault evolution process and introduces a training paradigm that combines offline pre-training and online fine-tuning in the policy learning stage. Step S7 uses expert demonstration sequences from the historical maintenance work order database for supervised pre-training, so that the policy network has learned the basic coping logic under common fault modes before contacting the real production environment. The subsequent online fine-tuning in the shadow cluster through fault injection enables the policy network to have the ability to explore and adapt to unseen fault types. The abnormal topology features and causal contribution features included in the state vector in step S5 have strong fault mode invariance and do not depend on specific thresholds or fixed rules, so that the same policy network can generalize to cope with connection pool avalanche fault variants caused by different slow query modes and different service call link combinations. This design enables the self-healing method to maintain stable decision quality when facing diverse slow query scenarios caused by fluctuations in optimizer statistics in the end-of-day reconciliation task.

[0017] (4) The closed-loop self-healing execution process constructed by this invention realizes a fully automated operation and maintenance closed loop from anomaly detection triggering, root cause localization, strategy decision-making to action execution and effect verification; Step S2, based on the anomaly detection mechanism of variational autoencoder, can start the decision-making process in the early stage of service level target index degradation, avoiding the passive situation of manual intervention only when the fault accumulates to a serious stage under the traditional threshold alarm mode; Step S8 encapsulates the trained converged policy network model into a Kubernetes custom resource definition controller, so that the self-healing decision-making capability is embedded into the production cluster control plane in a cloud-native manner, and the decision latency is compressed to the second level; Compared with the traditional work mode that relies on operation and maintenance personnel to log in to the server, manually analyze logs, and execute recovery instructions, this method parallelizes the three originally serial manual links of fault perception, diagnosis and recovery into an automated pipeline, which reduces the dependence on real-time presence of expert experience and effectively compresses the erosion period of business availability caused by the duration of the fault. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an IT operations and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning, as described in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 As shown, this invention is a reinforcement learning-based method for IT operations and maintenance anomaly detection and self-healing decision-making, comprising the following steps: Step S1: Collect real-time operation and maintenance data of the microservice system, including distributed call chain tracing span data, system resource indicator data, and database running status data, and construct a directed acyclic graph of the distributed call chain based on the tracing span data; Step S2: Perform anomaly detection on the response time series of the microservice interface. When an anomaly is detected, trigger the self-healing decision process, obtain the tracking span data from before the anomaly occurred to the current time to generate an anomaly impact subgraph, identify the path with the largest delay contribution in the anomaly impact subgraph, and locate the tracking span nodes with significantly higher time consumption than the historical level as the candidate root cause node set. Step S3: For each candidate root cause node in the candidate root cause node set, calculate the causal contribution of the candidate root cause node to the service level target indicator, and select the candidate root cause node with the largest causal contribution as the identified root cause node. Step S4: Define a set of executable self-healing actions. Generate a causal action mask vector based on the downstream affected range of the identified root cause node in the anomaly impact subgraph and the preset operation and maintenance security strategy to limit the effective execution range of the self-healing actions. Step S5: Extract system resource features, abnormal topology features, causal contribution features, and historical action statistics features, and concatenate them to form the state vector of the reinforcement learning semi-Markov decision process. Step S6: Construct an immediate reward function that includes service level target recovery items, causal alignment items, and action risk cost items to evaluate the quality of self-healing actions; Step S7: Use a value network with an action masking mechanism to learn and update the state-action value function, and train to obtain a self-healing decision-making strategy network; Step S8: Deploy the trained policy network in the production environment, respond to abnormal trigger signals in real time, select the optimal self-healing action based on the current state vector and execute it, continuously monitor the service level target indicators after execution, and decide whether to make a new round of decisions based on the recovery situation until the service returns to normal.

[0022] Step S1 includes the following steps: Step S11: Deploy extended Berkeley package filter probes on each worker node of the container orchestration cluster to perform system call tracing on microservice container processes, capture the entry and exit events of Hypertext Transfer Protocol requests and Remote Procedure Call requests, and generate tracing span data that conforms to the Open Telemetry Standard. Step S12: Collect container CPU usage, memory usage, network throughput, database connection count, active transaction count, and row lock wait count using monitoring and alarm tools; collect slow query logs from the database using log collection tools. Step S13: Deploy a distributed tracing system collector to receive tracing span data, aggregate the tracing span data with a preset time window, and construct a distributed call chain directed acyclic graph, where nodes represent tracing span operations and include service name, operation name, time consumption, and status code attributes, directed edges represent direct call relationships between tracing spans, and each edge records the average call time and the number of calls.

[0023] Step S2 includes the following steps: Step S21: Use a variational autoencoder to perform reconstruction probability estimation on the P99 response time series of the microservice interface. When the reconstruction error of a preset number of consecutive sampling points exceeds the dynamic threshold, determine that the corresponding interface has an anomaly and trigger the self-healing decision process. Step S22: Obtain all tracking span data from the first preset time before the exception occurred to the current time. Using the entry tracking span corresponding to the exception interface as the endpoint, traverse in reverse in the directed acyclic graph of the distributed call chain to generate the exception impact subgraph. Step S23: In the anomaly impact subgraph, for each path from the entry tracking span to the leaf tracking span, calculate the cumulative path time. The cumulative path time is used to characterize the total sum of delays of each tracking span node on a single call path, in order to identify the call path that contributes the most to the overall response time. The formula for calculating the cumulative path time is as follows: In the formula, The first subgraph representing the abnormal influence A path from the entry point to the leaf point, tracing the span; Representing a path The tracking span nodes passed through; Indicates tracking span nodes The time taken; Representing a path The cumulative sum of the time spent tracking all nodes across the span; Step S24: Identify the tracking span nodes whose cumulative time is higher than the historical level for the same period as candidate root cause nodes, and form a set of candidate root cause nodes.

[0024] Step S3 includes the following steps: Step S31: Select candidate root cause nodes The tracking span time variable is denoted as The service level target indicator is marked as Using historical observation data from normal and fault periods, variables are identified through a structural causal model. With variables The causal direction between them; Step S32: Calculate candidate root cause nodes Service level target indicators When determining the causal contribution, the backdoor adjustment formula in operator calculus is used to estimate the average causal effect; the average causal effect characterizes the effect on the variable. Under the condition that intervention causes it to take abnormally high values, the variable Expected value and variables variables under the condition of normal median The difference between the expected values ​​is used to quantify candidate root cause nodes. The extent to which the anomaly affects service level objectives; the formula for calculating the average causal effect is as follows: In the formula, Indicates candidate root cause nodes The causal contribution; This represents the interferometric quantifier in operator calculus; Represents the expectation operator; Indicates tracking span Time-consuming observations under the current anomalous state; Indicates tracking span The median time spent during a normal historical period; Step S33: Using the natural intervention events recorded in the historical slow query log as instrumental variables, approximate the expected value under the intervention distribution, and select... The candidate root cause node with the largest value is selected as the root cause node.

[0025] Step S4 includes the following steps: Step S41: Define the set of self-healing actions, which is represented as follows: In the formula, Represents the space of executable self-healing actions; This indicates the termination of the Structured Query Language thread action corresponding to the root cause tracing span; This indicates an action to switch read request traffic to the read-only replica database endpoint; This indicates actions that dynamically adjust the timeout threshold and concurrent execution limit of the microservice circuit breaker; This indicates an action that triggers service degradation logic and returns a cached approximate result. This indicates an action to temporarily increase the maximum number of connections in the database connection pool and restart the service instance; The self-healing action set is an exhaustive enumeration and formal definition of the recovery operations that the system can take after detecting an anomaly. Its purpose is to provide a clear and discrete action selection space for reinforcement learning policy networks, enabling the policy network to make decisions based on state characteristics from a limited number of candidate actions.

[0026] Step S42: Calculate the set of all downstream affected service nodes of the identified root cause node in the anomaly impact subgraph, and generate a causal action mask vector; the causal action mask vector is used to mask actions that cannot effectively block the fault propagation chain during the reinforcement learning decision-making process, limiting the action exploration space to the operational causal reachability range; causal action mask vector The Middle The rules for the values ​​of each component are as follows: In the formula, Representing state Next The mask component value corresponding to each action is 1, which indicates that the action is valid, and 0, which indicates that the action is invalid. Indicates the identification of root cause nodes The set of downstream affected service nodes in the anomaly impact subgraph; This represents the set of actions that are prohibited from being executed according to the preset operation and maintenance security policy; This represents the relationship between the historical work order knowledge mapping table and the identified root cause node. The set of high-frequency successful actions corresponding to the type; Step S43: Maintain the historical work order knowledge mapping table, record the high-frequency actions of each root cause type in historical successful recovery cases. If an action appears in the high-frequency action list, the mask component corresponding to the action is forcibly set to 1 according to the mask value rule.

[0027] In step S5, the features used to construct the state vector include: System resource characteristics include: active database connection pool connections, waiting connections, maximum number of connections, container CPU utilization, memory utilization, average database row lock wait time, and message queue consumption latency. The dimensions of abnormal topology features include: P99 time for identifying the root cause node and tracing the span, year-on-year increase in time, out-degree and in-degree of the root cause node in the abnormal impact subgraph, critical path delay ratio, and number of affected downstream services. The causal contribution feature dimensions include: normalized causal contribution degree and contribution ranking position; The statistical features of historical actions include: the maximum cosine similarity between the current fault fingerprint and similar faults in historical work orders, the average recovery time of similar fault cases, and the proportion of high-frequency actions used.

[0028] In step S6, the immediate reward function is used to evaluate the degree of improvement in the system state after the self-healing action is executed. It is calculated by a weighted combination of three parts: the recovery magnitude of the service level target, the degree of alignment of the causal path, and the risk cost of the action execution. The formula for calculating the immediate reward function is as follows: In the formula, express The instant reward value gained after performing a self-healing action at any time; This represents the percentage decrease in the response time of the entry interface after the action is performed, relative to the abnormal peak value. The value ranges from zero to one. Indicates causal alignment terms, when the selected action... The action applies to identifying the root cause node as a direct child node in the cause-effect graph, and the action... When the optimal action is consistent with the corresponding root cause type in the historical work order, the causal alignment term takes the value of a linear mapping of the causal contribution. Indicates action The corresponding risk costs are pre-set based on the degree of negative impact the action may have on the business; , , These are preset weight hyperparameters used to adjust the relative importance of the three components in the reward calculation.

[0029] Step S7 includes the following steps: Step S71: Construct a dual deep Q-network, including a shared feature extraction layer, a value stream branch, and a dominance stream branch; the state-action value function is used to estimate the value of actions in the state. Select action The expected cumulative reward can be represented by a combination of value stream and advantage stream, separating the value of the state itself from the relative merits of each action; the formula for calculating the output of the state-action value function is as follows: In the formula, Indicates the state Select action Expected cumulative return; Indicates only with state Related value functions; Indicates the state Take action below The superiority or inferiority functions; Indicates the size of the action space; This is an index variable used to iterate through all actions in the action space; Step S72: Apply the causal action mask vector to the output layer, set the Q value corresponding to the invalid action of the mask component to the preset minimum value, so that the policy ignores invalid actions during the selection or exploration process; Step S73: Offline pre-training stage, extract state transition samples from the historical operation and maintenance work order database, and use expert demonstration data to perform supervised pre-training on the network. The loss function is the mean square error between the Q value and the target Q value. Step S74: In the online fine-tuning stage, fault injection is simulated in the shadow cluster of the non-production environment. The policy network makes online decisions and executes actions. The newly generated transfer samples are stored in the priority experience replay buffer. The network parameters are updated by sampling according to the time difference error priority and using the adaptive moment estimation optimizer.

[0030] Step S8 includes the following steps: Step S81: Encapsulate the trained and converged policy network model into a container orchestration custom resource definition controller, and deploy it in the operation and maintenance plane namespace in the form of container group deployment; Step S82: After the anomaly detection module sends a trigger signal, it pulls data from the monitoring and alarm tool and the distributed tracing system in real time to construct the current state vector; Step S83: Input the current state vector into the policy network for forward computation and select the self-healing action corresponding to the maximum Q value; Step S84: Execute the selected self-healing action by calling the container orchestration application interface, database driver, or configuration center interface; Step S85: Continuously monitor the service level target indicators. If the indicators do not recover to the baseline level within the preset time window, re-enter the decision loop and store the failure record in the experience base. If the indicators recover to normal, write the fault cause-effect graph, state sequence and successful action into the work order knowledge base.

[0031] Example 1 This embodiment is applied to the core transaction system of a large commercial bank. The system adopts a distributed microservice architecture deployed by a Kubernetes container orchestration cluster, and is backed by a MySQL 8.0 master-slave database cluster to carry core financial businesses such as account query, risk control verification, and end-of-day batch reconciliation. The peak TPS of the business during the end-of-day reconciliation period reaches 5000, and the core business SLO hard requirement is that the interface P99 response time is ≤200ms and the service availability is ≥99.99%.

[0032] This embodiment specifically addresses typical failures in the daily batch reconciliation scenario of this system, such as slow query avalanches, database connection pool exhaustion, and unavailability of multi-core service cascading caused by outdated database optimizer statistics and degraded execution plans. The specific implementation steps are as follows: Step S1: Operation and maintenance data collection and construction of a directed acyclic graph (DAG) of the distributed call chain. Step S11: Tracking Data Collection Deploy extended Berkeley Package Filter (eBPF) probes on all worker nodes of the Kubernetes cluster to perform non-intrusive system call tracing of microservice container processes. Capture incoming and outgoing HTTP and RPC requests in real time, generate tracing span data that conforms to the OpenTelemetry open telemetry standard, and set the sampling interval to 100ms to ensure that the entire call chain is traceable.

[0033] Step S12: System and Database Indicator Collection The Prometheus monitoring and alerting tool is used to collect key metrics such as container CPU utilization, memory utilization, network throughput, database connection count, active transaction count, and row lock wait count in real time, with a collection frequency of 10 seconds per instance. The ELK log collection tool is used to collect slow query logs in real time, with the slow query threshold set to 2 seconds, and to synchronously record SQL statements, execution thread IDs, execution time, and lock wait time attributes.

[0034] Step S13: Call chain DAG construction The Jaeger distributed tracing system collector is deployed to receive all tracing span data. The span data is aggregated with a preset time window of 1 minute to construct a directed acyclic graph of distributed call chains. The graph nodes are tracing span operations, with core attributes such as service name, operation name, time consumption, and status code. The directed edges are the direct call relationships between spans. Each edge records the average call time and the number of calls, completing the real-time visualization construction of the entire link topology.

[0035] Step S2: Anomaly Detection and Generation of Candidate Root Cause Node Set Step S21: Anomaly Detection and Self-Healing Process Trigger A variational autoencoder (VAE) is used to reconstruct the probability estimation of the P99 response time series of the core reconciliation service and account query service interfaces. The time series input length is 60 sampling points (corresponding to 10 minutes of time series data). When the reconstruction error of 5 consecutive sampling points exceeds the 3σ dynamic threshold, it is determined that the corresponding interface has a performance abnormality and the self-healing decision process is immediately triggered.

[0036] In this embodiment, the response time of the reconciliation service P99 during the end-of-day reconciliation period suddenly increased from the normal 150ms to 1200ms. The reconstruction error of 8 consecutive sampling points exceeded the threshold, and the system automatically triggered the self-healing decision process.

[0037] Step S22: Generation of the subgraph of anomaly effects Obtain full tracking span data from 5 minutes before the anomaly occurred to the current moment. Using the entry tracking span corresponding to the anomaly reconciliation interface as the endpoint, traverse backwards in the global distributed call chain DAG, remove link nodes without anomaly association, and generate an anomaly impact subgraph containing only the fault propagation path to narrow down the root cause investigation scope.

[0038] Step S23: Identification of critical delay paths In the anomaly impact subgraph, traverse each complete path from the entry tracking span to the leaf tracking span, and calculate the total delay of a single path using the path cumulative time formula: In the formula, For the k-th complete path in the subgraph affected by the anomaly, For tracking span nodes on the path, This represents the node's execution time.

[0039] Calculations identified the core path that contributed the most to latency as reconciliation service entry → reconciliation aggregation logic execution → complex SQL query in the database → database transaction commit. This path accounted for 92% of the total interface response time.

[0040] Step S24: Generation of candidate root cause node set Using the median tracking span time under the same business volume during the same period 7 days before the failure as the historical baseline, tracking span nodes in the path whose time exceeds 3 times the baseline are identified as candidate root cause nodes. In this embodiment, 3 candidate root cause nodes are finally selected, namely: SQL execution span node of reconciliation service database, database transaction commit span node, and account query service dependency call span node, which constitute the candidate root cause node set.

[0041] Step S3: Identification of root cause nodes based on causal contribution Step S31: Causal Direction Identification Candidate root cause nodes The tracking span time variable is denoted as The core service SLO metric (interface P99 response time) is recorded as... Using historical observation data from normal and fault periods, structural causal modeling (SCM) is employed to identify... and The causal direction between them is determined by excluding interfering nodes that are only correlated but have no causal relationship.

[0042] Step S32: Calculation of causal contribution For each candidate root cause node, the backdoor adjustment formula in the do operator calculus is used to estimate its average causal effect on the SLO index, i.e., its causal contribution. The calculation formula is as follows: In the formula, These are the time-consuming observations of the node under its current abnormal state. The median time consumed during the historical normal period is used. In this embodiment, the SQL thread termination event recorded in the historical slow query log is used as an instrumental variable to approximate the expected value under the intervention distribution. Finally, the causal contribution of the three candidate root cause nodes is calculated to be 0.89, 0.08, and 0.03, respectively.

[0043] Step S33: Final Root Cause Identification The candidate root cause node with the largest causal contribution value is selected as the root cause node. In this embodiment, the causal contribution value of the SQL execution span node of the reconciliation service database is 0.89, which is the maximum value. It is finally identified as the root cause node of the failure. The corresponding failure reason is: the complex aggregation query of the end-of-day reconciliation caused the optimizer to select an incorrect execution plan due to the expiration of database statistics, which triggered a slow query, occupied database connection resources, and thus led to the exhaustion of the connection pool.

[0044] Step S4: Definition of self-healing action set and generation of causal action mask vector Step S41, Definition of Self-Healing Action Set Predefine discrete, implementable sets of self-healing actions to construct a complete action space: Each action is specifically defined as follows: : Terminate the abnormal SQL thread corresponding to the root cause span; Switch read request traffic to the read-only replica database endpoint; Dynamically adjust the timeout threshold and concurrent execution limit of the microservice circuit breaker; : Triggers service degradation logic and returns a cached approximate result; Temporarily increase the maximum number of connections in the database connection pool and restart the service instance.

[0045] Step S42: Causal Action Mask Vector Generation Preset operation and maintenance security policies prohibit aggregation : Execution is prohibited during peak trading hours Non-database root cause prohibits execution and Simultaneously, maintain the historical work order knowledge mapping table to identify the set of high-frequency successful actions for the root causes of slow queries. .

[0046] In the anomaly impact subgraph, calculate the set of downstream affected service nodes of the identified root cause node, and generate a causal action mask vector according to the mask value rules. : The final generated mask vector is That is, open , , Three effective actions are taken to block rate limiting and degradation actions that have no causal relationship, thus confining the learning and exploration space within an effective range.

[0047] Step S43: High-frequency action forced release In this embodiment, , Actions that have been successfully performed in the past are forcibly set to 1 according to the rules, ensuring that the policy network prioritizes exploring historically validated actions and reducing trial-and-error costs.

[0048] Step S5: Construction of Reinforcement Learning State Vectors Four key features were extracted, normalized to the [0,1] interval using the min-max method, and then concatenated to form an 18-dimensional state vector for a semi-Markov decision process. The details of each feature dimension are as follows: System resource characteristics (7 dimensions): active connections in the database connection pool, number of waiting connections, maximum number of connections, container CPU utilization, memory utilization, average waiting time for database row locks, and message queue consumption latency; Anomaly topology features (6 dimensions): time taken to trace the root cause node across span P99, year-on-year increase in time taken, out-degree and in-degree of the node in the anomaly impact subgraph, critical path delay ratio, and number of affected downstream services. Causal contribution characteristics (2-dimensional): normalized causal contribution degree and contribution ranking position; Historical action statistical characteristics (3-dimensional): the maximum cosine similarity between the current fault fingerprint and similar faults in historical work orders, the average recovery time of similar fault cases, and the proportion of high-frequency actions used.

[0049] Step S6: Construction of Multi-Objective Immediate Reward Function An immediate reward function is constructed that incorporates SLO recovery, causal alignment, and action risk cost to evaluate the merits of self-healing actions. The formula is as follows: Preset weight hyperparameters , , The calculation rules for each item are as follows: : The percentage decrease in the response time of the entry interface relative to the abnormal peak value after the action is executed, with a value range of [0,1]. : Causal alignment term, when the selected action acts on the direct child node of the root cause node and is consistent with the historical best action, the value is a linear mapping value of the causal contribution, and the maximum value in this embodiment is 1; Action risk costs are preset based on the degree of business impact; in this embodiment... The cost is 1. The cost is 2. The cost is 3. The cost is 4. The cost is 5.

[0050] Step S7: Training the DuelingDQN policy network with action masking Step S71: Network Architecture Setup Construct a dual deep Q-network (DuelingDQN), which includes a shared feature extraction layer, a value stream branch, and a dominance stream branch: Shared feature extraction layer: 2 fully connected layers, each with 64 neurons, using the ReLU activation function; Value Stream Branch: A 1-layer fully connected network with 32 neurons, outputting state value. ; Dominant Flow Branch: A single fully connected network with 32 neurons, outputting the dominance value for each action. ; The final state action value function output formula is: In the formula, The size of the action space is 5 in this embodiment.

[0051] Step S72, Application of Action Masking Mechanism Apply a causal action mask vector to the network output layer, setting the Q-value of invalid actions with a mask component of 0 to 0. (Preset minimum value) enables the policy network to completely ignore invalid actions during the decision-making and exploration process, and strictly adhere to the operation and maintenance security boundary.

[0052] Step S73, Offline Pre-training 100,000 valid state transition samples were extracted from the bank's two-year historical maintenance work order database. Supervised pre-training of the network was performed using demonstration data from maintenance experts. The loss function was the mean square error between the Q-value and the target Q-value. The pre-training consisted of 50 epochs with a batch size of 64 and a learning rate set to... This allows the network to learn the basic handling logic for common faults in advance.

[0053] Step S74: Online fine-tuning In a non-production shadow cluster configured identically to the production environment, simulated injections of slow query failures and connection pool exhaustion failures are performed, with the policy network making online decisions and executing actions. Newly generated state transition samples are stored in a priority experience replay buffer (buffer size 100,000), sampled according to time difference error priority, and network parameters are updated using the Adam optimizer. A discount factor is set. The target network updates every 100 steps. After 20 rounds of online fine-tuning, the network converges, completing the training of the policy network.

[0054] Step S8: Production Environment Deployment and Closed-Loop Self-Healing Execution Step S81, Model Deployment The converged policy network model is encapsulated as a Kubernetes custom resource definition (CRD) controller and deployed as a stateless container group in the cluster operation and maintenance plane namespace, and interfaced with the monitoring system, distributed tracing system, Kubernetes API, and database driver.

[0055] Step S82: Real-time status acquisition When the anomaly detection module sends a fault trigger signal, the controller pulls the corresponding data from the Prometheus, Jaeger, and ELK systems in real time and constructs the current state vector according to the rules in step S5.

[0056] Step S83, Optimal Action Decision The current state vector is input into the trained policy network to complete the forward computation, and the Q-values ​​of each effective action are output. The action with the largest Q-value is selected as the optimal self-healing action. In this embodiment, the policy network outputs... The action with the highest Q value is determined to be the final action to be executed.

[0057] Step S84, Action Execution Execute via database driver call The action terminates the corresponding abnormal SQL execution thread and releases the database connection resources it occupies.

[0058] Step S85: Effect Verification and Closed-Loop Iteration After the action is executed, the SLO index of the core service is continuously monitored, and the preset verification time window is 30 seconds. In this embodiment, 12 seconds after the action is executed, the response time of the reconciliation service interface P99 recovers to 160ms, falling back to the normal baseline level. The system determines that the fault has been recovered and writes the cause-effect graph of this fault, the state sequence, and the successfully executed action into the work order knowledge base to complete this self-healing closed loop.

[0059] If the indicators do not recover to the baseline within the preset time window, the system will re-enter the decision loop, execute a new round of state collection and action decision-making, and store the failure record in the experience base for subsequent model iteration and optimization.

[0060] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A reinforcement learning-based method for IT operations and maintenance anomaly detection and self-healing decision-making, characterized in that, Includes the following steps: Step S1: Collect real-time operation and maintenance data of the microservice system, including distributed call chain tracing span data, system resource indicator data, and database running status data, and construct a directed acyclic graph of the distributed call chain based on the tracing span data; Step S2: Perform anomaly detection on the response time series of the microservice interface. When an anomaly is detected, trigger the self-healing decision process, obtain the tracking span data from before the anomaly occurred to the current time to generate an anomaly impact subgraph, identify the path with the largest delay contribution in the anomaly impact subgraph, and locate the tracking span nodes with significantly higher time consumption than the historical level as the candidate root cause node set. Step S3: For each candidate root cause node in the candidate root cause node set, calculate the causal contribution of the candidate root cause node to the service level target indicator, and select the candidate root cause node with the largest causal contribution as the identified root cause node. Step S4: Define a set of executable self-healing actions. Generate a causal action mask vector based on the downstream affected range of the identified root cause node in the anomaly impact subgraph and the preset operation and maintenance security strategy to limit the effective execution range of the self-healing actions. Step S5: Extract system resource features, abnormal topology features, causal contribution features, and historical action statistics features, and concatenate them to form the state vector of the reinforcement learning semi-Markov decision process. Step S6: Construct an immediate reward function that includes service level target recovery items, causal alignment items, and action risk cost items to evaluate the quality of self-healing actions; Step S7: Use a value network with an action masking mechanism to learn and update the state-action value function, and train to obtain a self-healing decision-making strategy network; Step S8: Deploy the trained policy network in the production environment, respond to abnormal trigger signals in real time, select the optimal self-healing action based on the current state vector and execute it, continuously monitor the service level target indicators after execution, and decide whether to make a new round of decisions based on the recovery situation until the service returns to normal.

2. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Deploy extended Berkeley package filter probes on each worker node of the container orchestration cluster to perform system call tracing on microservice container processes, capture the entry and exit events of Hypertext Transfer Protocol requests and Remote Procedure Call requests, and generate tracing span data that conforms to the Open Telemetry Standard. Step S12: Collect container CPU usage, memory usage, network throughput, database connection count, active transaction count, and row lock wait count using monitoring and alarm tools; collect slow query logs from the database using log collection tools. Step S13: Deploy a distributed tracing system collector to receive tracing span data, aggregate the tracing span data with a preset time window, and construct a distributed call chain directed acyclic graph, where nodes represent tracing span operations and include service name, operation name, time consumption, and status code attributes, directed edges represent direct call relationships between tracing spans, and each edge records the average call time and the number of calls.

3. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, Step S2 includes the following steps: Step S21: Use a variational autoencoder to perform reconstruction probability estimation on the P99 response time series of the microservice interface. When the reconstruction error of a preset number of consecutive sampling points exceeds the dynamic threshold, determine that the corresponding interface has an anomaly and trigger the self-healing decision process. Step S22: Obtain all tracking span data from the first preset time before the exception occurred to the current time. Using the entry tracking span corresponding to the exception interface as the endpoint, traverse in reverse in the directed acyclic graph of the distributed call chain to generate the exception impact subgraph. Step S23: In the anomaly impact subgraph, for each path from the entry tracking span to the leaf tracking span, calculate the cumulative path time. The cumulative path time is used to characterize the total sum of delays of each tracking span node on a single call path, in order to identify the call path that contributes the most to the overall response time. The formula for calculating the cumulative path time is as follows: In the formula, The first subgraph representing the abnormal influence A path from the entry point to the leaf point, tracing the span; Representing a path The tracking span nodes passed through; Indicates tracking span nodes The time taken; Representing a path The cumulative sum of the time spent tracking all nodes across the span; Step S24: Identify the tracking span nodes whose cumulative time is higher than the historical level for the same period as candidate root cause nodes, and form a set of candidate root cause nodes.

4. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Select candidate root cause nodes The tracking span time variable is denoted as The service level target indicator is marked as Using historical observation data from normal and fault periods, variables are identified through a structural causal model. With variables The causal direction between them; Step S32: Calculate candidate root cause nodes Service level target indicators When determining the causal contribution, the backdoor adjustment formula in operator calculus is used to estimate the average causal effect; the average causal effect characterizes the effect on the variable. Under the condition that intervention causes it to take abnormally high values, the variable Expected value and variables variables under the condition of normal median The difference between the expected values ​​is used to quantify candidate root cause nodes. The extent to which the anomaly affects service level objectives; the formula for calculating the average causal effect is as follows: In the formula, Indicates candidate root cause nodes The causal contribution; This represents the interferometric quantifier in operator calculus; Represents the expectation operator; Indicates tracking span Time-consuming observations under the current anomalous state; Indicates tracking span The median time spent during a normal historical period; Step S33: Using the natural intervention events recorded in the historical slow query log as instrumental variables, approximate the expected value under the intervention distribution, and select... The candidate root cause node with the largest value is selected as the root cause node.

5. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: Define the set of self-healing actions, which is represented as follows: In the formula, Represents the space of executable self-healing actions; This indicates the termination of the Structured Query Language thread action corresponding to the root cause tracing span; This indicates an action to switch read request traffic to the read-only replica database endpoint; This indicates actions that dynamically adjust the timeout threshold and concurrent execution limit of the microservice circuit breaker; This indicates an action that triggers service degradation logic and returns a cached approximate result. This indicates an action to temporarily increase the maximum number of connections in the database connection pool and restart the service instance; The self-healing action set is an exhaustive enumeration and formal definition of the recovery operations that the system can take after detecting an anomaly. Its purpose is to provide a clear and discrete action selection space for the reinforcement learning policy network, so that the policy network can make decisions based on state characteristics among a limited number of candidate actions. Step S42: Calculate the set of all downstream affected service nodes of the identified root cause node in the anomaly impact subgraph, and generate a causal action mask vector; the causal action mask vector is used to mask actions that cannot effectively block the fault propagation chain during the reinforcement learning decision-making process, limiting the action exploration space to the operational causal reachability range; causal action mask vector The Middle The rules for the values ​​of each component are as follows: In the formula, Representing state Next The mask component value corresponding to each action is 1, which indicates that the action is valid, and 0, which indicates that the action is invalid. Indicates the identification of root cause nodes The set of downstream affected service nodes in the anomaly impact subgraph; This represents the set of actions that are prohibited from being executed according to the preset operation and maintenance security policy; This represents the relationship between the historical work order knowledge mapping table and the identified root cause node. The set of high-frequency successful actions corresponding to the type; Step S43: Maintain the historical work order knowledge mapping table, record the high-frequency actions of each root cause type in historical successful recovery cases. If an action appears in the high-frequency action list, the mask component corresponding to the action is forcibly set to 1 according to the mask value rule.

6. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, In step S5, the features for constructing the state vector include: System resource characteristics include: active database connection pool connections, waiting connections, maximum number of connections, container CPU utilization, memory utilization, average database row lock wait time, and message queue consumption latency. The dimensions of abnormal topology features include: P99 time for identifying the root cause node and tracing the span, year-on-year increase in time, out-degree and in-degree of the root cause node in the abnormal impact subgraph, critical path delay ratio, and number of affected downstream services. The causal contribution feature dimensions include: normalized causal contribution degree and contribution ranking position; The statistical features of historical actions include: the maximum cosine similarity between the current fault fingerprint and similar faults in historical work orders, the average recovery time of similar fault cases, and the proportion of high-frequency actions used.

7. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, In step S6, the immediate reward function is used to evaluate the degree of improvement in the system state after the self-healing action is executed. It is calculated by a weighted combination of three parts: the recovery magnitude of the service level target, the degree of alignment of the causal path, and the risk cost of the action execution. The formula for calculating the immediate reward function is as follows: In the formula, express The instant reward value gained after performing a self-healing action at any time; This represents the percentage decrease in the response time of the entry interface after the action is performed, relative to the abnormal peak value. The value ranges from zero to one. Indicates causal alignment terms, when the selected action... The action applies to identifying the root cause node as a direct child node in the cause-effect graph, and the action... When the optimal action is consistent with the corresponding root cause type in the historical work order, the causal alignment term takes the value of a linear mapping of the causal contribution. Indicates action The corresponding risk costs are pre-set based on the degree of negative impact the action may have on the business; , , These are preset weight hyperparameters used to adjust the relative importance of the three components in the reward calculation.

8. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, Step S7 includes the following steps: Step S71: Construct a dual deep Q-network, including a shared feature extraction layer, a value stream branch, and a dominance stream branch; the state-action value function is used to estimate the value of actions in the state. Select action The expected cumulative reward can be represented by a combination of value stream and advantage stream, separating the value of the state itself from the relative merits of each action; the formula for calculating the output of the state-action value function is as follows: In the formula, Indicates the state Select action Expected cumulative return; Indicates only with state Related value functions; Indicates the state Take action below The superiority or inferiority functions; Indicates the size of the action space; This is an index variable used to iterate through all actions in the action space; Step S72: Apply the causal action mask vector to the output layer, set the Q value corresponding to the invalid action of the mask component to the preset minimum value, so that the policy ignores invalid actions during the selection or exploration process; Step S73: Offline pre-training stage, extract state transition samples from the historical operation and maintenance work order database, and use expert demonstration data to perform supervised pre-training on the network. The loss function is the mean square error between the Q value and the target Q value. Step S74: In the online fine-tuning stage, fault injection is simulated in the shadow cluster of the non-production environment. The policy network makes online decisions and executes actions. The newly generated transfer samples are stored in the priority experience replay buffer. The network parameters are updated by sampling according to the time difference error priority and using the adaptive moment estimation optimizer.

9. The IT operation and maintenance anomaly detection and self-healing decision-making method based on reinforcement learning according to claim 1, characterized in that, Step S8 includes the following steps: Step S81: Encapsulate the trained and converged policy network model into a container orchestration custom resource definition controller, and deploy it in the operation and maintenance plane namespace in the form of container group deployment; Step S82: After the anomaly detection module sends a trigger signal, it pulls data from the monitoring and alarm tool and the distributed tracing system in real time to construct the current state vector; Step S83: Input the current state vector into the policy network for forward computation and select the self-healing action corresponding to the maximum Q value; Step S84: Execute the selected self-healing action by calling the container orchestration application interface, database driver, or configuration center interface; Step S85: Continuously monitor the service level target indicators. If the indicators do not recover to the baseline level within the preset time window, re-enter the decision loop and store the failure record in the experience base. If the indicators recover to normal, write the fault cause-effect graph, state sequence and successful action into the work order knowledge base.

Images