Reinforcement learning-based fault recovery method and device, equipment, medium and product

Abstract

The invention relates to a fault recovery method and device based on reinforcement learning, equipment, a storage medium and a product. The method comprises the following steps: acquiring operation monitoring data and system state diagrams of a plurality of service instances; wherein graph nodes of the system state graph are all the service instances, and graph edges of the system state graph are determined according to calling relations among all the service instances; updating the system state diagram according to the operation monitoring data, and encoding the updated system state diagram through a diagram neural network to obtain a system state encoding vector; inputting the system state coding vector into a reinforcement learning model to obtain a fault recovery operation path, and executing fault recovery according to the fault recovery operation path; the reinforcement learning model is obtained by training in a simulated fault environment according to a multi-target reward function. By adopting the method, the fault recovery operation path selection accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a fault recovery method, apparatus, computer device, computer-readable storage medium, and computer program product based on reinforcement learning. Background Technology

[0002] Currently, cloud computing environment fault recovery methods mainly rely on predefined static scripts and fixed recovery plans. When a system failure occurs, operations and maintenance personnel or automated systems will strictly follow these preset procedures to perform recovery operations.

[0003] However, this traditional approach has significant limitations when facing complex and ever-changing cloud computing environments. Due to the high dynamism and complexity of cloud computing systems, there are intricate dependencies between service instances, and failures in different services can have cascading effects. Static contingency plans cannot detect changes in the current state of the system and the actual dependencies between services in real time.

[0004] When a failure occurs, the relevant methods cannot dynamically adjust the recovery strategy according to the real-time status of the system. They often adopt a one-size-fits-all recovery solution, which leads to inaccurate determination of the recovery path and may result in problems such as excessive recovery time, waste of resources, or even the introduction of new failure risks. Summary of the Invention

[0005] Therefore, it is necessary to provide a reinforcement learning-based fault recovery method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of fault recovery operation path selection, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a fault recovery method based on reinforcement learning, comprising:

[0007] Obtain runtime monitoring data and system status graphs for multiple service instances; wherein, the graph nodes of the system status graph are each of the service instances, and the graph edges of the system status graph are determined based on the calling relationships between the service instances;

[0008] The system state diagram is updated based on the operation monitoring data, and the updated system state diagram is encoded using a graph neural network to obtain a system state encoding vector.

[0009] The system state encoding vector is input into the reinforcement learning model to obtain the fault recovery operation path, and fault recovery is performed according to the fault recovery operation path; the reinforcement learning model is trained in a simulated fault environment based on a multi-objective reward function.

[0010] In one embodiment, the training method of the reinforcement learning model includes:

[0011] Construct a simulation environment with the same topology as the production environment, and deploy service instance replicas corresponding to the production environment in the simulation environment;

[0012] A preset fault scenario is added to the simulation environment to obtain a simulated fault environment; the preset fault scenario includes one or more of the following: service instance stoppage, network connection interruption, and resource exhaustion.

[0013] Obtain the operation monitoring data of the simulated fault environment, and generate the simulated state encoding vector corresponding to the operation monitoring data;

[0014] Based on the multi-objective reward function and the simulated state encoding vector, a reinforcement learning agent is trained in the simulated fault environment to obtain a reinforcement learning model.

[0015] In one embodiment, training the reinforcement learning agent based on the multi-objective reward function and the simulated state encoding vector to obtain the reinforcement learning model includes:

[0016] The agent is initialized in the simulated fault environment; the agent includes a policy network and a value network.

[0017] Based on the simulated state encoding vector, a recovery action is determined through the policy network, and the recovery action is executed in the simulated fault environment to obtain a new simulated state encoding vector.

[0018] Based on the simulated state encoding vectors before and after the recovery action, the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score are calculated respectively through the multi-objective establishment function to obtain immediate reward feedback;

[0019] The process iteratively executes the step of determining the recovery action through the policy network based on the simulated state encoding vector, calculates value feedback data through the value network based on the immediate reward feedback, and trains the reinforcement learning agent based on the value feedback data to obtain a reinforcement learning model.

[0020] In one embodiment, the step of calculating value feedback data through the value network based on the instant reward feedback includes:

[0021] Based on the type of the preset fault scenario in the simulated fault environment, the reward weights corresponding to the business value recovery score, the recovery time penalty score, the data loss penalty score, and the resource consumption penalty score are determined respectively.

[0022] A comprehensive reward feedback is calculated based on the business value recovery score, the recovery time penalty score, the data loss penalty score, the resource consumption penalty score, and the corresponding reward weight;

[0023] The comprehensive reward feedback from each iteration is input into the value network to obtain value feedback data.

[0024] In one embodiment, the method further includes:

[0025] Acquire fault recovery data from the production environment; the fault recovery data includes at least one of the following: system state encoding vector at the time of the fault occurrence, recovery action sequence, and recovery effect evaluation result;

[0026] The fault recovery data is scored for quality; the quality score is calculated based on at least one of recovery success rate, recovery time, and business impact.

[0027] When the quality score exceeds a preset update threshold, the fault recovery data is added to the training dataset; the training dataset is used to train the reinforcement learning model.

[0028] In one embodiment, the reinforcement learning model includes a production version model and multiple candidate version models, the candidate version models being deployed in an isolated verification environment; after performing fault recovery according to the fault recovery operation path, the model further includes:

[0029] Based on the system state encoding vector, historical fault data is generated, and the environmental parameters of the isolation verification environment are adjusted based on the historical fault data.

[0030] The historical fault data is input into each of the candidate version models to obtain candidate recovery operation paths, and fault recovery is performed in the isolation verification environment according to the candidate recovery operation paths to obtain recovery performance data;

[0031] Based on the recovery performance data, calculate the performance improvement metrics for each candidate version model;

[0032] If the performance improvement index exceeds the preset deployment threshold, the corresponding candidate version model will be deployed to the production environment to replace the production version model as the new production version model.

[0033] Secondly, this application also provides a fault recovery device based on reinforcement learning, comprising:

[0034] The data acquisition module is used to acquire the operation monitoring data and system status graph of multiple service instances; wherein, the graph nodes of the system status graph are each of the service instances, and the graph edges of the system status graph are determined according to the calling relationship between the service instances;

[0035] The data processing module is used to update the system state diagram based on the operation monitoring data, and to encode the updated system state diagram through a graph neural network to obtain a system state encoding vector.

[0036] The fault recovery module is used to input the system state encoding vector into the reinforcement learning model to obtain the fault recovery operation path, and to perform fault recovery according to the fault recovery operation path; the reinforcement learning model is trained in a simulated fault environment based on a multi-objective reward function.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described in the first aspect.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps described in the first aspect.

[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps described in the first aspect.

[0040] The aforementioned reinforcement learning-based fault recovery method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire operational monitoring data and system state diagrams from multiple service instances. The graph nodes of the system state diagram represent each service instance, and the graph edges are determined based on the calling relationships between these service instances, thus solving the problem of existing methods failing to accurately perceive service dependencies. The system state diagram is updated based on the operational monitoring data, and encoded using a graph neural network to obtain a system state encoding vector. This ensures that the state diagram reflects the dynamic changes of the system, overcoming the limitation of static contingency plans failing to adapt to system state changes. The system state encoding vector is input into a reinforcement learning model to obtain a fault recovery operation path, and fault recovery is executed according to this path. The reinforcement learning model is trained in a simulated fault environment based on a multi-objective reward function, enabling the generation of targeted fault recovery operation paths according to the specific state of the current system. This allows the recovery path to be dynamically adjusted based on actual conditions, improving the accuracy of fault recovery operation path selection. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a diagram illustrating the application environment of a reinforcement learning-based fault recovery method in one embodiment.

[0043] Figure 2 This is a flowchart illustrating a reinforcement learning-based fault recovery method in one embodiment.

[0044] Figure 3 This is a flowchart illustrating the training method of a reinforcement learning model in one embodiment;

[0045] Figure 4 This is an overall architecture diagram of a reinforcement learning-based fault recovery method in one embodiment;

[0046] Figure 5 This is a flowchart illustrating the fault recovery method based on reinforcement learning in one embodiment.

[0047] Figure 6 This is a structural block diagram of a reinforcement learning-based fault recovery device in one embodiment;

[0048] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0051] The reinforcement learning-based fault recovery method provided in this application can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0052] In one exemplary embodiment, such as Figure 2 As shown, a fault recovery method based on reinforcement learning is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S202 to S206. Wherein:

[0053] Step S202: Obtain the running monitoring data and system status diagram of multiple service instances.

[0054] The system state diagram is a directed graph data structure used to describe all service instances and their interrelationships in the entire cloud computing environment. The graph nodes of the system state diagram are each service instance, and the graph edges are determined according to the calling relationships between each service instance. The graph edges can represent the dependency calling relationships between service instances, the direction of the edge indicates the direction of the call, and the weight of the edge can reflect the frequency or importance of the call.

[0055] Among them, operation monitoring data refers to multi-dimensional data reflecting the running status of service instances that are collected in real time through various monitoring tools and agent components. Specifically, it can include basic performance indicators such as CPU (Central Processing Unit) utilization, memory usage, disk I / O throughput, and network bandwidth utilization, as well as business-level indicators such as service response time, request processing speed, error rate, and number of concurrent connections, and call chain indicators such as inter-service call latency, call frequency, and call success rate obtained through link tracing technology.

[0056] A service instance refers to the actual deployment unit of a specific application service running in a cloud computing environment, such as a container instance of a user authentication service, a virtual machine instance of an order processing service, or a master-slave node instance of a database service.

[0057] For example, server 104 can collect runtime monitoring data through monitoring agent components deployed on various service instances. These monitoring agent components may include a Prometheus agent for collecting performance metrics, a Jaeger agent for collecting call chain data, and various log collectors for obtaining application runtime logs. Server 104 can uniformly collect and initially organize this heterogeneous monitoring data from different data sources, ensuring data format consistency and timestamp synchronization.

[0058] In some embodiments, server 104 can statically construct an initial system state graph by analyzing service configuration files, container orchestration files, and cloud platform API (Application Programming Interface) call records. Graph nodes are determined by traversing all deployed service instances, and each node may contain attribute information such as a unique identifier for the service instance, service type, and deployment location. Graph edges can be determined by analyzing configuration dependencies between services, network routing rules, and load balancer configurations to determine call relationships. When constructing graph edges, server 104 can use the service instance initiating the call as the starting point of the edge and the called service instance as the ending point, forming directed edges, and set the initial weight of the edges according to the importance level defined in the configuration file or the historical call frequency.

[0059] For example, server 104 can also employ a dynamic discovery method to construct the system state graph, that is, dynamically identifying dependencies between services by analyzing call chain tracing information in real-time operational monitoring data. Specifically, server 104 can parse call trajectory data collected by link tracing systems such as Jaeger, and identify call relationships by analyzing the source service identifier and target service identifier of each service call. When it is detected that service A frequently calls service B, a directed edge from node A to node B is created in the system state graph. Server 104 can also combine time window sliding analysis to discover implicit dependencies by observing changes in call patterns within a certain time window. For example, if the error rate of service B also increases significantly after a fixed delay when the response time of service A increases, it can be inferred that A has an indirect influence dependency on B, and server 104 can establish corresponding dependency edges in the state graph accordingly. During the dynamic construction process, server 104 can update the graph topology in real time, adding new edges when new service call relationships are discovered, and removing corresponding edges when certain call relationships no longer occur for a long time, ensuring that the state graph always reflects the true dependencies of the current system.

[0060] Step S204: Update the system state diagram based on the operation monitoring data, and encode the updated system state diagram using a graph neural network to obtain the system state encoding vector.

[0061] Among them, Graph Neural Network (GNN) is a deep learning model used to process graph-structured data, which can learn the feature representation of nodes by aggregating information about their neighbors. The system state encoding vector can refer to a fixed-dimensional vector representation obtained by performing feature extraction and dimensionality reduction on the entire system state graph through a graph neural network. This vector contains global information and local topological features of the current system state.

[0062] For example, server 104 can update the attribute information of each node and edge in the system state graph based on runtime monitoring data. This update process involves mapping real-time monitoring data to corresponding elements in the graph structure. For updating graph nodes, server 104 can use the latest runtime monitoring data of each service instance as the feature vector of that node. This includes standardizing performance metrics such as CPU utilization, memory usage, IO (Input / Output) traffic, and disk utilization and assigning them to the corresponding node. It can also calculate some derived metrics such as request success rate, average response time, and error frequency, which are business-level health metrics.

[0063] When updating node features, server 104 can use a sliding time window technique to calculate statistical features at different time scales, such as calculating the mean, maximum, and standard deviation of various indicators over the past 5, 15, and 60 minutes, to capture the temporal trend of service instance status changes. For updating graph edges, server 104 can update the edge weights and attributes based on call chain monitoring data, updating information such as service call frequency, average call latency, and call success rate to the features of the corresponding edge. When the call relationship represented by an edge is inactive during the monitoring period, server 104 can correspondingly reduce the weight of that edge or mark it as inactive.

[0064] In this embodiment, server 104 can employ a Graph Attention Network (GAT) to encode the updated system state graph. Server 104 can input the updated node feature vectors into the input layer of the GAT network. This network uses a multi-head attention mechanism to calculate the attention weights of each node to its neighboring nodes, enabling the network to adaptively focus on the most important neighbor information. During the attention weight calculation process, server 104 can consider the correlation between the node's own feature vector and the feature vectors of its neighboring nodes, determining the importance of different neighboring node information through the learned parameter matrix. Server 104 can propagate layer by layer through the multi-layered GAT network, ensuring that each node obtains a rich feature representation containing its multi-hop neighbor information. After obtaining the updated feature representations of all nodes, server 104 can use graph pooling to aggregate the feature information of all nodes into a global system state encoding vector. Specifically, this can be achieved by using average pooling, max pooling, or attention pooling to convert the variable-length node feature sequence into a fixed-dimensional vector representation.

[0065] Furthermore, during the graph neural network encoding process, server 104 can employ an improved graph attention network (GAT) architecture, introducing a time decay attention mechanism to handle the timeliness characteristics of the system state. Traditional graph attention networks only consider the similarity of node features when calculating attention weights between nodes, while server 104's improved scheme can also consider the freshness of the monitoring data. Specifically, server 104 can assign a time decay coefficient to the influence of each neighboring node during the attention weight calculation process. This coefficient is calculated based on the difference between the timestamp of the neighboring node's monitoring data and the current time. Data closer to the current time has a decay coefficient closer to 1, while data further away has a decay coefficient that decreases exponentially. Server 104 can apply this time decay weight to each attention head in the multi-head attention mechanism, allowing the graph neural network to focus more on the latest system changes when aggregating neighbor information, while giving less attention to older state information. Through this time-aware attention mechanism, the system state encoding vector generated by server 104 can more accurately reflect the current state trend of the system, improving the timeliness and accuracy of fault detection and recovery decisions.

[0066] For example, server 104 can also employ an improved graph attention network, which can introduce a time decay attention mechanism to better handle the time-sensitivity characteristics of the system state. When calculating attention weights, server 104 can consider the freshness of neighboring node information; neighboring node information further removed from the current time can be assigned exponentially decaying weights, ensuring that the graph neural network pays more attention to recent changes in the system state. Specifically, server 104 can introduce a time decay factor into the attention weight calculation formula. This factor calculates the decay level based on the difference between the timestamp of the monitored data and the current time, so that the latest monitored data has the highest attention weight, while older data has a gradually decreasing weight. Server 104 can also integrate node embedding technology into the graph neural network, using pre-trained node embedding vectors to enhance the semantic expressive power of node features, enabling service instances with similar functions to have closer representations in the feature space. During the encoding process, server 104 can dynamically adjust the number of network layers and hidden units, optimizing the encoding effect according to the size and complexity of the current system state graph, ensuring that the generated system state encoding vector maintains sufficient information and has good generalization ability.

[0067] Through the above steps, server 104 realizes the intelligent transformation from raw monitoring data to structured state representation. The generated system state encoding vector not only contains the real-time running status information of each service instance, but also integrates the topological features of complex dependencies between services, solving the problem that traditional methods cannot effectively process complex graph structure data and cannot capture the global state features of the system.

[0068] Step S206: Input the system state encoding vector into the reinforcement learning model to obtain the fault recovery operation path, and perform fault recovery according to the fault recovery operation path.

[0069] The reinforcement learning model is used to generate the optimal fault recovery operation path based on the current system state. A fault recovery operation path is an ordered sequence of recovery actions, each representing a specific recovery operation, such as restarting a service, switching traffic, or scaling up resources. The reinforcement learning model is trained in a simulated fault environment using a multi-objective reward function.

[0070] For example, server 104 can pass the system state encoding vector as input data to the currently deployed production version reinforcement learning model. This model can perform forward computation on the input state vector through its internal neural network structure. After receiving the system state encoding vector, the reinforcement learning model can analyze the current system state characteristics through a multi-layer fully connected network or a recurrent neural network (RNN) to identify abnormal patterns and fault characteristics in the system. The model's policy network can calculate the execution probability of various possible recovery actions based on the learned policy parameters, and then select the action with the highest probability as the optimal recovery action at the current moment. Server 104 can repeatedly call the reinforcement learning model to gradually generate a complete fault recovery operation path. Each time the model outputs a recovery action, server 104 can simulate the execution of the action and update the system state, and then input the new state back into the model to obtain the next action, until the model determines that the system has recovered normally or the preset maximum number of steps has been reached. After obtaining the complete fault recovery operation path, server 104 can execute these recovery operations one by one in the actual production environment according to the order of the actions in the path. After each action is executed, the system state changes are monitored to ensure that the operation takes effect as expected.

[0071] In some embodiments, server 104 can maintain a model management system that includes a production version model and multiple candidate version models. The reinforcement learning model may include a production version model and multiple candidate version models, with the candidate version models deployed in an isolated verification environment. The production version model refers to the reinforcement learning model version currently used in the production environment, responsible for handling real-world fault recovery tasks. Candidate version models refer to model versions trained based on new training data or improved algorithms, which need to be verified in an isolated environment before being deployed to the production environment.

[0072] For example, server 104 can also acquire fault recovery data from the production environment; the fault recovery data includes at least one of the system state encoding vector at the time of the fault occurrence, the recovery action sequence, and the recovery effect evaluation result; the fault recovery data is scored for quality; the quality score is calculated based on at least one of recovery success rate, recovery time, and business impact; when the quality score exceeds a preset update threshold, the fault recovery data is added to the training dataset; the training dataset is used to train the reinforcement learning model. Server 104 can periodically collect fault recovery data from the production environment, which may include the system state encoding vector captured at the time of the fault occurrence, the recovery action sequence generated by the model, and the effect evaluation result after the recovery operation is executed. Server 104 can score the collected fault recovery data for quality, which comprehensively considers the recovery success rate, i.e., whether the fault is completely resolved, the recovery time, i.e., the time spent from fault detection to complete recovery, and the business impact, i.e., the scope and severity of the impact on business indicators during the fault. When the quality score of a fault recovery case exceeds a preset update threshold, server 104 can add the relevant data of that case to the training dataset for subsequent incremental model training and optimization.

[0073] For example, in the dynamic path replanning mechanism of online inference, server 104 implements a real-time decision adjustment system based on state feedback, which can dynamically correct subsequent recovery paths according to the latest system state changes during the execution of recovery operations. After each recovery action is completed, server 104 immediately re-collects system operation monitoring data and generates an updated system state encoding vector according to step S204. This vector is then compared and analyzed with the state encoding vector before execution to evaluate whether the actual execution effect of the current recovery action meets expectations. When server 104 detects that the execution result deviates from the expected goal, such as a recovery action failing to effectively improve the health status of the target service or unexpectedly introducing new abnormal indicators, it can trigger the path replanning process. During the replanning process, server 104 re-inputs the updated system state encoding vector into the reinforcement learning model. Simultaneously, it can input the executed recovery actions and their effects as constraints into the model to avoid the model repeatedly selecting actions that have been proven ineffective. Server 104's reinforcement learning model recalculates the optimal recovery strategy based on the new state information and execution constraints, generating an alternative sequence of recovery actions to address the current situation. To prevent frequent path adjustments from causing system instability, server 104 can also set a replanning trigger threshold and cooldown time. A new replanning process will only be started when the state deviation exceeds the preset threshold and the time since the last replanning exceeds the minimum interval, ensuring the stability and continuity of system decision-making.

[0074] Furthermore, after performing fault recovery according to the fault recovery operation path, server 104 can also generate historical fault data based on the system state encoding vector, and adjust the environment parameters of the isolation verification environment based on the historical fault data; input the historical fault data into each candidate version model to obtain candidate recovery operation paths, and perform fault recovery in the isolation verification environment according to the candidate recovery operation paths to obtain recovery performance data; calculate the performance improvement index of each candidate version model based on the recovery performance data; if the performance improvement index exceeds the preset deployment threshold, deploy the corresponding candidate version model to the production environment to replace the production version model as the new production version model. Server 104 can dynamically adjust the environment parameters of the isolation verification environment based on the generated historical fault data, including adjusting the configuration of service instances, network latency settings, resource limits, etc., to ensure that the verification environment can accurately reproduce various fault scenarios in the production environment. After the verification environment is configured, server 104 can input historical fault data into each candidate version model, allowing each candidate model to generate its own candidate recovery operation path based on the same fault scenario. Server 104 can perform fault recovery simulations in the isolated verification environment according to each candidate recovery operation path, recording recovery performance data for each candidate model, including simulation recovery time, recovery success rate, resource consumption, and other indicators. Based on the collected recovery performance data, Server 104 can calculate the performance improvement indicators of each candidate version model relative to the current production version model, which may include the percentage improvement in recovery efficiency, the degree of improvement in success rate, and the optimization effect of resource utilization. When the performance improvement indicators of a candidate version model all exceed the preset deployment threshold, Server 104 can deploy the candidate version model to the production environment, replacing the current production version model as the new production version model. At the same time, the replaced model version is retained as a rollback backup to ensure that a rapid recovery to a stable version can be achieved if problems occur with the new model.

[0075] Through the above steps, server 104 realizes intelligent fault recovery decision-making based on artificial intelligence. It can dynamically generate the optimal recovery strategy according to the real-time status of the system. At the same time, through continuous model updates and verification mechanisms, it ensures the continuous optimization of the recovery strategy, which solves the fundamental problem that traditional static plans cannot adapt to complex dynamic environments, and significantly improves the intelligence level and recovery effect of fault recovery.

[0076] In the aforementioned reinforcement learning-based fault recovery method, operational monitoring data and a system state graph of multiple service instances are acquired. The graph nodes of the system state graph represent each service instance, and the graph edges are determined based on the calling relationships between these service instances, thus solving the problem of existing methods failing to accurately perceive service dependencies. The system state graph is updated based on the operational monitoring data, and the updated system state graph is encoded using a graph neural network to obtain a system state encoding vector. This ensures that the state graph reflects the dynamic changes of the system, overcoming the limitation of static contingency plans being unable to adapt to system state changes. The system state encoding vector is input into a reinforcement learning model to obtain a fault recovery operation path, and fault recovery is executed according to this path. The reinforcement learning model is trained in a simulated fault environment based on a multi-objective reward function. It can generate targeted fault recovery operation paths based on the specific state of the current system, allowing the recovery path to be dynamically adjusted according to the actual situation, thus improving the accuracy of fault recovery operation path selection.

[0077] In one exemplary embodiment, such as Figure 3 As shown, the training method for the reinforcement learning model includes steps S302 to S306. Wherein:

[0078] Step S302: Construct a simulation environment with the same topology as the production environment, deploy service instance replicas corresponding to the production environment in the simulation environment, and add preset fault scenarios to the simulation environment to obtain a simulated fault environment.

[0079] The preset fault scenarios include one or more of the following: service instance stoppage, network connection interruption, and resource exhaustion. A service instance replica refers to a test version deployed in a simulated environment that has the same functionality as the production environment service instance but with anonymized data; it can include application service containers, database instances, middleware components, etc. Preset fault scenarios refer to various fault types and combinations predefined based on historical fault data and expert experience, which can be used to provide rich learning samples for reinforcement learning training.

[0080] For example, server 104 can obtain a complete environment topology by analyzing the deployment configuration files, container orchestration files, and infrastructure configuration information of the production environment, including the deployment location of each service instance, network connection relationships, resource allocation, load balancing configuration, and other information. Based on the obtained topology information, server 104 can use container technologies such as Docker or Kubernetes in an isolated computing environment to rebuild the same service architecture, ensuring that the number, type, and deployment method of service instances in the simulated environment completely correspond to the production environment. When deploying service instance replicas, server 104 can use the same service image version and configuration parameters as the production environment, but replace the database connection, external API (Application Programming Interface) calls, file storage paths, and other configurations with the corresponding resources in the test environment, and anonymize sensitive data or use simulated data to replace it. Server 104 can also deploy a traffic generator in the simulated environment, which simulates real business access traffic based on historical traffic patterns and user behavior data, including traffic fluctuations at different time periods, sudden traffic bursts, and request patterns for various business scenarios.

[0081] In some embodiments, server 104 can employ layered fault injection to construct preset fault scenarios, which may include various types such as single point of failure, cascading failure, and performance degradation failure. For service instance shutdown faults, server 104 can randomly select one or more service instances for forced shutdown, simulating service crashes, abnormal process exits, container restart failures, and other situations. For network connection interruption faults, server 104 can simulate network problems such as network partitions, network device failures, and bandwidth limitations by modifying network routing rules, disconnecting network connections, increasing network latency, or increasing packet loss rate. For resource exhaustion faults, server 104 can limit the CPU, memory, disk I / O (Input / Output), or network bandwidth resources of certain service instances, simulating resource-related problems such as resource contention, memory leaks, and insufficient disk space.

[0082] For example, server 104 can also integrate chaos engineering tools such as Chaos Mesh to achieve more complex and realistic fault injection. Server 104 can precisely control parameters such as the injection time, duration, and scope of impact of faults through the Chaos Mesh API interface, achieving fine-grained management of fault scenarios. Server 104 can automatically discover typical fault patterns using clustering algorithms based on historical fault data. For example, by analyzing the system state characteristics at the time of a fault, it can identify a combined pattern of database connection pool exhaustion and memory leaks, or a cascading fault pattern of network partitioning and load balancing failure, and convert these discovered fault fingerprints into reproducible preset fault scenarios. Server 104 can also implement a dynamic scheduling mechanism for fault scenarios, adaptively adjusting the frequency and complexity of fault injection based on training progress and model learning effects, ensuring that the reinforcement learning agent can progressively learn to handle various fault situations from simple to complex.

[0083] Step S304: Obtain the operation monitoring data of the simulated fault environment and generate the simulated state encoding vector corresponding to the operation monitoring data.

[0084] Among them, operational monitoring data refers to the operational status data of each component of the system after fault injection in the simulated fault environment. It may include abnormal indicators and status change information of each service instance under the influence of the fault. The simulated state encoding vector is a vector representation obtained by extracting features of the simulated fault environment state through the same graph neural network encoding method as the production environment in the above steps. It can be used as the state input in the training phase.

[0085] For example, server 104 can deploy the same monitoring system components as the production environment in the simulated fault environment, including the Prometheus monitoring agent, Jaeger tracing component, and various log collectors, to ensure that operational monitoring data is collected in a manner consistent with the production environment. Once the fault scenario is injected into the simulated environment, server 104 can immediately begin collecting operational monitoring data from each service instance affected by the fault, reflecting the real-time changes in the system state after the fault occurs. The collected data includes abnormal indicators such as abnormal fluctuations in CPU utilization, sudden increases in memory usage, decreases in network connections, prolonged response times, and increased error rates for each service instance under the influence of the fault, as well as changes in inter-service call relationships, such as interruptions in certain call chains and abnormal call frequencies, indicating topology changes. Server 104 can continuously collect this operational monitoring data until the fault scenario ends or the system state stabilizes, ensuring the capture of complete fault lifecycle information.

[0086] In some embodiments, server 104 may use the exact same processing method as step S204 to generate simulated state encoding vectors to ensure consistency of data representation between the training and inference phases.

[0087] Furthermore, server 104 can also apply time series analysis methods during the generation of simulated state encoding vectors, considering not only the instantaneous state at the time of the fault occurrence but also the trend of state changes within a time window before and after the fault. Server 104 can calculate system state encoding vectors for the 30 minutes before fault injection, the moment of fault occurrence, and various time points after the fault occurrence, forming a state change sequence. This allows the reinforcement learning model to learn the dynamic process of fault development, rather than just a static snapshot of the fault state. Server 104 can also optimize and adjust the state encoding process according to different types of fault scenarios. For example, for network faults, it can enhance the weight of network topology features; for resource faults, it can highlight the importance of resource usage indicators; and for service faults, it can emphasize the changing characteristics of service call relationships.

[0088] Step S306: Train the reinforcement learning agent in a simulated fault environment based on the multi-objective reward function and the simulated state encoding vector to obtain the reinforcement learning model.

[0089] In this context, a reinforcement learning agent refers to an artificial intelligence program that optimizes its decision-making strategy through trial and error in an environment. It can consist of a policy network and a value network. The policy network is used to select the optimal action based on the current state, while the value network is used to evaluate the long-term value of the current state. A multi-objective reward function is a reward calculation method that comprehensively considers multiple business objectives. It can guide the agent to learn the optimal strategy by quantifying the benefits and costs in different dimensions.

[0090] For example, server 104 can initialize a reinforcement learning agent in a simulated fault environment. This agent can adopt an Actor-Critic architecture, where the policy network acts as the Actor, responsible for decision-making action selection, and the value network acts as the Critic, responsible for evaluating state value. The policy network receives the simulated state encoding vector as input and processes the state features through a multi-layer fully connected neural network or a Long Short-Term Memory (LSTM) network, ultimately outputting the execution probability distribution of various possible recovery actions. The value network can also receive the simulated state encoding vector as input and outputs an estimate of the long-term value of the current state through a similar network structure. At the start of training, server 104 can randomly initialize the parameters of these two networks and set appropriate hyperparameters such as the learning rate, number of network layers, and number of hidden units.

[0091] For example, server 104 can initialize an agent in a simulated fault environment; the agent includes a policy network and a value network; based on the simulated state encoding vector, the policy network determines the recovery action, and the recovery action is executed in the simulated fault environment to obtain a new simulated state encoding vector; based on the simulated state encoding vectors before and after the recovery action is executed, a multi-objective establishment function is used to calculate the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score to obtain immediate reward feedback; the steps of determining the recovery action based on the simulated state encoding vector through the policy network are iteratively executed, the value feedback data is calculated through the value network based on the immediate reward feedback, and the reinforcement learning agent is trained based on the value feedback data to obtain a reinforcement learning model.

[0092] In the immediate reward feedback: the business value recovery score can be quantified by assigning preset business criticality weights to different services, APIs, or business processes, with core business recovery having a higher weight; the recovery time penalty score can be the time spent on a single action or the total time spent to reach a phased recovery target, and can adopt a non-linear penalty, with the penalty increasing exponentially as the recovery time increases, incentivizing the agent to recover quickly; the data loss penalty score can be quantified by combining the PPO (Proximal Policy Optimization) metric and the current data synchronization latency; the resource consumption penalty score can be quantified based on the additional consumption data generated by expansion, switching regions, and starting backup resources, incentivizing the agent to choose a cost-effective recovery path; the immediate reward feedback can also include an operational error score as a penalty for invalid operations, repeated operations, or the introduction of new faults.

[0093] Furthermore, when calculating the data loss penalty score, server 104 can combine RPO (Recovery Point Objective) to accurately quantify the risk of data loss. Server 104 can first obtain the preset RPO threshold for each service instance. This threshold represents the maximum data loss time window that the business can tolerate. For example, the RPO threshold for the core transaction service is set to 15 seconds, and the RPO threshold for the ordinary log service is set to 5 minutes. After the recovery action is executed, server 104 can detect the current data synchronization latency and calculate the actual data lag time by comparing indicators such as the last synchronization timestamp of the master and slave databases, the consumption lag of the message queue, and the replica consistency status of the distributed storage. Server 104 can use a segmented penalty function to calculate the data loss penalty score. When the data lag time exceeds the RPO threshold of the corresponding service, the penalty score will increase sharply. The specific data loss penalty score can be calculated based on the maximum value function of each service instance. Through this RPO-sensitive penalty mechanism, server 104 can train the agent to prioritize recovery strategies that maximize the protection of data integrity.

[0094] In some embodiments, server 104 can employ the Proximal Policy Optimization (PPO) algorithm to train the reinforcement learning agent. In each training round, server 104 inputs the current simulated state encoding vector into the policy network. The policy network calculates the selection probability of various recovery actions based on the learned policy parameters, and then selects a specific recovery action to execute in the simulated environment through sampling. After executing the recovery action, the simulated environment undergoes a corresponding state change. Server 104 can collect new operational monitoring data and generate a new simulated state encoding vector. Based on the changes in the simulated state encoding vector before and after the recovery action, server 104 can calculate the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score using a multi-objective reward function. These scores are then weighted and summed according to preset weights to obtain immediate reward feedback. Server 104 can store the immediate reward feedback and state transition information in an experience buffer. When sufficient experience data is collected, the value network is used to calculate the value estimate for each state, and the dominance function is calculated using a temporal difference method. Finally, the pruning objective function of the PPO algorithm is used to update the parameters of the policy network and the value network.

[0095] Furthermore, server 104 can determine the reward weights corresponding to the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score according to the type of preset failure scenario in the simulated failure environment; calculate the comprehensive reward feedback based on the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score and their corresponding reward weights; and input the comprehensive reward feedback of each iteration into the value network to obtain value feedback data.

[0096] For example, server 104 can implement an adaptive reward weight adjustment mechanism to handle the specific needs of different types of fault scenarios. Server 104 can dynamically determine the weight coefficients of each reward component based on the preset fault scenario type injected into the current simulated fault environment. For example, for database-related faults, the weight of the data loss penalty score can be increased; for high-concurrency scenario faults, the weight of the recovery time penalty score can be increased; and for business period faults, the weight of the business value recovery score can be increased. Server 104 can also further adjust the weights based on the business period identifier. By obtaining the corresponding adjustment factor through a preset period adjustment factor table, each weight in the basic weight combination is multiplied by the corresponding adjustment factor to obtain a dynamic weight combination adapted to the current scenario. When calculating the comprehensive reward feedback, server 104 can use these dynamic weights to perform a weighted summation of each reward score to ensure that the reward function can accurately reflect the optimization objective under different fault scenarios. Server 104 can input the comprehensive reward feedback sequence generated by each training iteration into the value network and calculate the value feedback data through Monte Carlo methods or temporal difference learning to guide the parameter update direction of the policy network, enabling the agent to learn the long-term optimal decision strategy. During training, server 104 can also periodically test the performance of the current model on verification failure scenarios, monitor metrics such as average cumulative reward, recovery success rate, and convergence speed, and stop training and save the final reinforcement learning model when these metrics reach the preset convergence criteria.

[0097] Furthermore, in the specific implementation of the reward function weight adjustment mechanism, server 104 can maintain a complete dynamic weight adjustment system to manage the adaptive changes of weight coefficients w1 to w5. Server 104 can first construct a fault type weight mapping table, which can predefine basic weight combinations for different types of fault scenarios. For example, for database fault types, the data loss weight w3 can be set to a higher value of 0.4; for high-concurrency fault types, the time cost weight w2 can be set to a higher value of 0.35; and for network fault types, the business value weight w1 can be set to a higher value of 0.5. Server 104 can also establish a business period adjustment factor table, adjusting the relative importance of each weight according to different business period characteristics. For example, during peak business periods, the time cost adjustment factor is set to 1.2; during data-sensitive operations, the data integrity adjustment factor is set to 1.5; and during system maintenance, the resource cost adjustment factor is set to 0.8. When calculating the overall reward value, server 104 can obtain the corresponding basic weight combination from the fault type weight mapping table based on the specific fault type identifier, and then obtain the corresponding adjustment factor from the adjustment factor table based on the business time period to which the current time belongs. Each weight in the basic weight combination is then multiplied by its corresponding adjustment factor to obtain the dynamic weight combination. Server 104 can also continuously optimize the weight adjustment strategy based on feedback from historical fault recovery effects. By analyzing recovery effect data under different weight configurations, it adjusts the parameter settings of the basic weight mapping table and the adjustment factor table, achieving self-optimization and evolution of the weight adjustment mechanism.

[0098] In one exemplary embodiment, such as Figure 4The illustrated architecture implements a complete data processing and decision-making process from offline training to online inference. In the offline training module, the chaos engineering tool first injects preset fault scenarios into the high-fidelity simulation environment, causing corresponding system state changes based on the fault type. The data acquisition agent components in the environment state data and encoding module include Prometheus monitoring agent, Filebeat log collector, and Jaeger tracing component, which can collect real-time monitoring data of the simulation environment. The data processing and graph construction submodule processes this multi-source heterogeneous data uniformly, constructing a dynamic state graph with service instances as nodes and call relationships as edges. The state encoder uses a graph neural network to extract and encode features from the state graph, generating a system state encoding vector containing global topology information. The RL Agent in the reinforcement learning decision engine module receives the state encoding vector and outputs a recovery action selection via PPO or DQN algorithms. This action can be executed in the simulation environment, generating a new environmental state. Multi-objective reward values ​​can be calculated based on the state changes before and after the action execution, and this reward information is fed back to the RL Agent for policy learning and parameter optimization. After numerous training iterations, the mature model is deployed to the online inference stage. In the production environment, the monitoring and alarm system continuously monitors the operational status of application services and infrastructure. When an anomaly is detected, it can trigger a fault recovery process. Real-time data from the production environment is also processed by the environmental status data and encoding module to generate an encoded vector of the current fault state, which is then input into the trained reinforcement learning model. The action execution and orchestration module receives the recovery strategy output by the model, translates the abstract recovery actions into specific platform instructions, converts them into executable commands through an instruction translator, and then implements the recovery operation in the production environment through an executor. The verifier is responsible for verifying the execution effect of each operation and feeding back to the automated operation and maintenance platform to ensure the safety and effectiveness of the recovery process.

[0099] Furthermore, such as Figure 5As shown, during the offline training phase, various fault scenarios are first injected into the high-fidelity simulation environment through the chaos engineering tool module, simulating fault types that may occur in a real production environment, such as service instance shutdown, network connection interruption, and resource exhaustion. Upon receiving the fault injection, the high-fidelity simulation environment will undergo corresponding state changes. The data acquisition agent component in the state awareness and environment modeling module can monitor these changes in real time and collect operational monitoring data. The data processing and graph construction submodule cleans, standardizes, and performs correlation analysis on the collected monitoring data, constructing a dynamic graph structure reflecting the current system state. Then, the state encoder encodes the state graph using a graph neural network (GNN) to generate a system state encoding vector. The RL Agent in the reinforcement learning decision engine receives the system state encoding vector as input and selects the optimal recovery action using either the Proximal Policy Optimization (PPO) or Deep Q-Network (DQN) algorithm. The reward function module calculates multi-objective reward values ​​based on the action execution results, including a comprehensive evaluation of multiple dimensions such as business value recovery, time cost, and data loss risk. The model training and iteration module uses these reward feedbacks to update the agent's network parameters. After millions of training iterations, the mature model is stored in the model storage module. During the online inference phase, application services and infrastructure in the production environment continuously provide real-time operational data to the state awareness module through a monitoring and alarm system. When a fault is detected, the current state encoding vector is quickly generated and input into the pre-trained reinforcement learning model. After the model outputs the optimal recovery strategy, the specific recovery operation is executed in the production environment through an automated operations and maintenance platform.

[0100] Through the above steps, server 104 can successfully train a reinforcement learning model that can intelligently generate the optimal recovery strategy based on the fault state. This model learns the ability to weigh different business objectives under the guidance of a multi-objective reward function, solving the problem that traditional methods cannot adaptively optimize recovery strategies.

[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0102] Based on the same inventive concept, this application also provides a reinforcement learning-based fault recovery device for implementing the reinforcement learning-based fault recovery method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more reinforcement learning-based fault recovery device embodiments provided below can be found in the limitations of the reinforcement learning-based fault recovery method described above, and will not be repeated here.

[0103] In one exemplary embodiment, such as Figure 6 As shown, a fault recovery device based on reinforcement learning is provided, comprising: a data acquisition module 402, a data processing module 404, and a fault recovery module 406, wherein:

[0104] The data acquisition module 402 is used to acquire the operation monitoring data and system status diagram of multiple service instances; wherein, the graph nodes of the system status diagram are each service instance, and the graph edges of the system status diagram are determined according to the calling relationship between each service instance;

[0105] Data processing module 404 is used to update the system status diagram based on the operation monitoring data, and to encode the updated system status diagram through a graph neural network to obtain the system status encoding vector.

[0106] The fault recovery module 406 is used to input the system state encoding vector into the reinforcement learning model to obtain the fault recovery operation path, and to execute the fault recovery according to the fault recovery operation path; the reinforcement learning model is trained in a simulated fault environment based on a multi-objective reward function.

[0107] In one embodiment, the apparatus further includes: a model training module, configured to construct a simulated environment with the same topology as the production environment, deploy service instance replicas corresponding to the production environment in the simulated environment; add preset fault scenarios to the simulated environment to obtain a simulated fault environment; the preset fault scenarios include one or more of service instance stoppage, network connection interruption, and resource exhaustion; acquire operational monitoring data of the simulated fault environment and generate a simulated state encoding vector corresponding to the operational monitoring data; and train a reinforcement learning agent in the simulated fault environment according to a multi-objective reward function and the simulated state encoding vector to obtain a reinforcement learning model.

[0108] In one embodiment, the model training module is specifically used to initialize an agent in a simulated fault environment. The agent includes a policy network and a value network. Based on the simulated state encoding vector, the policy network determines the recovery action and executes the recovery action in the simulated fault environment to obtain a new simulated state encoding vector. Based on the simulated state encoding vectors before and after the recovery action, a multi-objective establishment function is used to calculate the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score to obtain immediate reward feedback. The step of determining the recovery action based on the simulated state encoding vector through the policy network is iteratively executed. Based on the immediate reward feedback, the value feedback data is calculated through the value network, and the reinforcement learning agent is trained based on the value feedback data to obtain a reinforcement learning model.

[0109] In one embodiment, the model training module is further configured to determine the reward weights corresponding to the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score, respectively, based on the type of preset fault scenario in the simulated fault environment; calculate the comprehensive reward feedback based on the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score and their corresponding reward weights; and input the comprehensive reward feedback of each iteration into the value network to obtain value feedback data.

[0110] In one embodiment, the apparatus further includes: a data update unit for acquiring fault recovery data in the production environment; the fault recovery data includes at least one of a system state encoding vector at the time of the fault occurrence, a recovery action sequence, and a recovery effect evaluation result; performing a quality score on the fault recovery data; the quality score is calculated based on at least one of recovery success rate, recovery time, and business impact; when the quality score exceeds a preset update threshold, the fault recovery data is added to the training dataset; the training dataset is used to train a reinforcement learning model.

[0111] In one embodiment, the reinforcement learning model includes a production version model and multiple candidate version models, with the candidate version models deployed in an isolated verification environment. The device also includes a model switching unit, used to generate historical fault data based on the system state encoding vector, and adjust the environmental parameters of the isolated verification environment based on the historical fault data; input the historical fault data into each candidate version model to obtain candidate recovery operation paths, and perform fault recovery in the isolated verification environment according to the candidate recovery operation paths to obtain recovery performance data; calculate the performance improvement index of each candidate version model based on the recovery performance data; and deploy the corresponding candidate version model to the production environment to replace the production version model as the new production version model when the performance improvement index exceeds a preset deployment threshold.

[0112] The modules in the aforementioned reinforcement learning-based fault recovery device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0113] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a reinforcement learning-based fault recovery method.

[0114] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0116] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0117] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0120] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A fault recovery method based on reinforcement learning, characterized in that, The method includes: Obtain runtime monitoring data and system status graphs for multiple service instances; wherein, the graph nodes of the system status graph are each of the service instances, and the graph edges of the system status graph are determined based on the calling relationships between the service instances; The system state diagram is updated based on the operation monitoring data, and the updated system state diagram is encoded using a graph neural network to obtain a system state encoding vector. The system state encoding vector is input into the reinforcement learning model to obtain the fault recovery operation path, and fault recovery is performed according to the fault recovery operation path; the reinforcement learning model is trained in a simulated fault environment based on a multi-objective reward function.

2. The method according to claim 1, characterized in that, The training methods for the reinforcement learning model include: Construct a simulation environment with the same topology as the production environment, and deploy service instance replicas corresponding to the production environment in the simulation environment; A preset fault scenario is added to the simulation environment to obtain a simulated fault environment; the preset fault scenario includes one or more of the following: service instance stoppage, network connection interruption, and resource exhaustion. Obtain the operation monitoring data of the simulated fault environment, and generate the simulated state encoding vector corresponding to the operation monitoring data; Based on the multi-objective reward function and the simulated state encoding vector, a reinforcement learning agent is trained in the simulated fault environment to obtain a reinforcement learning model.

3. The method according to claim 2, characterized in that, The step of training a reinforcement learning agent based on the multi-objective reward function and the simulated state encoding vector to obtain a reinforcement learning model includes: The agent is initialized in the simulated fault environment; the agent includes a policy network and a value network. Based on the simulated state encoding vector, a recovery action is determined through the policy network, and the recovery action is executed in the simulated fault environment to obtain a new simulated state encoding vector. Based on the simulated state encoding vectors before and after the recovery action, the business value recovery score, recovery time penalty score, data loss penalty score, and resource consumption penalty score are calculated respectively through the multi-objective establishment function to obtain immediate reward feedback; The process iteratively executes the step of determining the recovery action through the policy network based on the simulated state encoding vector, calculates value feedback data through the value network based on the immediate reward feedback, and trains the reinforcement learning agent based on the value feedback data to obtain a reinforcement learning model.

4. The method according to claim 3, characterized in that, The step of calculating value feedback data through the value network based on the instant reward feedback includes: Based on the type of the preset fault scenario in the simulated fault environment, the reward weights corresponding to the business value recovery score, the recovery time penalty score, the data loss penalty score, and the resource consumption penalty score are determined respectively. A comprehensive reward feedback is calculated based on the business value recovery score, the recovery time penalty score, the data loss penalty score, the resource consumption penalty score, and the corresponding reward weight; The comprehensive reward feedback from each iteration is input into the value network to obtain value feedback data.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Acquire fault recovery data from the production environment; the fault recovery data includes at least one of the following: system state encoding vector at the time of the fault occurrence, recovery action sequence, and recovery effect evaluation result; The fault recovery data is scored for quality; the quality score is calculated based on at least one of recovery success rate, recovery time, and business impact. When the quality score exceeds a preset update threshold, the fault recovery data is added to the training dataset; the training dataset is used to train the reinforcement learning model.

6. The method according to any one of claims 1 to 4, characterized in that, The reinforcement learning model includes a production version model and multiple candidate version models, wherein the candidate version models are deployed in an isolated verification environment. After performing fault recovery according to the fault recovery operation path, the method further includes: Based on the system state encoding vector, historical fault data is generated, and the environmental parameters of the isolation verification environment are adjusted based on the historical fault data. The historical fault data is input into each of the candidate version models to obtain candidate recovery operation paths, and fault recovery is performed in the isolation verification environment according to the candidate recovery operation paths to obtain recovery performance data; Based on the recovery performance data, calculate the performance improvement metrics for each candidate version model; If the performance improvement index exceeds the preset deployment threshold, the corresponding candidate version model will be deployed to the production environment to replace the production version model as the new production version model.

7. A fault recovery device based on reinforcement learning, characterized in that, The device includes: The data acquisition module is used to acquire the operation monitoring data and system status graph of multiple service instances; wherein, the graph nodes of the system status graph are each of the service instances, and the graph edges of the system status graph are determined according to the calling relationship between the service instances; The data processing module is used to update the system state diagram based on the operation monitoring data, and to encode the updated system state diagram through a graph neural network to obtain a system state encoding vector. The fault recovery module is used to input the system state encoding vector into the reinforcement learning model to obtain the fault recovery operation path, and to perform fault recovery according to the fault recovery operation path; the reinforcement learning model is trained in a simulated fault environment based on a multi-objective reward function.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images