Service fault recovery method and system for micro-service platform, and storage medium

Abstract

The invention provides a service fault recovery method and system for a micro-service platform, a storage medium and equipment. The method comprises the following steps: analyzing a service state in real time, and collecting service state index data; determining a fault service, collecting fault state data, and obtaining structured data of service call graph information based on the service state index data and the fault state data; constructing a reinforcement learning fault recovery model, inputting the structured data of the service call graph information into the reinforcement learning fault recovery model, and training the reinforcement learning fault recovery model based on known historical data; and generating a recovery strategy of the fault service by using the trained reinforcement learning fault recovery model, and executing the recovery strategy to recover the fault. According to the service fault recovery method, the fault recovery efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of automated operation and maintenance, and in particular relates to a method, system, storage medium and device for recovering service failures on a microservice platform. Background Art

[0002] In modern microservice application systems, due to the number of services and the complexity of call relationships, higher requirements are placed on service operation and maintenance. At present, with the development of artificial intelligence technology, various intelligent operation and maintenance models have begun to play a role in service exception handling, fault analysis and other fields. However, related technical research in the field of fault recovery is relatively backward, and the current mainstream method is still to handle faults through manual operations.

[0003] There are two main directions for fault recovery methods at present: one is to propose targeted and specific recovery actions to achieve fault recovery in the operation and maintenance process based on the specific characteristics of the current system and the defined fault scenarios, for example, a recovery plan that restores the system to a valid state after a fault. The recovery plan can be a checkpoint-based rollback strategy, which regularly creates valid snapshots of the system state and returns the system to a valid state when a fault occurs. However, this approach is usually system-specific and may impose a great burden on system designers and operators.

[0004] Another fault recovery method is to use several specific basic recovery methods as alternative actions. According to the different error states presented by the system, the strategy is selected according to the system state and the constructed recovery action selection strategy, and the corresponding alternative means are selected for recovery. This method needs to reasonably arrange potential repair operations according to strategies such as state operation rules. The strategy may include: recursively trying the remaining actions with the lowest cost. One of the basic recovery methods is a simple restart technology. A large part of the faults can be repaired by restarting technology even when the exact cause is unknown. This recovery solution can be applied at different levels and is currently adopted by many important systems.

[0005] However, this simple strategy may not be sufficient in real environments due to imprecise fault localization, recurring faults, or failed repair operations, thus resulting in poor fault recovery. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a service fault recovery method, system, storage medium and device for a microservice platform. A reinforcement learning fault recovery model is constructed based on a deep reinforcement learning algorithm. According to the service status indicators and fault status data obtained by real-time monitoring, recovery actions are automatically selected and executed to achieve automatic recovery of faults, thereby improving fault recovery efficiency and service availability.

[0007] The present invention is achieved through the following technical solutions:

[0008] Conduct real-time analysis of service status and collect service status indicator data;

[0009] Determine the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status indicator data and the fault status data;

[0010] Constructing a reinforcement learning fault recovery model, inputting the structured data of the service call graph information into the reinforcement learning fault recovery model, and training the reinforcement learning fault recovery model based on known historical data;

[0011] The trained reinforcement learning fault recovery model is used to generate a recovery strategy for the faulty service, and the recovery strategy is executed to recover from the fault.

[0012] Optional,

[0013] The service status indicator data includes: inter-node indicator data, container indicator data and server indicator data.

[0014] Optional,

[0015] The structured data of obtaining service call graph information based on service status indicator data and fault status data includes:

[0016] Based on the indicator data between nodes, obtain the calling relationship between services and build a service calling chain;

[0017] The container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain to construct structured data of the service call graph information.

[0018] Optional,

[0019] The aggregating container indicator data and server indicator data into service node information and combining the service call chain to construct structured data of service call graph information includes:

[0020] By querying container metrics and server metrics, obtain the physical machine information of each service in the microservice where the faulty service is located;

[0021] According to the physical machine information to which each service belongs, the container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain of the microservice project, structured data of the service call graph information consisting of the feature vector and adjacency matrix of the node is constructed.

[0022] Optional,

[0023] The construction of the reinforcement learning fault recovery model includes:

[0024] Define the state space, action space, and rewards of the reinforcement learning fault recovery model.

[0025] Optional,

[0026] The using the trained reinforcement learning fault recovery model to generate a recovery strategy for the fault service, and executing the recovery strategy to recover from the fault, includes:

[0027] Obtain service status indicator data from the target system in real time through system monitoring tools;

[0028] Use anomaly detection and fault location tools to identify faulty services and obtain fault status data;

[0029] Based on the service status indicator data and the fault status data, a recovery strategy for the fault service is generated and executed using a trained reinforcement learning fault recovery model.

[0030] The present invention also provides a microservice platform service fault recovery system for implementing the above method, the system comprising:

[0031] Data collection module, used to analyze service status in real time and collect service status indicator data;

[0032] A fault information processing module, used to determine the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status indicator data and the fault status data;

[0033] A reinforcement learning module, used for building a reinforcement learning fault recovery model, inputting the structured data of the service call graph information into the reinforcement learning fault recovery model, and training the reinforcement learning fault recovery model based on known historical data;

[0034] The fault recovery module is used to generate a recovery strategy for the fault service using the trained reinforcement learning fault recovery model, and execute the recovery strategy to recover from the fault.

[0035] Optional,

[0036] The fault information processing module is further configured to:

[0037] Based on the indicator data between nodes, obtain the calling relationship between services and build a service calling chain;

[0038] The container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain to construct structured data of the service call graph information.

[0039] Optional,

[0040] The system is also used to:

[0041] By querying container metrics and server metrics, obtain the physical machine information of each service in the microservice where the faulty service is located;

[0042] According to the physical machine information to which each service belongs, the container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain of the microservice project, structured data of the service call graph information consisting of the feature vector and adjacency matrix of the node is constructed.

[0043] Optional,

[0044] The fault recovery module is further configured to:

[0045] Obtain service status indicator data from the target system in real time through system monitoring tools;

[0046] Use anomaly detection and fault location tools to identify faulty services and obtain fault status data;

[0047] Based on the service status indicator data and the fault status data, a recovery strategy for the fault service is generated and executed using a trained reinforcement learning fault recovery model.

[0048] The present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned microservice platform service failure recovery method.

[0049] The present invention also provides a device, comprising a processor, a communication interface, a computer-readable storage medium and a communication bus; wherein the processor, the communication interface and the computer-readable storage medium communicate with each other via the communication bus;

[0050] The processor is configured to execute a program stored in a computer-readable storage medium.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] 1. The microservice platform service fault recovery method proposed in the present invention builds a reinforcement learning fault recovery model based on a deep reinforcement learning algorithm, obtains service status indicators and fault status information through real-time monitoring and analysis, and automatically selects and executes recovery actions after a fault is discovered, thereby achieving automatic recovery of the fault, improving operation and maintenance efficiency, and further improving service availability.

[0053] 2. Use the operation and maintenance indicators and call relationship diagram of the microservice system to describe the current state of the system, model the fault recovery problem under the microservice architecture as a fully observable Markov decision process, and use the graph convolutional network to fit the strategy of selecting recovery actions to improve the availability of the microservice system.

[0054] 3. The reinforcement learning fault recovery model is trained through the reinforcement learning algorithm, and the learning effect of the intelligent agent is stabilized and accelerated through the priority experience replay mechanism. It learns from historical fault scenarios and recovery actions and optimizes the decision-making strategy of the reinforcement learning fault recovery model.

[0055] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A flow chart of a method for recovering a service failure on a microservice platform is shown;

[0058] Figure 2 A structural schematic block diagram of a microservice platform service fault recovery system is shown;

[0059] Figure 3 A schematic diagram of a service failure automatic recovery framework according to an embodiment of the present invention is shown;

[0060] Figure 4 It is a structural schematic diagram of a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See attached Figure 1 , the method of the present invention comprises:

[0063] S1. Analyze the service status in real time and collect service status indicator data;

[0064] The service status indicator data includes: inter-node indicator data, container indicator data and server indicator data.

[0065] S2. Determine the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status indicator data and the fault status data;

[0066] Among them, based on the indicator data between nodes, the calling relationship between services is obtained and the service calling chain is constructed;

[0067] Aggregate container indicator data and server indicator data into service node information, and construct structured data of service call graph information by combining the service call chain;

[0068] Among them, by querying container indicators and server indicators, obtain the physical machine information of each service in the microservice where the faulty service is located;

[0069] According to the physical machine information to which each service belongs, the container indicators and server indicators are aggregated into service node information, and combined with the service call chain of the microservice project, structured data of the service call graph information consisting of the feature vector and adjacency matrix of the node is constructed.

[0070] S3, constructing a reinforcement learning fault recovery model, inputting the structured data of the service call graph information into the reinforcement learning fault recovery model, and training the reinforcement learning fault recovery model based on known historical data;

[0071] The step of constructing a reinforcement learning fault recovery model includes:

[0072] Define the state space, action space, and rewards of the reinforcement learning fault recovery model.

[0073] S4. Generate a recovery strategy for the faulty service using the trained reinforcement learning fault recovery model, and execute the recovery strategy to recover from the fault.

[0074] Among them, through the system monitoring tool, the service status indicator data is obtained from the target system in real time;

[0075] Use anomaly detection and fault location tools to identify faulty services and obtain fault status data;

[0076] Based on the service status indicator data and the fault status data, a recovery strategy for the fault service is generated and executed using a trained reinforcement learning fault recovery model.

[0077] Specifically,

[0078] 1. Conduct real-time analysis of service status and collect service status indicator data.

[0079] For the working nodes and each microservice on the microservice platform, use monitoring analysis, anomaly detection, and root cause location tools to perform real-time analysis of the service status, and collect and store information such as service status indicators.

[0080] In this embodiment, tools such as Prometheus and Istio can be used to implement real-time analysis of service status and collection of service status indicators. Service status indicators include but are not limited to: service call relationship related indicators and related indicator data such as the physical machine where the service is located.

[0081] The service status indicator data also includes: inter-node indicator data, container indicator data, and server indicator data.

[0082] 2. Determine the faulty service, collect fault status data, and obtain structured data of the service call graph information based on the service status indicator data and the fault status data.

[0083] The structured data of obtaining service call graph information based on service status indicator data and fault status data includes:

[0084] Based on the indicator data between nodes, obtain the calling relationship between services and build a service calling chain;

[0085] The container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain to construct structured data of the service call graph information.

[0086] Aggregate container indicator data and server indicator data into service node information, and combine the service call chain to construct structured data of service call graph information, including:

[0087] By querying container metrics and server metrics, obtain the physical machine information of each service in the microservice where the faulty service is located;

[0088] According to the physical machine information to which each service belongs, the container indicators and server indicators are aggregated into service node information, and combined with the service call chain of the microservice project, structured data of the service call graph information consisting of the feature vector and adjacency matrix of the node is constructed.

[0089] When a service in the platform fails, the faulty service is identified based on the existing anomaly detection and fault location tools, and the fault status data is collected. By processing the service status indicator data and fault status data, the structured data of the service call graph information is obtained.

[0090] In this embodiment, anomaly detection tools include but are not limited to: FVAE, anomaly-transformer, DCdetector, etc.; fault location tools include but are not limited to: Dapper, Pinpoint, Microscope, AutoMap, etc.

[0091] Process the service status indicator data and fault status data to obtain structured data of the service call graph information, including:

[0092] S201. Query the indicator corresponding to the faulty service in the inter-node indicators, obtain all service data packet information under the microservice project where the faulty service is located, determine the starting service node and the ending service node according to the data packet information, and then obtain the calling relationship between services, thereby constructing the service calling chain of the microservice project.

[0093] S202. By querying the container index and the server index, determine the service name under the corresponding namespace and the address of the physical machine where it is located, and obtain the physical machine to which each service in the microservice where the faulty service is located belongs. According to the physical machine information of each service, the container index and the server index are aggregated into the information of the service node, and combined with the service call chain of the microservice project, the structured data of the service call graph information composed of the feature vector and adjacency matrix of the node is constructed.

[0094] In this embodiment, the two indicators of TCP data transmission volume and Request transmission volume corresponding to the faulty service are queried in the inter-node indicators, and the Request and TCP data packet information received and sent by all services under the microservice project can be obtained. The received and sent Request and TCP data packet information obtained by the query are analyzed to obtain the "destination_workload" and "source_workload" attributes. According to the above attributes, the starting service node and the ending service node can be determined, thereby obtaining the call relationship between services, and thus constructing the service call chain of the microservice project.

[0095] By querying the CPU usage time of each service in the microservice where the faulty service is located, the query results "container" and "instance" return the service name and the physical machine address of the corresponding namespace, respectively, and then obtain the physical machine to which each service belongs. According to the physical machine information of each service, the container index and server index are aggregated into the information of the service node, combined with the service call chain of the microservice project, to construct a structured data of the service call graph information consisting of the node feature vector and adjacency matrix. The feature matrix X and the adjacency matrix A are shown in formula (1):

[0096]

[0097] Among them, the number of rows m of the feature matrix represents the number of nodes, the number of columns n represents the number of features of each node, and the adjacency matrix represents the information of adjacent nodes.

[0098] 3. Build a reinforcement learning fault recovery model.

[0099] Construct a reinforcement learning fault recovery model, taking the structured data of the service call graph information as input data f t , input into the reinforcement learning fault recovery model, and train the reinforcement learning fault recovery model in combination with known historical data.

[0100] Constructing a reinforcement learning fault recovery model includes: defining a state space, an action space, and a reward of the reinforcement learning fault recovery model.

[0101] 1. Define the state space.

[0102] The state space is composed of the feature vectors f of the m nodes in the current microservice project t , the error node e obtained according to the fault status data j 、Action a performed in the previous step t-1 And the information r of the recovery result is used to describe it.

[0103] Define the state space S at time t t as follows:

[0104] S t =(f t ,e j ,a t-1 , r)

[0105] In this embodiment, the feature matrix X and the adjacency matrix A are used to represent the feature vectors f of the m nodes in the current microservice project. t :

[0106]

[0107] Through these key performance indicator data of operation and maintenance that intuitively reflect the fault status, the system status can be accurately described and the state space can be constructed.

[0108] 2. Define the action space.

[0109] The action space is the set of all possible actions that the agent can take in a given state. The reinforcement learning fault recovery model outputs the agent to select a recovery action based on the current state, thereby changing the environment to the next state.

[0110] The design of the action space is critical for both microservice system control and fault recovery. A good action design can reduce the difficulty of system control and thus better solve problems. At the same time, the construction of the action space also needs to be complete enough to ensure that the problem can be solved effectively. For example, in some cases, some actions may not be obvious, but they may be very important to solve the problem. Therefore, when constructing the action space, it is necessary to ensure that it can provide diverse and precise control to solve the problem.

[0111] In this embodiment, the action space consists of optional recovery methods, including but not limited to: micro-reboot, service instance migration, service instance horizontal expansion, and service instance vertical expansion.

[0112] Micro-Reboot: Individually restart fine-grained application components. This requires well-isolated stateless components, saving all important application states (including current resource limits, host, etc.) in a dedicated state store, and directly restarting service instances according to the stored state when taking actions.

[0113] Service instance migration (Migrate): Consider the resource occupancy of different working nodes and migrate the current service / component to the working node in good condition.

[0114] Scale out a service instance: Scale out copies of the current service instance on other nodes (proportionally), while maintaining the current instance state.

[0115] Scale up a service instance: Without changing the number of replicas of the current service instance, add more CPU and memory resource quotas to the service instance to vertically scale the performance of the instance.

[0116] Service instance rollback: During the continuous integration of the microservice system, the service instance is also updated and changed. The rollback operation can roll back the current replica instance to the previous deployment version.

[0117] These specific recovery methods constitute an action space, and the intelligent agent determines from the action space which action is more conducive to improving the state.

[0118] 3. Reward design.

[0119] Reward design is an important part of successful learning for the agent. It serves as an "action guide" for the agent to take action and reflects the evaluation index of specific recovery actions. Depending on the agent's goals, rewards can be considered from multiple aspects, including but not limited to: recovery result, recovery time, and recovery cost.

[0120] Recovery Result: After taking this action in a fault state, the environment changes to the next state. Whether the next state is no longer judged as a fault state is an intuitive measure of the recovery effect.

[0121] Recovery Time: The time cost of the steps required to recover from a faulty state to a normal state. The time taken for the system recovery process may directly affect the commercial benefits of the system.

[0122] Recovery Cost: All costs incurred by all API functions included in the recovery operation. Recovery operations such as starting a new instance will incur additional performance overhead costs. Different recovery operations mean different costs, which is also a part that needs to be considered.

[0123] In this embodiment, the recovery result RR, the recovery time RT and the recovery cost RC are used as three criteria for evaluating an action, and the reward or penalty for the action can be expressed as follows:

[0124] Reward action =RR+RT+RC

[0125] 4. Reinforcement learning fault recovery model training.

[0126] The structured data of the service call graph information is used as input data and input into a reinforcement learning fault recovery model, and the reinforcement learning fault recovery model is trained in combination with known historical data.

[0127] The reinforcement learning network consists of two graph convolution layers with ReLU activation function and two fully connected layers. The graph convolution layer can well aggregate the information around the node to the current node, fully reflecting the call relationship. The information of the graph structure is extracted through two layers of graph convolution, and the extracted graph structure information data is flattened into one dimension, and the data is mapped to the action space through the fully connected layer of the two-layer network.

[0128] The reinforcement learning fault recovery model is trained through the fault recovery algorithm, and the recovery strategy for selecting recovery actions is optimized, so that the action selection of the reinforcement learning fault recovery model finds a balance between "exploration and utilization".

[0129] In this embodiment, the fault recovery algorithm training process includes:

[0130] Initialize learning rate α and network parameters θ;

[0131] Initialize the training batch B for each training step, the number of updates U for each training batch, and the size N of each batch;

[0132] Initialize Bol tamann strategy temperature τ; #Used to adjust the randomness of the strategy when making decisions

[0133] Initialize the experience memory with the maximum batch size K, and initialize ε and η; #ε and η are exploration parameters used to control the smoothness and strength of the priority calculation.

[0134] for m=1,…,MAX STEPS do: #Perform multi-step exploration training in each experiment

[0135] Use the current strategy to collect and store h experiences (s i ,a i ,r i ,s′ i ,terminal); #The algorithm collects a certain amount (h) of experience by interacting with the environment, which is then used for training.

[0136] Among them, r i For state s i Next, take action i The immediate reward obtained later is calculated according to the reward function; s′ i For state s i Next, take action i The new state to be transferred to is determined by the state transfer function and the previous state and action; a i For state s′ i Next, the action is chosen according to the current policy.

[0137] When the algorithm executes the strategy, it updates the state and reward based on the response of the environment.

[0138] for b=1to B do:

[0139] Extract the real experience of batch b from the experience replay memory;

[0140] for u=1,…,U do: #Number of parameter updates performed in each training batch. Each update adjusts the network parameters based on the samples in the batch.

[0141] for i=1,…,N do: #Traverse the samples in each training batch, process each sample and update the network parameters.

[0142] #Calculate the target Q value for each sample:

[0143]

[0144] Among them, when s′ i Upon termination, Otherwise, 1;

[0145] # Calculate the absolute TD error for each sample:

[0146]

[0147] end for;

[0148] #Calculate loss, using MSE:

[0149]

[0150] #Update network parameters:

[0151]

[0152] Calculate the priority of each sample:

[0153]

[0154] Update the experience replay memory based on the new priority;

[0155] end for

[0156] end for

[0157] Attenuation τ

[0158] end for

[0159] In this embodiment, the fault recovery model is reinforced by training the fault recovery algorithm, so that the recovery strategy can be optimized, and the recovery action can be better selected.

[0160] 4. Use the trained reinforcement learning fault recovery model to generate and execute recovery strategies.

[0161] Deploy and configure the reinforcement learning fault recovery model in the service platform to achieve rapid recovery from faults.

[0162] In this embodiment, service status indicator data is obtained from the target system in real time through system monitoring tools, and faulty services are determined and fault status data is obtained using anomaly detection and fault location tools. Based on the service status indicator data and fault status data, a recovery strategy is automatically generated using a trained reinforcement learning fault recovery model, and based on the generated recovery strategy, specific recovery actions are selected and executed to achieve automatic recovery of service failures.

[0163] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0164] See attached Figure 2 , shows the structure of a microservice platform service fault recovery system for implementing the above method, including a data acquisition module, a fault information processing module, a reinforcement learning module, and a fault recovery module.

[0165] The data collection module is used to analyze the service status in real time and collect service status indicator data;

[0166] The fault information processing module is used to determine the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status indicator data and the fault status data;

[0167] The reinforcement learning module is used to construct a reinforcement learning fault recovery model, input the structured data of the service call graph information into the reinforcement learning fault recovery model, and train the reinforcement learning fault recovery model based on known historical data;

[0168] The fault recovery module is used to generate a recovery strategy for the fault service using the trained reinforcement learning fault recovery model, and execute the recovery strategy to recover from the fault.

[0169] Figure 3 This is a schematic diagram of the automatic recovery framework for service faults in an embodiment of the present invention. The indicator information is obtained from the target system; anomaly detection is performed on the target system, and the fault location is obtained according to the anomaly list, and the error status information is collected; a reinforcement learning fault recovery model is used to generate a recovery strategy for the faulty service, and a recovery action is selected according to the recovery strategy to perform error recovery. During the anomaly inspection process, the service anomaly log is recorded, including anomaly information, recovery status, etc., for the optimization and generation of recovery strategies.

[0170] In addition, an embodiment of the present invention further provides a microservice platform service failure recovery device, including:

[0171] The data collection module performs real-time analysis on the service status and collects service status indicator data; the service status indicator data includes: inter-node indicator data, container indicator data and server indicator data.

[0172] The fault information processing module determines the faulty service, collects fault status data, and obtains structured data of the service call graph information based on the service status indicator data and the fault status data; wherein, based on the indicator data between nodes, the call relationship between services is obtained and a service call chain is constructed; the container indicator data and the server indicator data are aggregated into service node information, and the structured data of the service call graph information is constructed in combination with the service call chain.

[0173] A reinforcement learning module is provided to construct a reinforcement learning fault recovery model, input the structured data of the service call graph information into the reinforcement learning fault recovery model, and train the reinforcement learning fault recovery model based on known historical data; wherein, the construction of the reinforcement learning fault recovery model includes: defining the state space, action space and reward of the reinforcement learning fault recovery model.

[0174] A fault recovery module uses the trained reinforcement learning fault recovery model to generate a recovery strategy for the faulty service and executes the recovery strategy to recover from the fault; wherein, service status indicator data is obtained from the target system in real time through a system monitoring tool; anomaly detection and fault location tools are used to determine the faulty service and obtain fault status data; based on the service status indicator data and fault status data, a recovery strategy for the faulty service is generated and executed using the trained reinforcement learning fault recovery model.

[0175] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned microservice platform service failure recovery method.

[0176] like Figure 4 As shown, an embodiment of the present invention further provides a device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus.

[0177] The memory is a computer-readable storage medium for storing one or more programs.

[0178] The processor is used to execute the program stored in the computer-readable storage medium.

[0179] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or may exist independently without being assembled into the device / apparatus.

[0180] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recovering a service failure on a microservice platform, characterized in that: include: Conduct real-time analysis of service status and collect service status indicator data; Determine the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status indicator data and the fault status data; Constructing a reinforcement learning fault recovery model, inputting the structured data of the service call graph information into the reinforcement learning fault recovery model, and training the reinforcement learning fault recovery model based on known historical data; The trained reinforcement learning fault recovery model is used to generate a recovery strategy for the faulty service, and the recovery strategy is executed to recover from the fault.

2. The method according to claim 1, characterized in that: The service status indicator data includes: inter-node indicator data, container indicator data and server indicator data.

3. The method according to claim 1, characterized in that The structured data of obtaining service call graph information based on service status indicator data and fault status data includes: Based on the indicator data between nodes, obtain the calling relationship between services and build a service calling chain; The container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain to construct structured data of the service call graph information.

4. The method according to claim 3, characterized in that The aggregating container indicator data and server indicator data into service node information and combining the service call chain to construct structured data of service call graph information includes: By querying container metrics and server metrics, obtain the physical machine information of each service in the microservice where the faulty service is located; According to the physical machine information to which each service belongs, the container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain of the microservice project, structured data of the service call graph information consisting of the feature vector and adjacency matrix of the node is constructed.

5. The method according to any one of claims 1 to 4, characterized in that: The construction of the reinforcement learning fault recovery model includes: Define the state space, action space, and rewards of the reinforcement learning fault recovery model.

6. The method according to any one of claims 1 to 4, characterized in that: The using the trained reinforcement learning fault recovery model to generate a recovery strategy for the fault service, and executing the recovery strategy to recover from the fault, includes: Obtain service status indicator data from the target system in real time through system monitoring tools; Use anomaly detection and fault location tools to identify faulty services and obtain fault status data; Based on the service status indicator data and the fault status data, a recovery strategy for the fault service is generated and executed using a trained reinforcement learning fault recovery model.

7. A microservice platform service failure recovery system, characterized in that: The system comprises: Data collection module, used to analyze service status in real time and collect service status indicator data; A fault information processing module, used to determine the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status indicator data and the fault status data; A reinforcement learning module, used for building a reinforcement learning fault recovery model, inputting the structured data of the service call graph information into the reinforcement learning fault recovery model, and training the reinforcement learning fault recovery model based on known historical data; The fault recovery module is used to generate a recovery strategy for the fault service using the trained reinforcement learning fault recovery model, and execute the recovery strategy to recover from the fault.

8. The system according to claim 7, characterized in that The fault information processing module is further configured to: Based on the indicator data between nodes, obtain the calling relationship between services and build a service calling chain; The container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain to construct structured data of the service call graph information.

9. The system according to claim 8, characterized in that The system is also used to: By querying container metrics and server metrics, obtain the physical machine information of each service in the microservice where the faulty service is located; According to the physical machine information to which each service belongs, the container indicator data and the server indicator data are aggregated into service node information, and combined with the service call chain of the microservice project, structured data of the service call graph information consisting of the feature vector and adjacency matrix of the node is constructed.

10. The system according to any one of claims 7 to 9, characterized in that: The fault recovery module is further configured to: Obtain service status indicator data from the target system in real time through system monitoring tools; Use anomaly detection and fault location tools to identify faulty services and obtain fault status data; Based on the service status indicator data and the fault status data, a recovery strategy for the fault service is generated and executed using a trained reinforcement learning fault recovery model.

11. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed, the microservice platform service failure recovery method described in any one of claims 1 to 6 can be implemented.

12. An electronic device comprising a processor, a communication interface, the computer-readable storage medium of claim 11, and a communication bus; wherein: The processor, the communication interface, and the computer-readable storage medium communicate electronically with each other via a communication bus; characterized in that: The processor is configured to execute a program stored in a computer-readable storage medium.

Images