Fault-tolerant processing method and device, medium and computer program product

Abstract

The invention relates to the technical field of communication, and particularly discloses a fault-tolerant processing method and device, a medium and a computer program product, after state information of each resource node of a target distributed system is monitored and a fault resource node is determined, a service task on the fault resource node is used as a to-be-processed task, and the priority of the to-be-processed task is determined; performing fault-tolerant processing on the to-be-processed task according to a sequence of priorities from high to low, generating a plurality of fault-tolerant processing actions of the to-be-processed task by utilizing a fault-tolerant processing action generation model according to the state information at the current moment, performing availability evaluation on each fault-tolerant processing action, determining an actual fault-tolerant processing action according to an availability evaluation result, and performing fault-tolerant processing on the to-be-processed task according to the actual fault-tolerant processing action. According to the method and the device, the to-be-processed task is subjected to the fault-tolerant processing action, so that the task on the resource node can be subjected to feasibility scheduling when the resource node in the distributed system fails, the continuous execution of the task is guaranteed, and the reliability of the distributed system is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a fault-tolerant processing method, device, medium and computer program product. Background Art

[0002] A distributed system refers to a system with multiple resource nodes, which may include but are not limited to computing resource nodes or storage resource nodes. In order to improve the reliability and availability of distributed systems, they need to have strong fault tolerance, which mainly includes fault avoidance and fault tolerance. Among them, fault tolerance includes active fault tolerance and passive fault tolerance. Active fault tolerance refers to pre-processing faults by predicting them. Passive fault tolerance refers to responding to faults by timely discovering faults that have occurred and adopting fault handling strategies.

[0003] Based on the above fault-tolerant processing ideas, there are many fault-tolerant strategies for distributed systems in related technologies. However, these fault-tolerant strategies only consider the processing of device failures at the current moment, and do not consider the ability to ensure the continuous operation of the services provided by the distributed system.

[0004] Improving the ability of fault-tolerant scheduling of distributed systems to ensure the continuous operation of services provided in the distributed systems is a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a fault-tolerant processing method, device, medium and computer program product for improving the ability of fault-tolerant scheduling of a distributed system to ensure the continuous operation of services provided in the distributed system.

[0006] In order to solve the above technical problems, the present invention provides a fault-tolerant processing method, comprising: Monitor the status information of each resource node of the target distributed system; After detecting a faulty resource node, determining that a service task on the faulty resource node is a task to be processed; Determine the priority of the pending tasks, and perform fault-tolerant processing on the pending tasks in order of priority from high to low; When performing fault-tolerant processing on the task to be processed, a trained fault-tolerant processing action generation model is used to generate multiple fault-tolerant processing actions for the task to be processed according to the status information at the current moment, and an availability evaluation is performed on each of the fault-tolerant processing actions. The actual fault-tolerant processing action is determined according to the availability evaluation result, and the actual fault-tolerant processing action is executed on the task to be processed.

[0007] On the one hand, determining the priority of the task to be processed includes: Determine a fault impact parameter of the faulty resource node according to the fault type of the faulty resource node; Determine a task evaluation parameter of the task to be processed according to a load state of the task to be processed on the faulty resource node; The priority score of the task to be processed is determined according to the fault impact parameter and the task evaluation parameter.

[0008] On the other hand, determining the fault impact parameter of the faulty resource node according to the fault type of the faulty resource node includes: Determine a predicted value of the fault duration of the faulty resource node according to historical operation data of the faulty resource node; The predicted value of the fault duration is used as the fault impact parameter.

[0009] On the other hand, determining the predicted value of the fault duration of the faulty resource node according to the historical data corresponding to the fault type of the faulty resource node includes: Determine an expected value of the fault duration of the faulty resource node according to the fault type of the faulty resource node and a corresponding fault duration probability distribution model; Using the expected value of the fault duration as the predicted value of the fault duration; The fault duration probability distribution model corresponding to the fault type is determined according to the historical operation data of the resource node.

[0010] On the other hand, determining the task evaluation parameter of the task to be processed according to the load state of the task to be processed on the faulty resource node includes: The resource occupancy parameter of the task to be processed on the faulty resource node and the remaining execution time of the task to be processed on the faulty resource node are used as the load state of the task to be processed on the faulty resource node, so as to determine the task evaluation parameter according to the resource occupancy parameter and the remaining execution time.

[0011] On the other hand, the step of determining the resource occupancy parameter includes: Determine the resource occupation parameter according to the amount of multiple types of resources occupied by the task to be processed on the faulty resource node; Determining the task evaluation parameter according to the resource occupancy parameter and the remaining execution time includes: The ratio of the resource occupancy parameter to the remaining execution time is used as the task evaluation parameter.

[0012] On the other hand, determining the task evaluation parameter of the task to be processed according to the load state of the task to be processed on the faulty resource node includes: The task evaluation parameter is determined according to the load state of the task to be processed on the faulty resource node and the task type priority parameter of the task to be processed.

[0013] On the other hand, the fault impact parameter is a predicted value of the fault duration of the faulty resource node; The task evaluation parameter is the resource consumption per unit time of the task to be processed at the faulty resource node; Determining the priority score of the task to be processed according to the fault impact parameter and the task evaluation parameter includes: The ratio of the resource consumption value per unit time to the predicted fault duration value is used as the priority score.

[0014] On the other hand, the training step of the fault-tolerant processing action generation model includes: Inputting sample state information into the fault-tolerant processing action generation model, and outputting a sample fault-tolerant processing action for the sample state information; After executing the sample fault tolerance processing action, obtaining updated sample status information; Determine an environment reward value according to the updated sample state information and reward function; Taking a group of the sample fault-tolerant processing actions, the sample fault-tolerant processing actions, the updated sample state information and the environmental reward value as a training sample; The model parameters of the fault-tolerant processing action generation model are updated using the training samples.

[0015] On the other hand, the reward function is a function of the fault-tolerant processing time of the task.

[0016] On the other hand, the fault-tolerant processing time includes a first processing time of a task migration action, a second processing time of a task waiting action, and a third processing time of a task scaling action; Among them, the task migration action is to migrate the pending task from the faulty resource node to the first resource node, and the first processing time includes the data transmission time and the service recovery time at the first resource node; the task waiting action is to set the pending task to a service interruption state at the faulty resource node, and the second processing time is the service interruption waiting time; the task scaling action is to start the pending task at the second resource node where a copy of the pending task exists and terminate the pending task at the faulty resource node, and the third processing time is the service recovery time at the second resource node.

[0017] On the other hand, the availability of each fault-tolerant processing action is evaluated, and the actual fault-tolerant processing action is determined according to the availability evaluation result, including: The availability of the fault-tolerant processing actions is evaluated in descending order of the corresponding environmental reward values, until the fault-tolerant processing actions satisfying the availability evaluation conditions are obtained as the actual fault-tolerant processing actions.

[0018] In order to solve the above technical problems, the present invention further provides a fault-tolerant processing device, comprising: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the fault-tolerant processing method as described in any one of the above items are implemented.

[0019] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the fault-tolerant processing method as described in any one of the above items are implemented.

[0020] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the fault-tolerant processing method as described in any one of the above items are implemented.

[0021] The fault-tolerant processing method provided by the present invention has the beneficial effect of monitoring the status information of each resource node of the target distributed system, after monitoring the faulty resource node, determining the business task on the faulty resource node as the task to be processed, and determining the priority of the task to be processed, so as to perform fault-tolerant processing of the task to be processed in order of priority from high to low; when performing fault-tolerant processing of the task to be processed, using a trained fault-tolerant processing action generation model to generate multiple fault-tolerant processing actions for the task to be processed according to the status information at the current moment, performing availability evaluation on each fault-tolerant processing action, determining the actual fault-tolerant processing action according to the availability evaluation result, and performing the actual fault-tolerant processing action on the task to be processed, so that when a resource node fails in the distributed system, the tasks on the resource node can be feasibly scheduled to ensure the continuous execution of these tasks, without causing a long service interruption due to the failure of the node or the inability to continue to provide services after migration due to migration to an inappropriate node, thereby ensuring the continuous operation of the services provided by the distributed system and effectively improving the reliability of the distributed system.

[0022] The fault-tolerant processing device, non-volatile storage medium and computer program product provided by the present invention have the above-mentioned beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of a fault-tolerant processing method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a distributed system fault-tolerant scheduling architecture provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a fault-tolerant processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The core of the present invention is to provide a fault-tolerant processing method, device, medium and computer program product for improving the ability of fault-tolerant scheduling of a distributed system to ensure the continuous operation of services provided in the distributed system.

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] To facilitate understanding of the technical solution provided by the embodiment of the present invention, some key terms used in the embodiment of the present invention are first explained here.

[0028] A distributed system is a system composed of multiple computers connected through communication lines. Therefore, each node in the distributed system contains its own processor and memory, and has the function of independently processing data. A large task can be divided into several subtasks, which can be executed on different nodes in the distributed system. Usually for users, a distributed system is a unified whole, and when executing user tasks, it uses the distributed system's intelligent agent to schedule tasks.

[0029] Depending on the design goals, a distributed system can be designed as a distributed storage system or a distributed computing system. Distributed storage systems are mainly used to handle storage business tasks, aiming to store and manage large amounts of data, and provide data persistence, reliability, and accessibility. Distributed computing systems are mainly used to provide computing business tasks, aiming to handle large-scale computing tasks in a distributed manner and improve computing efficiency and processing capabilities.

[0030] In the fault-tolerant processing solution provided in the embodiment of the present invention, since the nodes in the distributed system provide the resources required to execute business tasks, these nodes are called resource nodes. A resource node is usually a computer or a server, and is configured as a computing resource node or a storage resource node or other focused resource nodes according to business needs.

[0031] The edge network is a distributed computing environment that deploys data storage and analysis capabilities close to the data generation endpoints or devices to achieve localized data processing.

[0032] Edge data centers are an emerging technology in recent years and have received widespread attention from industry users including operators. They can provide computing power, memory, storage and other resource support for mobile terminal users, and achieve interconnection through the communication network established by operators. In the same geographical area, edge data centers can achieve load balancing through mutual collaboration, which greatly expands the business scope of edge service providers and can respond to users quickly. For example, mobile communication operators place some communication services or network elements at the edge to reduce user response delays.

[0033] In distributed systems, especially edge computing systems (compared with cloud data centers, edge computing systems are usually located in a more unreliable environment, including physical environments that are prone to power outages, downtime due to construction, and management methods that prevent them from being inspected or repaired in a timely manner), in order to improve the reliability and availability of the system, it is necessary for it to have strong fault tolerance.

[0034] In related technologies, the fault tolerance technology of distributed systems is mainly divided into two categories: fault avoidance and fault tolerance.

[0035] Among them, fault avoidance techniques involve probabilistic reliability-aware algorithms for task allocation, virtual machine placement, service scheduling, or resource provisioning in distributed systems. These techniques consider the probability of failure of data, nodes, or communication links based on Poisson distribution or using historical data combined with the mean and variance of previous failures in the system. The goal of such fault-tolerance methods is usually to optimize latency, energy consumption, cost, and bandwidth.

[0036] Fault tolerance can be divided into active fault tolerance and passive fault tolerance. The difference lies in the means of fault perception. Active fault tolerance refers to pre-processing faults by predicting them. Passive fault tolerance refers to responding to faults by timely discovering faults that have occurred and adopting fault handling strategies.

[0037] Whether it is active fault tolerance, passive fault tolerance, or fault avoidance, it mainly manages the scheduling strategy in a mobile manner to minimize the impact of the fault. For the selection of fault tolerance strategies, the main methods currently used include resource redundancy, robust network design, and other real-time fault tolerance methods based on fault perception.

[0038] Based on this, the main challenges facing the current fault-tolerant technology of distributed systems are that the fault-tolerant strategy only considers the reliability of the network and resources themselves, but does not consider the ability to ensure the continuous operation of running services; resource availability factors are not sufficiently considered; reliability is strong, but flexibility is poor.

[0039] In order to alleviate the problem of distributed system unavailability caused by such failures and ensure that various services running in them do not suffer large-scale interruptions and failures, thereby avoiding large losses to users, it is necessary to improve the fault tolerance of distributed systems to minimize the losses caused by failures, ensure the continuity of user services, and improve the reliability of distributed systems.

[0040] In order to improve the guarantee capability of the fault-tolerant scheduling of the distributed system for the continuous operation of the services provided in the distributed system, the fault-tolerant processing scheme provided by the embodiment of the present invention monitors the status information of each resource node of the target distributed system, and after detecting the faulty resource node, determines the business task on the faulty resource node as the task to be processed, and determines the priority of the task to be processed, so as to perform the fault-tolerant processing of the task to be processed in descending order of priority; when performing the fault-tolerant processing of the task to be processed, a trained fault-tolerant processing action generation model is used to generate multiple fault-tolerant processing actions for the task to be processed according to the status information at the current moment, and an availability evaluation is performed on each fault-tolerant processing action, and an actual fault-tolerant processing action is determined according to the availability evaluation result, and the actual fault-tolerant processing action is performed on the task to be processed, so that when a resource node fails in the distributed system, the tasks on the resource node can be feasibly scheduled to ensure the continuous execution of these tasks, without causing a long service interruption due to the failure of the node or the inability to continue to provide services after migration due to migration to an inappropriate node, thereby ensuring the continuous operation of the services provided by the distributed system and effectively improving the reliability of the distributed system.

[0041] The fault-tolerant processing method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0042] Figure 1 A flowchart of a fault-tolerant processing method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a distributed system fault-tolerant scheduling architecture provided by an embodiment of the present invention.

[0043] like Figure 1 As shown, the fault tolerance processing method provided by the embodiment of the present invention may include: S101: monitoring the status information of each resource node of the target distributed system; S102: after detecting a faulty resource node, determining that a business task on the faulty resource node is a task to be processed; S103: Determine the priority of the tasks to be processed, and perform fault-tolerant processing on the tasks to be processed in descending order of priority; S104: When performing fault-tolerant processing on the task to be processed, a trained fault-tolerant processing action generation model is used to generate multiple fault-tolerant processing actions for the task to be processed according to the current state information, and availability evaluation is performed on each fault-tolerant processing action. The actual fault-tolerant processing action is determined according to the availability evaluation result, and the actual fault-tolerant processing action is executed on the task to be processed.

[0044] In an embodiment of the present invention, as introduced in the above concepts, the target distributed system may be any type of distributed system, such as a distributed computing system, a distributed storage system, an edge computing system, and the like.

[0045] The fault-tolerant processing method provided by the embodiment of the present invention can be applied to one or more resource nodes in the target distributed system, and can also be applied to one or more computing devices outside the target distributed system.

[0046] like Figure 2 As shown, by deploying an agent decision network and an action filter on a computing device, the state information in the target distributed system can be perceived, and the agent decision network can be used to generate a fault-tolerant processing action according to the state information. The action filter can be used to filter out the actual fault-tolerant processing action from the fault-tolerant processing action. After the actual fault-tolerant processing action is executed in the target distributed system, the state information in the target distributed system can continue to be perceived, and the reward function can be used to calculate the environment reward value, which is returned to the agent decision network.

[0047] In the specific implementation, for S101, the real-time status information of the target distributed system can be collected by deploying software and hardware detection devices in the target distributed system, and potential or existing system failures can be mined through data processing technology to support fault-tolerant processing of the failure location in subsequent steps.

[0048] like Figure 2 As shown, in the embodiment of the present invention, the status information may include but is not limited to basic environment status information, task status information and fault information.

[0049] The basic environment status information may include network topology information of the target distributed system, resource status information of resource nodes, and network status information between resource nodes.

[0050] The network topology information can include the resource nodes in the target distributed system and the connection relationship between resource nodes, which can be represented by a relationship diagram. Indicates. Among them, Represents the target distributed system resource nodes, Represents the edge formed by the interconnection between resource nodes.

[0051] The resource status information of the resource node may include information about the computing resources and storage resources on the resource node. Among them, the storage resources may include memory resources and persistent storage resources. The resource status information can be as follows: ; in, , , Represents resource nodes The total amount of computing resources, available computing resources, and occupied computing resources are , , Represents resource nodes The total amount of memory resources, available memory resources, and occupied memory resources are , , Represents resource nodes The current total amount of persistent storage resources, the available amount of persistent storage resources, and the occupied amount of persistent storage resources.

[0052] The network status information between resource nodes is used to indicate the interconnection status between resource nodes. To represent resource nodes and resource nodes The interconnection status information between Represents a resource node To the resource node The actual bandwidth between Indicates the average delay level. The actual bandwidth capacity indicates the data transmission capacity, while the delay level is affected by the location of the resource nodes in the network topology, the link distance, and other external interferences that cause data transmission and reception delays. Therefore, Changes with system dynamics.

[0053] Task status information may include information about various business tasks (for user services) deployed on each resource node. Business tasks can be considered as tasks with time duration. express Moment Resource Node A collection of various business tasks running on the server. Information indicating one of the business tasks, which may include: ; in, , , Respectively represent business tasks The amount of computing resources, memory resources, and persistent storage resources used by a single copy of the database. Indicates business tasks The number of replicas running in the resource node. The total consumption of the replicas in the resource node can be calculated based on the service information. Indicates business tasks The occupation period of the current resource. The occupation period may refer to the time taken for a business task to be fully executed once on the resource node.

[0054] Fault information may include the type of fault that occurred on the resource node and the duration of the fault. Fault information It can be expressed as: ; in, Indicates the fault type. The state quantity may include resource node failure (which can be represented by 0), resource node failure (which can be represented by n, and the specific value of n represents the specific failure type), and resource node related network failure (which can be represented by e, and the specific value of e represents the specific network or resource node pointed to). Among them, the resource node related network may include a network composed of all resource nodes that have a connection relationship with the resource node. For other resource nodes directly connected to the resource node, other resource nodes interconnected with the resource node and other resource nodes unidirectionally pointed to by the resource node may be included in the resource node related network; for other resource nodes not directly connected to the resource node, a hop count threshold may be pre-set, and other resource nodes whose connection with the resource node is less than or equal to the hop count threshold may be included in the resource node related network, and other resource nodes whose connection with the resource node is greater than the hop count threshold may not be considered.

[0055] Indicates the fault duration, The state quantity may include the expected duration of the fault (which can be represented by T, and the specific value of T can be a specific duration) and the unknown duration of the fault (which can be represented by TE). The unknown duration of the fault means that the current fault of the resource node cannot be recovered and the duration of the fault cannot be determined.

[0056] By collecting historical data of each resource node, the regularity of the fault duration corresponding to each fault type can be obtained. Represents a resource node The actual fault type set that occurs on Represents a resource node The probability distribution model of the fault duration corresponding to the actual fault type that occurs on the network can be obtained by statistically analyzing historical data.

[0057] Then in S101, the quantized perception target data can be obtained by perceiving the above state information. ,in, Represents the network topology information of the target distributed system, Represents a resource node Resource status information, Represents a resource node and resource nodes The interconnection status information between express Moment Resource Node A collection of various business tasks running on the Represents a resource node Fault information.

[0058] The intelligent agent decision network is used to manage the resource nodes that are failing based on the perceived target data, that is, to perform fault-tolerant scheduling of the business tasks on the failed resource nodes.

[0059] At the current moment, there may be multiple faulty resource nodes in the target distributed system, and there may be multiple business tasks on a faulty resource node. According to the device scheduling capability and system design, the tasks to be processed can be scheduled one by one, or multiple tasks to be processed can be scheduled at a time and completed in batches. Therefore, it is necessary to determine the priority of the tasks to be processed, and perform fault-tolerant processing on the tasks to be processed in order from high to low priority. The priority of the tasks to be processed can be determined based on at least one of the task type, workload, and user-set parameters of the tasks to be processed.

[0060] The fault-tolerant processing action generation model can be trained by reinforcement learning or deep reinforcement learning. The trained fault-tolerant processing action generation model is used to generate multiple fault-tolerant processing actions for the tasks to be processed according to the current state information. During operation, the parameters of the fault-tolerant processing action generation model can also be fine-tuned according to the operating state of the target distributed system.

[0061] In order to improve the availability of fault-tolerant scheduling results, in an embodiment of the present invention, an action filter is designed to filter multiple fault-tolerant processing actions output by the fault-tolerant processing action generation model, based on pre-configured availability evaluation indicators, to ensure that the actual fault-tolerant processing actions available in the target distributed system at the current moment are filtered out and executed.

[0062] When there are still pending tasks that have not been fault-tolerantly scheduled in the target distributed system, S104 is continued for the next one or several pending tasks. When the next one or several pending tasks are processed, the next moment is entered, that is, the state information of the target distributed system is updated, and S104 is executed according to the state information at the next moment.

[0063] The fault-tolerant processing method provided by the embodiment of the present invention monitors the status information of each resource node of the target distributed system, and after detecting the faulty resource node, determines that the business task on the faulty resource node is a task to be processed, and determines the priority of the task to be processed, so as to perform fault-tolerant processing on the task to be processed in order of priority from high to low; when performing fault-tolerant processing on the task to be processed, a trained fault-tolerant processing action generation model is used to generate multiple fault-tolerant processing actions for the task to be processed according to the status information at the current moment, and an availability evaluation is performed on each fault-tolerant processing action, and the actual fault-tolerant processing action is determined according to the availability evaluation result, and the actual fault-tolerant processing action is performed on the task to be processed, so that when a resource node fails in the distributed system, the tasks on the resource node can be feasibly scheduled to ensure the continuous execution of these tasks, without causing a long service interruption due to the failure of the node or the inability to continue to provide services after migration due to migration to an inappropriate node, thereby ensuring the continuous operation of the services provided by the distributed system and effectively improving the reliability of the distributed system.

[0064] Based on the above embodiment, the embodiment of the present invention further illustrates the step of determining the priority of the task to be processed.

[0065] In some optional implementations of the embodiments of the present invention, determining the priority of the task to be processed in S103 may include: determining the fault impact parameter of the faulty resource node according to the fault type of the faulty resource node; determining the task evaluation parameter of the task to be processed according to the load status of the task to be processed on the faulty resource node; and determining the priority score of the task to be processed according to the fault impact parameter and the task evaluation parameter.

[0066] The embodiment of the present invention measures the priority of the task to be processed from two aspects: the fault impact of the fault resource node and the load state of the task to be processed on the fault resource node.

[0067] Among them, determining the fault impact parameter of the faulty resource node according to the fault type of the faulty resource node may include: determining the fault duration prediction value of the faulty resource node according to the historical operation data of the faulty resource node; and using the fault duration prediction value as the fault impact parameter.

[0068] In the above embodiment of the present invention, it is introduced that by counting the historical data of each resource node, the law of the fault duration corresponding to each fault type can be obtained. Then, determining the predicted value of the fault duration of the faulty resource node according to the historical data corresponding to the fault type of the faulty resource node may include: determining the expected value of the fault duration of the faulty resource node according to the fault type of the faulty resource node and the corresponding fault duration probability distribution model; using the expected value of the fault duration as the predicted value of the fault duration; wherein the fault duration probability distribution model corresponding to the fault type is determined according to the historical operation data of the resource node.

[0069] In the embodiment of the present invention, the expected value of the fault duration can be obtained by Calculated, that is, given Moment Resource Node Under the conditions of the actual fault types, the resource nodes are determined according to the probability distribution model of the fault duration corresponding to these fault types. The expected value of the fault duration.

[0070] In some optional implementations of the embodiments of the present invention, determining the task evaluation parameters of the task to be processed according to the load state of the task to be processed on the faulty resource node may include: taking the resource occupancy parameter of the task to be processed on the faulty resource node and the remaining execution time of the task to be processed on the faulty resource node as the load state of the task to be processed on the faulty resource node, and determining the task evaluation parameters according to the resource occupancy parameter and the remaining execution time. That is, the task evaluation parameters of the task to be processed may be determined by considering only the workload.

[0071] The step of determining the resource occupancy parameter may include: determining the resource occupancy parameter according to the amount of multiple types of resources occupied by the task to be processed on the faulty resource node. Determining the task evaluation parameter according to the resource occupancy parameter and the remaining execution time may include: taking the ratio of the resource occupancy parameter to the remaining execution time as the task evaluation parameter. Then the task evaluation parameter of the task to be processed can be calculated by the following formula: ; in, , , Respectively represent business tasks The amount of computing resources, memory resources, and persistent storage resources used by a single copy of Indicates business tasks The number of replicas running on the resource node. Indicates business tasks The occupation period of the current resource. Among them, is the same unit, , , , , can all be normalized parameters. It can be used to indicate the average amount of resources consumed by pending tasks on a faulty resource node per unit time.

[0072] In the embodiment of the present invention, the fault impact parameter may be a predicted fault duration of a faulty resource node, and the task evaluation parameter may be a resource consumption per unit time of the task to be processed at the faulty resource node. Then, determining the priority score of the task to be processed based on the fault impact parameter and the task evaluation parameter may include: taking the ratio of the resource consumption per unit time to the predicted fault duration as the priority score. The priority score of the task to be processed may be calculated by the following formula: ; in, express Time Fault Resource Node Pending tasks on The priority score.

[0073] When comparing the priority scores, all pending tasks on all faulty resource nodes in the target distributed system can be uniformly sorted, or they can be grouped according to the parallel capability of fault-tolerant scheduling and sorted by priority scores in the groups.

[0074] In some other optional implementations of the embodiments of the present invention, determining the task evaluation parameter of the task to be processed according to the load state of the task to be processed on the faulty resource node may also include: determining the task evaluation parameter according to the load state of the task to be processed on the faulty resource node and the task type priority parameter of the task to be processed. That is to say, when determining the priority score of the task to be processed, the task type priority of the task to be processed may also be combined, and specifically, the task type priority score of the task to be processed may be added to the priority score calculation formula of the task to be processed.

[0075] Based on the above embodiments, the embodiments of the present invention further illustrate the fault-tolerant processing action generation model.

[0076] In the above embodiment of the present invention, it is introduced that the fault-tolerant processing action generation model can be trained by reinforcement learning or deep reinforcement learning. Then the training steps of the fault-tolerant processing action generation model may include: inputting sample state information into the fault-tolerant processing action generation model, and outputting sample fault-tolerant processing actions for the sample state information; after executing the sample fault-tolerant processing action, obtaining updated sample state information; determining the environmental reward value according to the updated sample state information and the reward function; taking a group of sample fault-tolerant processing actions, sample fault-tolerant processing actions, updated sample state information and environmental reward values ​​as a training sample; and using the training sample to update the model parameters of the fault-tolerant processing action generation model.

[0077] In the embodiment of the present invention, the environmental reward value may adopt the fault-tolerant cost of executing the fault-tolerant processing action. In some optional implementations of the embodiment of the present invention, the fault-tolerant cost of executing the fault-tolerant processing action may be measured from the perspective of the time cost required to execute the fault-tolerant processing action, and the reward function may be a function of the fault-tolerant processing time of the task.

[0078] like Figure 2 As shown in the figure, fault-tolerant processing actions can be divided into three categories, namely task migration actions, task waiting actions, and task scaling actions. The time cost required to execute fault-tolerant processing actions can be determined by combining the service interruption time and transmission time caused by executing fault-tolerant processing actions. For the above three types of actions: Task migration action: Migrate the pending tasks on the current faulty resource node to the resource node that is operating normally. The main time cost incurred in this process includes data transmission time and service recovery time.

[0079] Task waiting action: If the fault can be recovered in a short time, migration is not required and pending tasks are interrupted until the faulty resource node is recovered. The main cost of this process is the waiting time for service interruption.

[0080] Task scaling action: If the task to be processed has the same copy on other resource nodes, the services on other resource nodes can be scaled, and the service of the failed resource node will be terminated immediately. This process does not generate data migration, and the main cost is the service recovery time.

[0081] The fault-tolerant processing time may include a first processing time for a task migration action, a second processing time for a task waiting action, and a third processing time for a task scaling action; wherein the task migration action is to migrate the task to be processed from the faulty resource node to the first resource node, and the first processing time includes the data transmission time and the service recovery time at the first resource node; the task waiting action is to set the task to be processed to a service interruption state at the faulty resource node, and the second processing time is the service interruption waiting time; the task scaling action is to start the task to be processed at the second resource node where a copy of the task to be processed exists and terminate the task to be processed at the faulty resource node, and the third processing time is the service recovery time at the second resource node.

[0082] According to the training steps of the fault-tolerant processing action generation model and the types of fault-tolerant processing actions introduced in the embodiment of the present invention, the specific training steps of a fault-tolerant processing action generation model are introduced below.

[0083] set up The state parameters of the target distributed system observed at all times are ,in, express The basic environment status information of the target distributed system at all times (including network topology information, resource status information of resource nodes, and network status information between resource nodes), express The collection of various business tasks being run by all resource nodes in the target distributed system at any given moment. express Fault information of all faulty resource nodes in the target distributed system at all times.

[0084] set up The decision variable at time is ,in, Indicates the task migration action. Indicates that the task is waiting for action. Indicates a task scaling action.

[0085] Among them, you can configure and , When both are 0, it means that the pending tasks will be migrated to the resource node On. When and , When both are 0, it means that the pending tasks will be kept on the faulty resource node and no other operations will be performed. and , When both are 0, it means that the pending task will be processed on another resource node. Stretch on.

[0086] After executing the fault-tolerant processing action, the state information in the target distributed system is collected again, and the environmental reward value can be obtained , represents the time cost required to perform fault-tolerant processing actions to restore the service of pending tasks. , Indicates the first processing time of the task migration action, Indicates the second processing time of the task waiting action. represents the third processing time of the task scaling action. Usually, since the decision variable is Only one type of fault-tolerant processing action can be executed within a period of time, and the time costs of the three types of fault-tolerant processing actions are mutually exclusive.

[0087] Construct an online decision generation network for deep reinforcement learning, that is, the initial fault-tolerant processing action generation model. The fault-tolerant processing action generation model can adopt a deep Q learning model (DQN). The initial fault-tolerant processing action generation model is , Generate model parameters for fault-tolerant processing actions. Input fault-tolerant processing action generation model, output fault-tolerant processing action After execution, the current stage environmental reward value is calculated through the reward function , the system enters the next observation state .

[0088] Multiple training samples are collected in the above way. Stored in the buffer pool D. When the number of training samples stored in the buffer pool D reaches the preset number of samples, the parameter update step of the fault-tolerant processing action generation model is entered.

[0089] In the process of updating the parameters of the fault-tolerant action generation model, M training samples are randomly selected from the cache pool D, and the target Q value is calculated according to the following formula: ; in, is a real coefficient between 0 and 1.

[0090] The loss function of the fault-tolerant action generation model can be: .

[0091] Update network parameters based on gradient back propagation , until the network converges. After convergence, the fault-tolerant processing action generation model outputs decision actions, namely fault-tolerant processing actions, based on the state information of the target distributed system at the current moment, and makes scheduling management decisions for pending tasks on faulty resource nodes at different stages.

[0092] Based on the above embodiment, the embodiment of the present invention further illustrates the action filter.

[0093] To ensure the availability of the fault-tolerant processing actions actually performed, such as Figure 2 As shown, the embodiment of the present invention first uses the fault-tolerant processing action generation model to generate multiple fault-tolerant processing actions, and then uses the action filter to filter out the available actual fault-tolerant processing actions.

[0094] In the embodiment of the present invention, the availability evaluation of each fault-tolerant processing action in S104 and the determination of the actual fault-tolerant processing action according to the availability evaluation result may include: the availability evaluation of the fault-tolerant processing actions is performed in descending order of the corresponding environmental reward values, until a fault-tolerant processing action that meets the availability evaluation condition is obtained as the actual fault-tolerant processing action. That is, the fault-tolerant processing action with the largest environmental reward value may be selected from the available fault-tolerant processing actions as the actual fault-tolerant processing action.

[0095] In a specific implementation, using an action filter to perform availability evaluation on a fault-tolerant processing action may include performing a security evaluation on the fault-tolerant processing action. If the fault-tolerant processing action meets the preset security conditions, then it is determined that the fault-tolerant processing action meets the availability evaluation conditions. Then, it can be set express The safety action indicator at the moment first sorts the fault-tolerant processing actions output by the fault-tolerant processing action generation model according to the environmental reward value from large to small, selects the first K fault-tolerant processing actions, and then checks whether they meet the availability evaluation conditions one by one according to the environmental reward value from large to small. If not, , until you select The fault-tolerant processing action is taken as the actual fault-tolerant processing action and executed.

[0096] On the basis of the above embodiments, in the process of obtaining the actual fault-tolerant processing action by using the fault-tolerant processing action generation model and the action filter, in addition to the time cost, factors such as energy consumption cost and service quality may also be considered.

[0097] Then in some optional implementations of the embodiments of the present invention, when the reward function of the fault-tolerant processing action generation model is a function of time cost, the use of an action filter to perform availability evaluation on the fault-tolerant processing action may include: sorting the fault-tolerant processing actions output by the fault-tolerant processing action generation model from large to small according to the environmental reward value, selecting the first K fault-tolerant processing actions, determining the energy consumption cost and service quality parameters for each fault-tolerant processing action respectively, determining the availability evaluation result of the fault-tolerant processing action according to the environmental reward value, the energy consumption cost and the service quality parameters, and taking the fault-tolerant processing action with the largest availability evaluation result as the actual fault-tolerant processing action. Among them, the energy consumption cost can be determined based on the computing resources, storage resources, network resources and power consumption required to execute the fault-tolerant processing action to restore the task to be processed. The service quality parameter can be determined based on the execution result of the task to be processed after the fault-tolerant processing action is executed to restore the task to be processed, and can specifically be the model accuracy.

[0098] By using the above-mentioned fault-tolerant processing action generation model and action filter to generate actual fault-tolerant processing actions, the training process and reasoning process of the fault-tolerant processing action generation model can be simplified, and the optimal actual fault-tolerant processing action can be quickly obtained while comprehensively considering the multi-dimensional goals of time cost, energy consumption cost and service quality.

[0099] In some other optional implementations of the embodiments of the present invention, the reward function of the fault-tolerant processing action generation model can also be set as a function of time cost, energy consumption cost and service quality parameters. Then, using the action filter to perform availability evaluation on the fault-tolerant processing action can include: sorting the fault-tolerant processing actions output by the fault-tolerant processing action generation model from large to small according to the environmental reward value, selecting the first K fault-tolerant processing actions, and then checking them one by one in order from large to small according to the environmental reward value to see whether they meet the preset safety conditions, and then determining that the fault-tolerant processing action meets the availability evaluation conditions. The preset safety conditions can include: if the fault-tolerant processing action is a task migration action or a task scaling action, then when the task is migrated or the task is scaled to another resource node, it meets the requirements for executing the task to be processed when the fault-tolerant processing action is about to be executed, including being in a normal operating state and having the resources required to execute the task to be processed.

[0100] The above-mentioned fault-tolerant processing action generation model and action filter are used to generate the actual fault-tolerant processing action scheme. On the basis of generating the fault-tolerant processing action based on the multi-dimensional goals of comprehensive time cost, energy consumption cost and service quality, the fault-tolerant processing action is checked again before the actual execution to see whether it meets the preset safety conditions, thereby ensuring that the actual fault-tolerant processing action with the best environmental reward and high availability is selected.

[0101] It should be noted that in the embodiments of the fault-tolerant processing methods of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the fault-tolerant processing methods provided in the embodiments of the present invention.

[0102] The above describes in detail various embodiments corresponding to the fault-tolerant processing method. On this basis, the present invention also discloses a fault-tolerant processing device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0103] The fault-tolerant processing device provided by the embodiment of the present invention may include: A monitoring unit is used to monitor the status information of each resource node of the target distributed system; after detecting a faulty resource node, the business task on the faulty resource node is determined as a task to be processed; A task priority determination unit, used to determine the priority of the tasks to be processed, and perform fault-tolerant processing on the tasks to be processed in order of priority from high to low; The fault-tolerant action generation unit is used to generate multiple fault-tolerant processing actions for the task to be processed according to the current state information using the trained fault-tolerant processing action generation model when performing fault-tolerant processing for the task to be processed, perform availability evaluation on each fault-tolerant processing action, determine the actual fault-tolerant processing action according to the availability evaluation result, and perform the actual fault-tolerant processing action on the task to be processed.

[0104] It should be noted that in each implementation of the fault-tolerant processing device provided in the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0105] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0106] Figure 3 A schematic diagram of the structure of a fault-tolerant processing device provided by an embodiment of the present invention.

[0107] like Figure 3As shown, the fault-tolerant processing device provided by an embodiment of the present invention includes: a memory 310, used to store a computer program 311; a processor 320, used to execute the computer program 311, and when the computer program 311 is executed by the processor 320, the steps of the fault-tolerant processing method provided by any of the above embodiments are implemented.

[0108] Among them, the processor 320 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 320 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array. The processor 320 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 320 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 320 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0109] The memory 310 may include one or more non-volatile storage media, which may be non-transitory. The memory 310 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 310 is at least used to store the following computer program 311, wherein the computer program 311, after being loaded and executed by the processor 320, can implement the relevant steps in the fault-tolerant processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 310 may also include an operating system 312 and data 313, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 312 may be Windows or other types of operating systems. Data 313 may include, but is not limited to, the data involved in the above method.

[0110] In some embodiments, the fault-tolerant processing device may further include a display screen 330 , a power supply 340 , a communication interface 350 , an input / output interface 360 ​​, a sensor 370 , and a communication bus 380 .

[0111] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the fault-tolerant processing device, and may include more or less components than those shown in the figure.

[0112] The fault-tolerant processing device provided in the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the fault-tolerant processing method provided in the above embodiment, and the effect is the same as above.

[0113] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the fault-tolerant processing method provided in any one of the above embodiments can be implemented.

[0114] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0115] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the fault-tolerant processing method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0116] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the fault-tolerant processing method provided in any one of the above embodiments.

[0117] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the fault-tolerant processing method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0118] The above is a detailed introduction to a fault-tolerant processing method, device, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can refer to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0119] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A fault-tolerant processing method, characterized in that: include: Monitor the status information of each resource node of the target distributed system; After detecting a faulty resource node, determining that a service task on the faulty resource node is a task to be processed; Determine the priority of the pending tasks, and perform fault-tolerant processing on the pending tasks in order of priority from high to low; When performing fault-tolerant processing on the task to be processed, a trained fault-tolerant processing action generation model is used to generate multiple fault-tolerant processing actions for the task to be processed according to the status information at the current moment, and an availability evaluation is performed on each of the fault-tolerant processing actions. The actual fault-tolerant processing action is determined according to the availability evaluation result, and the actual fault-tolerant processing action is executed on the task to be processed.

2. The fault-tolerant processing method according to claim 1, characterized in that: Determine the priority of the pending tasks, including: Determine a fault impact parameter of the faulty resource node according to the fault type of the faulty resource node; Determine a task evaluation parameter of the task to be processed according to a load state of the task to be processed on the faulty resource node; The priority score of the task to be processed is determined according to the fault impact parameter and the task evaluation parameter.

3. The fault-tolerant processing method according to claim 2, characterized in that: Determining a fault impact parameter of the faulty resource node according to the fault type of the faulty resource node includes: Determine a predicted value of the fault duration of the faulty resource node according to historical operation data of the faulty resource node; The predicted value of the fault duration is used as the fault impact parameter.

4. The fault-tolerant processing method according to claim 3, characterized in that: Determining a predicted value of a fault duration of the faulty resource node according to historical data corresponding to the fault type of the faulty resource node includes: Determine an expected value of the fault duration of the faulty resource node according to the fault type of the faulty resource node and a corresponding fault duration probability distribution model; Using the expected value of the fault duration as the predicted value of the fault duration; The fault duration probability distribution model corresponding to the fault type is determined according to the historical operation data of the resource node.

5. The fault-tolerant processing method according to claim 2, characterized in that: Determining a task evaluation parameter of the task to be processed according to a load state of the task to be processed on the faulty resource node includes: The resource occupancy parameter of the task to be processed on the faulty resource node and the remaining execution time of the task to be processed on the faulty resource node are used as the load state of the task to be processed on the faulty resource node, so as to determine the task evaluation parameter according to the resource occupancy parameter and the remaining execution time.

6. The fault-tolerant processing method according to claim 5, characterized in that: The step of determining the resource occupancy parameter comprises: Determine the resource occupation parameter according to the amount of multiple types of resources occupied by the task to be processed on the faulty resource node; Determining the task evaluation parameter according to the resource occupancy parameter and the remaining execution time includes: The ratio of the resource occupancy parameter to the remaining execution time is used as the task evaluation parameter.

7. The fault-tolerant processing method according to claim 2, characterized in that: Determining a task evaluation parameter of the task to be processed according to a load state of the task to be processed on the faulty resource node includes: The task evaluation parameter is determined according to the load state of the task to be processed on the faulty resource node and the task type priority parameter of the task to be processed.

8. The fault-tolerant processing method according to claim 2, characterized in that: The fault impact parameter is a predicted value of the fault duration of the fault resource node; The task evaluation parameter is the resource consumption per unit time of the task to be processed at the faulty resource node; Determining the priority score of the task to be processed according to the fault impact parameter and the task evaluation parameter includes: The ratio of the resource consumption value per unit time to the predicted fault duration value is used as the priority score.

9. The fault-tolerant processing method according to claim 1, characterized in that: The training step of the fault-tolerant processing action generation model includes: Inputting sample state information into the fault-tolerant processing action generation model, and outputting a sample fault-tolerant processing action for the sample state information; After executing the sample fault tolerance processing action, obtaining the updated sample status information; Determine an environment reward value according to the updated sample state information and reward function; Taking a group of the sample fault-tolerant processing actions, the sample fault-tolerant processing actions, the updated sample state information and the environmental reward value as a training sample; The model parameters of the fault-tolerant processing action generation model are updated using the training samples.

10. The fault-tolerant processing method according to claim 9, characterized in that: The reward function is a function of the fault-tolerant processing time of the task.

11. The fault-tolerant processing method according to claim 10, characterized in that: The fault-tolerant processing time includes a first processing time for a task migration action, a second processing time for a task waiting action, and a third processing time for a task scaling action; Among them, the task migration action is to migrate the pending task from the faulty resource node to the first resource node, and the first processing time includes the data transmission time and the service recovery time at the first resource node; the task waiting action is to set the pending task to a service interruption state at the faulty resource node, and the second processing time is the service interruption waiting time; the task scaling action is to start the pending task at the second resource node where a copy of the pending task exists and terminate the pending task at the faulty resource node, and the third processing time is the service recovery time at the second resource node.

12. The fault-tolerant processing method according to claim 1, characterized in that: Performing availability evaluation on each of the fault-tolerant processing actions, and determining an actual fault-tolerant processing action according to the availability evaluation result, including: The availability of the fault-tolerant processing actions is evaluated in descending order of the corresponding environmental reward values, until the fault-tolerant processing actions satisfying the availability evaluation conditions are obtained as the actual fault-tolerant processing actions.

13. A fault-tolerant processing device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the fault-tolerant processing method according to any one of claims 1 to 12 are implemented.

14. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the fault-tolerant processing method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the fault-tolerant processing method according to any one of claims 1 to 12 are implemented.

Images