A method, device and medium for handling host failure

Abstract

The present invention discloses a method, device and medium for processing host failures, which are applicable to the field of server technology. Priority is given to the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue; the recovery tasks of the priority queue are restored according to the fuse rule and the current limiting rule to complete the processing of the host failure. In the event of a large-scale host failure, the order of cloud host recovery after the failure is determined by priority sorting, and then the fuse rule is used to terminate the host failure recovery process in time when the cloud platform's carrying capacity is insufficient or an error occurs, so as to avoid the risk impact on the entire cloud platform; through the current limiting rule, when the cloud platform fails, the host failure is guaranteed to be restored in a stable and orderly manner within the range that the platform can bear. In summary, by processing the recovery tasks of both the failed host node and the corresponding cloud hosts, the cloud host is guaranteed to be restored in a stable and orderly manner, thereby improving the high availability of the cloud platform.

Description

Technical Field

[0001] The present invention relates to the technical field of servers, and in particular to a method, device and medium for processing host failures. Background Art

[0002] Cloud computing platform is a service based on hardware resources and software resources, providing computing, network and storage capabilities. Cloud platform manages various physical servers through hardware virtualization technology to allocate resources to users on demand. In order to achieve stable operation of cloud platform and provide services, cloud platform needs to have necessary fault detection capabilities and certain self-recovery capabilities.

[0003] At present, cloud platforms provide basic high availability capabilities in computing, networking, storage, and management. For large-scale cluster environments, computing high availability (HA) has gradually become a must-have function. When a host node fails and can no longer provide services, the multiple cloud hosts corresponding to the host node can be evacuated to other physical hosts through the high availability function to ensure that the cloud host can be used normally and user services are not affected. The existing recovery process evacuates cloud hosts from failed host nodes to other healthy host nodes, which is completely applicable to the failure of a single host node. When a large-scale host fails, the corresponding recovery process for multiple cloud hosts lacks the corresponding processing capacity. If multiple cloud hosts are evacuated at the same time, it may exceed the tolerance of the cloud platform, and the stable and orderly recovery of the cloud hosts cannot be guaranteed, which reduces high availability and even brings risks to the entire cloud platform.

[0004] Therefore, how to improve the high availability of cloud platforms is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device and medium for processing host failures to improve the high availability of a cloud platform.

[0006] In order to solve the above technical problems, the present invention provides a method for processing a host failure, comprising:

[0007] Get the cloud hosts corresponding to the failed host node;

[0008] Prioritize the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each cloud host to obtain a priority queue;

[0009] According to the fuse rules and current limiting rules, the recovery tasks of the priority queue are restored to complete the processing of the host failure.

[0010] Preferably, the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each cloud host are prioritized to obtain a priority queue, including:

[0011] The recovery task corresponding to the failed host node is used as the node recovery task, and the recovery task corresponding to each cloud host is used as the cloud host recovery task;

[0012] The node recovery tasks and the cloud host recovery tasks are prioritized respectively to obtain the node recovery task queue and the cloud host recovery task queue.

[0013] Preferably, the circuit breaker rules are formulated through the following steps:

[0014] The circuit breaker rules include cloud host circuit breaker rules and node circuit breaker rules, wherein the cloud host circuit breaker rules include at least the first circuit breaker rule and the second circuit breaker rule, and the node circuit breaker rules include at least the third circuit breaker rule, the fourth circuit breaker rule and the fifth circuit breaker rule;

[0015] Among them, the first fuse rule is that when the number of failed recovery tasks of each cloud host exceeds the first threshold, the cloud host corresponding to the current recovery is obtained and the remaining unrecovered cloud hosts are determined according to the cloud host corresponding to the current recovery, and the remaining unrecovered cloud hosts are controlled not to be recovered;

[0016] The second circuit breaker rule is a rule that when the number of recovery times of the cloud host recovery task of the current cloud host exceeds the second threshold within the preset time, the cloud host corresponding to the cloud host recovery task will not be restored;

[0017] The third fuse rule is a rule that when the number of faulty host nodes exceeds the third threshold, the faulty host nodes will not be restored;

[0018] The fourth fuse rule is that when the number of failed node recovery tasks of the faulty host node exceeds the fourth threshold, the faulty host node corresponding to the current recovery is obtained and the faulty host node to be recovered is determined according to the faulty host node corresponding to the current recovery, and the rule that the faulty host node to be recovered is controlled not to perform recovery processing;

[0019] The fifth fuse rule is a rule that when the number of node recovery tasks of the current faulty host node exceeds a fifth threshold within a predetermined time, the current faulty host node will not be recovered.

[0020] Preferably, the current limiting rules are formulated through the following steps:

[0021] The current limiting rules include cloud host current limiting rules and node current limiting rules;

[0022] The cloud host current limiting rule is to obtain the number of cloud hosts corresponding to the current faulty host node within a specified time. If the number of cloud hosts exceeds the first current limiting threshold, the cloud hosts are evacuated according to the priority of the recovery tasks of each cloud host corresponding to the current faulty host node.

[0023] The node current limiting rule is a rule for evacuating the multiple failed host nodes according to the priorities of the recovery tasks corresponding to the multiple failed host nodes when the number of the failed host nodes exceeds the second current limiting threshold within a specific time.

[0024] Preferably, the recovery task of the priority queue is recovered according to the fuse rule and the current limiting rule to complete the processing of the host failure, including:

[0025] Determine whether the recovery task of the priority queue meets the circuit breaker rules;

[0026] If the circuit breaker rules are met, determine whether the recovery task of the priority queue meets the current limiting rules;

[0027] If the flow limiting rules are met, the recovery task will be restored;

[0028] If any of the current limiting rules is not met, the state of the corresponding recovery task is set to a waiting state and the process returns to the step of prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue;

[0029] If any of the circuit breaker rules are not met, the corresponding recovery task will not be recovered.

[0030] Preferably, the recovery task of the priority queue is recovered according to the fuse rule and the current limiting rule to complete the processing of the host failure, including:

[0031] Determine whether the recovery task of the priority queue meets the current limiting rules;

[0032] If the flow limit rules are met, determine whether the recovery task meets the circuit breaker rules;

[0033] If the circuit breaker rules are met, the recovery task will be restored;

[0034] If any of the circuit breaker rules are not met, the recovery task will not be restored;

[0035] If any of the current limiting rules is not met, the state of the corresponding recovery task is set to a waiting state and the process returns to the step of prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue.

[0036] Preferably, it also includes:

[0037] The recovery tasks that are not to be restored are collected into a table for easy viewing by users.

[0038] In order to solve the above technical problems, the present invention further provides a host failure processing device, comprising:

[0039] An acquisition module is used to obtain each cloud host corresponding to the failed host node;

[0040] A sorting module is used to sort the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each cloud host to obtain a priority queue;

[0041] The recovery module is used to recover the recovery tasks of the priority queue according to the fuse rules and current limiting rules to complete the processing of the host failure.

[0042] In order to solve the above technical problems, the present invention further provides a host failure processing device, comprising:

[0043] Memory for storing computer programs;

[0044] A processor is used to implement the steps of the host failure processing method as described above when executing a computer program.

[0045] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the host failure processing method as described above are implemented.

[0046] The present invention provides a method for handling host failures, including obtaining each cloud host corresponding to the failed host node; prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue; and restoring the recovery tasks of the priority queue according to the fuse rule and the current limiting rule to complete the handling of the host failure. In the event of a large-scale host failure, this method first determines the order of cloud host recovery after the failure by priority sorting, and then uses the fuse rule to terminate the host failure recovery process in time when the cloud platform's carrying capacity is insufficient or an error occurs, thereby avoiding the risk impact on the entire cloud platform; through the current limiting rule, when the cloud platform fails, it is ensured that the host failure is restored stably and orderly within the range that the platform can bear. In summary, by processing the recovery tasks of both the failed host node and the corresponding cloud hosts, the cloud host is guaranteed to be restored stably and orderly, thereby improving the high availability of the cloud platform.

[0047] In addition, the present invention also provides a host failure processing device and medium, which have the same beneficial effects as the above-mentioned host failure processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a method for handling a host failure provided by an embodiment of the present invention;

[0050] Figure 2 A structural diagram of a host failure processing device provided by an embodiment of the present invention;

[0051] Figure 3 A structural diagram of another host failure processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The core of the present invention is to provide a method, device and medium for processing host failures to improve the high availability of the cloud platform.

[0054] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0055] It should be noted that the host failure processing method provided by the present invention can be applied to multi-node failure processing and single-node failure processing. The basic idea of ​​implementing high availability includes three steps: monitoring, isolation and recovery; monitoring is used to detect faulty host nodes, isolation is used to isolate faulty host nodes, and recovery is used to evacuate cloud hosts on faulty host nodes to other healthy host nodes. The present invention is only for the recovery process of high availability. The existing manufacturers are basically similar in the recovery process, which only mechanically evacuates cloud hosts from faulty host nodes to other healthy host nodes, and does not effectively control the recovery process. For example, multiple cloud hosts are evacuated at the same time, which makes the evacuation channel congested, resulting in packet loss or inability to evacuate, affecting the normal use of the cloud platform.

[0056] Figure 1 A flowchart of a method for processing a host failure provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0057] S11: Obtain each cloud host corresponding to the failed host node.

[0058] It is understandable that the faulty host node is a server host that has a fault, which is used as a node in multiple servers, and one host server corresponds to multiple cloud hosts. The faulty host node obtained in this embodiment can be a single host server or multiple host servers. When there are multiple host servers, the corresponding cloud hosts are multiple cloud hosts.

[0059] S12: Prioritize the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each cloud host to obtain a priority queue.

[0060] Specifically, the recovery tasks corresponding to the faulty host node and the recovery tasks corresponding to each cloud host are obtained. The two recovery tasks can be merged and the priorities can be determined according to the different resource types. For example, the recovery tasks of the cloud host are determined to have a higher priority and the recovery tasks of the faulty host node are determined to have a lower priority, and then the same resource type is subdivided according to the evacuation capacity. It is also possible to process two different recovery tasks in parallel, set two priority queues, and perform subsequent recovery processes according to the priorities of their respective types. For example, there are 5 faulty host nodes, and their corresponding recovery tasks are also 5; 1 faulty host node corresponds to 6 cloud hosts, and the corresponding recovery task for 1 cloud host is 1, that is, combined with the faulty host node, there are a total of 5 recovery tasks for the faulty host node and 30 recovery tasks for the cloud host. Sort the two recovery tasks by their respective priorities to obtain the corresponding priority queues.

[0061] Priority sorting can be performed by calculating the priority of each recovery task according to a specific algorithm, queuing the task in the task queue to be executed and waiting for processing; it can also be sorted according to specific attributes, such as the priority of evacuation capacity, etc. The present invention does not make specific limitations, as long as the priority order can be sorted. The recovery tasks with higher priorities are processed first, and the recovery tasks with lower priorities are processed later.

[0062] S13: Recover the recovery task of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure.

[0063] The recovery task of the priority queue obtained in step S12 is then restored according to the fuse rule and current limiting rule. The role of service fuse is similar to that of a household fuse. When a service main line is unavailable or the response times out, in order to prevent the entire system from avalanching, the call to it is temporarily stopped. Once the current service fuses the downstream service, when a request arrives, the current service no longer calls the downstream service, and returns directly using the set strategy (such as building default values).

[0064] The present invention is only aimed at stopping the execution of the fuse rule, that is, in the fuse link, firstly, the fuse is initialized and the corresponding fuse rule is loaded according to the resource type of the recovery task, and then, the recovery task is checked and judged by the fuse rule. When the recovery task cannot pass a certain rule, the recovery task is set to the ignored state and is no longer executed. When all the fuse rules are passed, the recovery task can proceed to the next link.

[0065] Current limiting means that when the query rate (Query Per Second, QPS) of the upstream service exceeds the threshold, the request volume of the upstream service is limited through certain strategies (such as delay processing, rejection processing) to ensure that the service is not overwhelmed, so as to maintain stable service. Common current limiting algorithms include sliding window, token bucket or leaky bucket, etc. The current limiting link of the embodiment of the present invention first initializes the current limiter and loads the corresponding current limiting rules for the resource type of the recovery task. Secondly, the recovery task is checked and judged by the current limiting rules in turn. When the recovery task cannot pass a certain rule, the recovery task is set to a waiting state and returns to the queue of tasks to be executed. When all the current limiting rules are passed, the recovery task can continue to execute.

[0066] It should be noted that the present invention does not limit the priority order of the fuse rule and the current limiting rule. The fuse rule can be performed first and then the current limiting rule, or the current limiting rule can be performed first and then the fuse rule. It can be set according to the actual situation. For the two rules, this embodiment is performed at the cloud host and the fault host node level, so the resource types of the two rules are divided into two categories, one is based on the cloud host, and the other is based on the fault host node. There can be multiple rules for a type of rule in a rule, and the present invention does not make specific limitations.

[0067] A method for handling host failures provided by an embodiment of the present invention includes obtaining each cloud host corresponding to a failed host node; prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue; and restoring the recovery tasks of the priority queue according to the fuse rule and the current limiting rule to complete the handling of the host failure. In the event of a large-scale host failure, this method first determines the order of cloud host recovery after the failure by priority sorting, and then uses the fuse rule to terminate the host failure recovery process in time when the cloud platform's carrying capacity is insufficient or an error occurs, thereby avoiding the risk impact on the entire cloud platform; through the current limiting rule, when a cloud platform fails, it is ensured that the host failure is restored stably and orderly within the range that the platform can bear. In summary, by processing the recovery tasks of both the failed host node and the corresponding cloud hosts, the cloud host is guaranteed to be restored stably and orderly, thereby improving the high availability of the cloud platform.

[0068] Based on the above embodiment, in step S12, the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each cloud host are prioritized to obtain a priority queue, including:

[0069] The recovery task corresponding to the failed host node is used as the node recovery task, and the recovery task corresponding to each cloud host is used as the cloud host recovery task;

[0070] The node recovery tasks and the cloud host recovery tasks are prioritized respectively to obtain the node recovery task queue and the cloud host recovery task queue.

[0071] As a preferred embodiment, priority is sorted according to different resource types, and the recovery tasks corresponding to the failed host node are used as node recovery tasks, and the recovery tasks corresponding to each cloud host are used as cloud host recovery tasks. The two types of recovery tasks are prioritized respectively, and the obtained priority queues include two types, one is the node recovery task queue, and the other is the cloud host recovery task queue.

[0072] In addition, the priority sorting method can be determined by a specific priority algorithm, or the corresponding priority can be determined according to the evacuation capabilities of different recovery tasks. The present invention does not make specific limitations, and it only needs to determine the priority order.

[0073] The embodiment of the present invention provides a priority queue for prioritizing the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each cloud host, which is convenient for performing subsequent recovery processes according to the two different recovery tasks. Through priority queuing, the relative order of evacuation of cloud hosts after a failure can be determined, and the cloud hosts are restored on new host nodes in order, starting with the cloud host with the highest grayscale priority, and then restored in turn according to the priority order until all recovery tasks are restored.

[0074] Based on the above embodiment, the formulation of the fuse rule in step S13 is carried out through the following steps:

[0075] The circuit breaker rules include cloud host circuit breaker rules and node circuit breaker rules, wherein the cloud host circuit breaker rules include at least the first circuit breaker rule and the second circuit breaker rule, and the node circuit breaker rules include at least the third circuit breaker rule, the fourth circuit breaker rule and the fifth circuit breaker rule;

[0076] Among them, the first fuse rule is that when the number of failed recovery tasks of each cloud host exceeds the first threshold, the cloud host corresponding to the current recovery is obtained and the remaining unrecovered cloud hosts are determined according to the cloud host corresponding to the current recovery, and the remaining unrecovered cloud hosts are controlled not to be recovered;

[0077] The second circuit breaker rule is a rule that when the number of recovery times of the cloud host recovery task of the current cloud host exceeds the second threshold within the preset time, the cloud host corresponding to the cloud host recovery task will not be restored;

[0078] The third fuse rule is a rule that when the number of faulty host nodes exceeds the third threshold, the faulty host nodes will not be restored;

[0079] The fourth fuse rule is that when the number of failed node recovery tasks of the faulty host node exceeds the fourth threshold, the faulty host node corresponding to the current recovery is obtained and the faulty host node to be recovered is determined according to the faulty host node corresponding to the current recovery, and the rule that the faulty host node to be recovered is controlled not to perform recovery processing;

[0080] The fifth fuse rule is a rule that when the number of node recovery tasks of the current faulty host node exceeds a fifth threshold within a predetermined time, the current faulty host node will not be recovered.

[0081] Specifically, the circuit breaker rules are divided into two types according to the resource type of the recovery task. One is the circuit breaker rule based on the cloud host, and the other is the circuit breaker rule based on the faulty host node, namely, the cloud host circuit breaker rule and the node circuit breaker rule. The cloud host circuit breaker rule includes at least the first circuit breaker rule and the second circuit breaker rule, and the node circuit breaker rule includes at least the third circuit breaker rule, the fourth circuit breaker rule and the fifth circuit breaker rule, that is, other rules may also be included.

[0082] The first fuse rule is that when the data in the table of cloud hosts that need to be evacuated due to a single faulty host node exceeds the first threshold value when recovering, it means that the remaining cloud hosts on the faulty host node no longer have the ability to continue recovery, and the cloud hosts that have not been evacuated will no longer be evacuated. Each cloud host corresponds to a faulty host node. When the number of failed recovery tasks of its corresponding cloud hosts exceeds the first threshold value, the remaining unrecovered cloud hosts are determined based on the cloud hosts currently recovered, and the remaining unrecovered cloud hosts are controlled not to be recovered. For example, a faulty host node has 10 cloud hosts. When recovering to the 5th cloud host, the number of cloud hosts that have failed to recover is 3, and the first threshold value is 2. This exceeds the threshold value, and the remaining unrecovered cloud hosts from the 6th to the 10th will no longer be evacuated and recovered.

[0083] The second fuse rule is that a single cloud host is only allowed to perform no more than a set number of recovery within a preset time. If the current recovery task exceeds the second threshold, no evacuation will be performed. If the number of recovery times of the cloud host recovery task of the current cloud host exceeds the second threshold within the preset time, the cloud host corresponding to the cloud host recovery task will not be restored. For example, if the set recovery value of the cloud host is 5 times a day, and the actual number of recovery times exceeds 5 times, the cloud host will not be restored.

[0084] The third fuse rule is that when the number of faulty host nodes exceeds the set value within a certain period of time, it is determined that the high-availability cluster no longer has the ability to recover, and the faulty host nodes to be recovered will no longer be recovered. When the number of faulty host nodes exceeds the third threshold, the faulty host nodes will not be recovered. This indicates that there are currently many faulty host nodes and the recovery process needs to be stopped.

[0085] The fourth circuit breaker rule is that when the number of repeated failures of a faulty host node within a certain time period exceeds the set value, it is determined that the high-availability cluster no longer has the ability to recover, and the faulty host node to be recovered will no longer be recovered. When the number of failed node recovery tasks of the faulty host node exceeds the fourth threshold, the faulty host node corresponding to the current recovery is obtained and the faulty host node to be recovered is determined based on the faulty host node corresponding to the current recovery, and the faulty host node to be recovered is controlled not to be recovered. The description method is the same as the first circuit breaker rule, and will not be repeated here.

[0086] The fifth fuse rule is that a single host node is only allowed to perform no more than a set number of recovery times within a certain time period. If the current recovery task exceeds the set value, the faulty host node will no longer perform fault recovery. If the node recovery task of the current faulty host node exceeds the fifth threshold value within the predetermined time, the current faulty host node will not be recovered. This is described in the same way as the second fuse rule and will not be described in detail here.

[0087] It should be noted that the first threshold, the second threshold, the third threshold, the fourth threshold and the fifth threshold may be the same or different, and are not specifically limited.

[0088] The circuit breaker rules provided in the embodiment of the present invention include cloud host circuit breaker rules and node circuit breaker rules. Recovery is performed according to different rules under each circuit breaker rule. When the cloud platform's carrying fault handling capacity is insufficient or an error occurs, the host's fault recovery process is terminated in time, which can avoid greater danger to the entire cloud platform when the faulty host node cannot be recovered.

[0089] Based on the above embodiment, the current limiting rule in step S13 is formulated through the following steps:

[0090] The current limiting rules include cloud host current limiting rules and node current limiting rules;

[0091] The cloud host current limiting rule is to obtain the number of cloud hosts corresponding to the current faulty host node within a specified time. If the number of cloud hosts exceeds the first current limiting threshold, the cloud hosts are evacuated according to the priority of the recovery tasks of each cloud host corresponding to the current faulty host node.

[0092] The node current limiting rule is a rule for evacuating the multiple failed host nodes according to the priorities of the recovery tasks corresponding to the multiple failed host nodes when the number of the failed host nodes exceeds the second current limiting threshold within a specific time.

[0093] Specifically, the flow limiting rules are divided into two types according to the resource type of the recovery task. One is the flow limiting rule based on the cloud host, and the other is the flow limiting rule based on the faulty host node, namely the cloud host flow limiting rule and the node flow limiting rule.

[0094] The cloud host current limiting rule is the number of cloud hosts allowed to be evacuated on a certain faulty host node within a specified time. If the current limiting setting value is exceeded and the number of cloud hosts exceeds the first current limiting threshold, evacuation is performed based on the priority of the recovery tasks corresponding to the cloud hosts. For example, if the number of cloud hosts corresponding to the faulty host node is 10 and the first current limiting threshold is 3, the corresponding cloud host recovery tasks are processed first according to the recovery tasks with higher priorities, and the 3 cloud host recovery tasks with higher priorities are processed first, and then processed in order.

[0095] The node current limiting rule is the number of faulty host nodes that are allowed to be processed at a certain time point or a certain time period when multiple host nodes on the platform fail at the same time. If the current limiting setting value is exceeded, the faulty host nodes with lower priority will wait for recovery. That is, when the number of multiple faulty host nodes obtained within a specific time exceeds the second current limiting threshold, priority evacuation is performed according to the corresponding recovery tasks. The priority order is similar to the cloud host current limiting rule, so it is not repeated here.

[0096] It should be noted that the first current limiting threshold and the second current limiting threshold may be the same or different, and are not specifically limited.

[0097] The current limiting rules provided in the embodiment of the present invention include cloud host current limiting rules and node current limiting rules. Recovery is performed according to the rules under each current limiting rule. When a large-scale host failure occurs, it can be ensured that the cloud host and the physical host can be restored stably and orderly within the tolerable range of the platform.

[0098] As a preferred embodiment, in step S13, the recovery task of the priority queue is restored according to the fuse rule and the current limiting rule to complete the processing of the host failure, including:

[0099] Determine whether the recovery task of the priority queue meets the circuit breaker rules;

[0100] If the circuit breaker rules are met, determine whether the recovery task of the priority queue meets the current limiting rules;

[0101] If the flow limiting rules are met, the recovery task will be restored;

[0102] If any of the current limiting rules is not met, the state of the corresponding recovery task is set to a waiting state and the process returns to the step of prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue;

[0103] If any of the circuit breaker rules are not met, the corresponding recovery task will not be recovered.

[0104] Specifically, the circuit breaker rule is performed first and then the current limiting rule is performed. It can also be used as a preferred embodiment to perform the current limiting rule first and then the circuit breaker rule, which is specifically as follows:

[0105] Determine whether the recovery task of the priority queue meets the current limiting rules;

[0106] If the flow limit rules are met, determine whether the recovery task meets the circuit breaker rules;

[0107] If the circuit breaker rules are met, the recovery task will be restored;

[0108] If any of the circuit breaker rules are not met, the recovery task will not be restored;

[0109] If any of the current limiting rules is not met, the state of the corresponding recovery task is set to a waiting state and the process returns to the step of prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue.

[0110] The present invention provides a method for restoring the recovery tasks of the priority queue according to the fuse rules and the current limiting rules to complete the processing of the host failure. No matter which rule is performed first or later, through the fuse, when the cloud platform's fault handling capacity is insufficient or an error occurs, the host's fault recovery process is terminated in time, which can avoid greater danger to the entire cloud platform when the faulty host cannot be recovered; through current limiting, when a large-scale node failure occurs on the cloud platform, it can ensure that the host failure is recovered stably and orderly within the range that the platform can bear.

[0111] Based on the above embodiment, the method further includes:

[0112] The recovery tasks that are not to be restored are collected into a table for easy viewing by users.

[0113] The recovery tasks corresponding to the non-recovery processing are collected, where the recovery tasks without recovery processing include cloud host recovery tasks and node recovery tasks, and they are collected into table information so that when the faulty host node fails later, the recovery task can be skipped and the next recovery task can be directly performed.

[0114] The recovery tasks corresponding to the non-recovery processing provided in this embodiment are collected into table information, so that when the faulty host node fails later, the recovery task can be skipped and the next recovery task can be carried out directly, thereby reducing the recovery time and making it easier for users to view the progress of the current recovery task.

[0115] The above describes in detail various embodiments corresponding to the method for processing host failures. On this basis, the present invention also discloses a host failure processing device corresponding to the above method. Figure 2 The structure diagram of a host failure processing device provided by an embodiment of the present invention. Figure 2 As shown, the host failure processing device includes:

[0116] An acquisition module 11 is used to acquire each cloud host corresponding to the failed host node;

[0117] A sorting module 12 is used to sort the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each cloud host to obtain a priority queue;

[0118] The recovery module 13 is used to recover the recovery task of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure.

[0119] Since the embodiments of the device part correspond to the above embodiments, please refer to the description of the embodiments of the method part for the embodiments of the device part, and will not be repeated here.

[0120] An embodiment of the present invention provides a device for processing host failures, including obtaining each cloud host corresponding to a failed host node; prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue; and restoring the recovery tasks of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure. In the event of a large-scale host failure, the priority ranking is first used to determine the order in which the cloud hosts are restored after the failure occurs, and then the fuse rule is used to terminate the host failure recovery process in time when the cloud platform's ability to carry the failure is insufficient or an error occurs, so as to avoid the risk impact on the entire cloud platform; through the current limiting rule, when the cloud platform fails, it is ensured that the host failure is restored stably and orderly within the range that the platform can bear. In summary, by processing the recovery tasks of both the failed host node and the corresponding cloud hosts, the cloud host is guaranteed to be restored stably and orderly, thereby improving the high availability of the cloud platform.

[0121] Figure 3 A structural diagram of another host failure processing device provided by an embodiment of the present invention, such as Figure 3 As shown, the device comprises:

[0122] A memory 21, used for storing computer programs;

[0123] The processor 22 is used to implement the steps of the method for handling host failure when executing a computer program.

[0124] The host failure processing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0125] Among them, the processor 22 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 22 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 22 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0126] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the host failure processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, data involved in the method for processing host failures, etc.

[0127] In some embodiments, the host failure processing device may also include a display screen 23 , an input and output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0128] Those skilled in the art can understand that Figure 3 The structure shown in the figure does not constitute a limitation on the device for processing host failure, and may include more or fewer components than those shown in the figure.

[0129] The processor 22 implements the host failure processing method provided by any of the above embodiments by calling the instructions stored in the memory 21.

[0130] An embodiment of the present invention provides a device for processing host failures, including obtaining each cloud host corresponding to a failed host node; prioritizing the recovery tasks corresponding to the failed host node and each cloud host to obtain a priority queue; and restoring the recovery tasks of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure. In the event of a large-scale host failure, the priority ranking is first used to determine the order in which the cloud hosts are restored after the failure occurs, and then the fuse rule is used to terminate the host failure recovery process in time when the cloud platform's ability to carry the failure is insufficient or an error occurs, so as to avoid the risk impact on the entire cloud platform; through the current limiting rule, when the cloud platform fails, it is ensured that the host failure is restored stably and orderly within the range that the platform can bear. In summary, by processing the recovery tasks of both the failed host node and the corresponding cloud hosts, the cloud host is guaranteed to be restored stably and orderly, thereby improving the high availability of the cloud platform.

[0131] Furthermore, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor 22, the steps of the host failure processing method as described above are implemented.

[0132] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0133] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above host failure processing method.

[0134] The above is a detailed introduction to a method for handling a host failure, a device for handling a host failure, and a medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0135] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for handling a host failure, characterized in that: include: Get the cloud hosts corresponding to the failed host node; Prioritize the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each of the cloud hosts to obtain a priority queue; Recovering the recovery task of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure; The step of prioritizing the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each of the cloud hosts to obtain a priority queue includes: The recovery task corresponding to the failed host node is used as a node recovery task, and the recovery task corresponding to each of the cloud hosts is used as a cloud host recovery task; Prioritizing the node recovery tasks and the cloud host recovery tasks to obtain the node recovery task queue and the cloud host recovery task queue; The circuit breaker rules are formulated through the following steps: The fuse rules include cloud host fuse rules and node fuse rules, wherein the cloud host fuse rules include at least the first fuse rule and the second fuse rule, and the node fuse rules include at least the third fuse rule, the fourth fuse rule and the fifth fuse rule; Wherein, the first fuse rule is that when the number of failed recovery of cloud host recovery tasks of each of the cloud hosts exceeds a first threshold, the cloud host corresponding to the current recovery is obtained and the remaining unrecovered cloud hosts are determined according to the cloud host corresponding to the current recovery, and the remaining unrecovered cloud hosts are controlled not to be recovered; The second fuse rule is a rule that when the number of recovery times of the cloud host recovery task of the current cloud host exceeds a second threshold within a preset time, the cloud host corresponding to the cloud host recovery task will not be restored; The third fuse rule is a rule that when the number of the faulty host nodes exceeds a third threshold, the faulty host nodes are not restored; The fourth fuse rule is that when the number of failed node recovery tasks of the faulty host node exceeds the fourth threshold, the faulty host node corresponding to the current recovery is obtained and the faulty host node to be recovered is determined according to the faulty host node corresponding to the current recovery, and the rule that the faulty host node to be recovered is controlled not to perform recovery processing; The fifth fuse rule is a rule that when the number of node recovery tasks of the current faulty host node exceeds the fifth threshold within a predetermined time, the current faulty host node will not be recovered; The current limiting rules are formulated through the following steps: The current limiting rules include cloud host current limiting rules and node current limiting rules; The cloud host current limiting rule is to obtain the number of cloud hosts corresponding to the current faulty host node within a specified time, and if the number of cloud hosts exceeds the first current limiting threshold, evacuate the cloud hosts according to the priority of the recovery tasks of each cloud host corresponding to the current faulty host node; The node current limiting rule is a rule for evacuating according to the priorities of the recovery tasks corresponding to the multiple faulty host nodes when the number of the multiple faulty host nodes obtained within a specific time exceeds the second current limiting threshold.

2. The method for handling host failure according to claim 1, characterized in that: The step of restoring the recovery task of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure includes: Determine whether the recovery task of the priority queue complies with the fuse rule; If the circuit breaker rule is met, determining whether the recovery task of the priority queue meets the current limiting rule; If the current limiting rule is met, the recovery task is restored; If any of the current limiting rules is not met, the state of the corresponding recovery task is set to a waiting state and the process returns to the step of prioritizing the recovery tasks corresponding to the failed host node and each of the cloud hosts to obtain a priority queue; If any of the circuit breaker rules is not met, the corresponding recovery task will not be recovered.

3. The method for handling host failure according to claim 1, characterized in that: The step of restoring the recovery task of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure includes: Determine whether the recovery task of the priority queue complies with the current limiting rule; If the current limiting rule is met, determining whether the recovery task meets the fuse rule; If the circuit breaker rule is met, the recovery task is restored; If any of the circuit breaker rules are not met, the recovery task will not be recovered; If any of the current limiting rules is not met, the state of the corresponding recovery task is set to a waiting state and the process returns to the step of prioritizing the recovery tasks corresponding to the failed host node and each of the cloud hosts to obtain a priority queue.

4. The method for handling host failure according to claim 2 or 3, characterized in that: Also includes: The recovery tasks that are not to be restored are collected into a table for easy viewing by users.

5. A host failure processing device, characterized in that: include: An acquisition module is used to obtain each cloud host corresponding to the failed host node; A sorting module, used to sort the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each of the cloud hosts to obtain a priority queue; A recovery module, used to recover the recovery task of the priority queue according to the fuse rule and the current limiting rule to complete the processing of the host failure; The step of prioritizing the recovery tasks corresponding to the failed host node and the recovery tasks corresponding to each of the cloud hosts to obtain a priority queue includes: The recovery task corresponding to the failed host node is used as a node recovery task, and the recovery task corresponding to each of the cloud hosts is used as a cloud host recovery task; Prioritizing the node recovery tasks and the cloud host recovery tasks to obtain the node recovery task queue and the cloud host recovery task queue; The circuit breaker rules are formulated through the following steps: The fuse rules include cloud host fuse rules and node fuse rules, wherein the cloud host fuse rules include at least the first fuse rule and the second fuse rule, and the node fuse rules include at least the third fuse rule, the fourth fuse rule and the fifth fuse rule; Wherein, the first fuse rule is that when the number of failed recovery of cloud host recovery tasks of each of the cloud hosts exceeds a first threshold, the cloud host corresponding to the current recovery is obtained and the remaining unrecovered cloud hosts are determined according to the cloud host corresponding to the current recovery, and the remaining unrecovered cloud hosts are controlled not to be recovered; The second fuse rule is a rule that when the number of recovery times of the cloud host recovery task of the current cloud host exceeds a second threshold within a preset time, the cloud host corresponding to the cloud host recovery task will not be restored; The third fuse rule is a rule that when the number of the faulty host nodes exceeds a third threshold, the faulty host nodes are not restored; The fourth fuse rule is that when the number of failed node recovery tasks of the faulty host node exceeds the fourth threshold, the faulty host node corresponding to the current recovery is obtained and the faulty host node to be recovered is determined according to the faulty host node corresponding to the current recovery, and the rule that the faulty host node to be recovered is controlled not to perform recovery processing; The fifth fuse rule is a rule that when the number of node recovery tasks of the current faulty host node exceeds the fifth threshold within a predetermined time, the current faulty host node will not be recovered; The current limiting rules are formulated through the following steps: The current limiting rules include cloud host current limiting rules and node current limiting rules; The cloud host current limiting rule is to obtain the number of cloud hosts corresponding to the current faulty host node within a specified time, and if the number of cloud hosts exceeds the first current limiting threshold, evacuate the cloud hosts according to the priority of the recovery tasks of each cloud host corresponding to the current faulty host node; The node current limiting rule is a rule for evacuating according to the priorities of the recovery tasks corresponding to the multiple faulty host nodes when the number of the multiple faulty host nodes obtained within a specific time exceeds the second current limiting threshold.

6. A host failure processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for handling a host failure as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for processing a host failure according to any one of claims 1 to 4 are implemented.

Images