A cloud computing node control method, device and medium

Abstract

The application relates to the field of cloud computing, and discloses a cloud computing node control method and device and a medium, which comprises the following steps: when a fault node of a cloud computing platform is detected, target cloud hosts corresponding to each fault node are determined; each recovery task is prioritized to obtain a priority queue, the recovery task comprising a cloud node recovery task and a cloud host recovery task; each recovery task in the priority queue is executed, and whether each recovery task in the priority queue is successfully executed is judged; if the recovery task is not successfully executed, the priority of the failed task is lowered, and the failed task is added to the priority queue. When a large-scale host fault is detected, if the recovery task fails to be executed, the priority of the failed task is lowered, and the failed task is added to the task queue again, so that the cloud host can be stably and orderly recovered. It can be seen that the priority of the failed task is lowered and the failed task is executed, so that the cloud host can be stably recovered, and the reliability and stability of the cloud platform are improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a cloud computing node control method, device, and medium. Background Technology

[0002] Cloud computing platforms manage various physical devices through hardware virtualization technology to provide cloud computing services to users. In order for cloud computing platforms to provide stable cloud services to users, they typically need to have high availability capabilities. That is, when a host node in the platform fails, the cloud hosts running on the failed node can be distributed to other normal nodes to ensure that user services are not affected.

[0003] Currently, the high availability features provided by various cloud computing platforms only allow for the transfer of cloud servers running on failed nodes to healthy nodes upon discovery of a failure, without effective control over the recovery process. When there are a large number of failed nodes in a cloud computing platform, the failure of a recovery task may cause chaos in the cloud server failure recovery system, preventing the recovery task from executing normally and ultimately leading to the collapse of the cloud computing platform.

[0004] Therefore, it is evident that providing a method for managing cloud computing nodes in a cloud computing platform to ensure the normal operation of cloud hosts when a large-scale node failure occurs is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a cloud computing node control method, device, and medium to prevent the cloud computing platform from crashing due to the failure of recovery tasks when a large-scale node failure occurs, thereby ensuring the normal operation of the cloud platform.

[0006] This application provides a cloud computing node control method, including:

[0007] When a faulty node is detected in the cloud computing platform, the target cloud host corresponding to each faulty node is determined.

[0008] Prioritize each recovery task to obtain a priority queue, wherein the recovery tasks include cloud node recovery tasks and cloud host recovery tasks;

[0009] Execute each of the recovery tasks in the priority queue, and determine whether each of the recovery tasks in the priority queue has been successfully executed;

[0010] If the recovery task fails to execute, the priority of the failed task is reduced, and the failed task is added to the priority queue so that the recovery tasks in the priority queue can be executed again.

[0011] Preferably, executing each of the recovery tasks in the priority queue includes:

[0012] Determine whether the recovery task in the priority queue meets the circuit breaker rules;

[0013] If the circuit breaker rule is met, then determine whether the recovery task of the priority queue meets the rate limiting rule;

[0014] If the rate limiting rules are met, the recovery task will be restored.

[0015] If any of the rate limiting rules are not met, the status of the corresponding recovery task is set to waiting and the process returns to the step of prioritizing the recovery tasks corresponding to the faulty cloud node and each cloud host to obtain a priority queue.

[0016] If any of the circuit breaker rules are not met, the corresponding recovery task will not be processed.

[0017] Preferably, the step of prioritizing each recovery task to obtain a priority queue includes:

[0018] The cloud node recovery tasks and the cloud host recovery tasks are prioritized to obtain the node recovery task queue and the cloud host recovery task queue, respectively.

[0019] Preferably, reducing the priority of failed tasks includes:

[0020] Determine whether the failed task meets the preset conditions;

[0021] If the preset conditions are met, the priority of the failed task is reduced.

[0022] Preferably, the preset conditions include:

[0023] The failed task has a priority higher than a preset priority, and the number of times the failed task is executed is less than a threshold.

[0024] Preferably, the formulation of the circuit breaker rules includes:

[0025] The circuit breaker rules include cloud host circuit breaker rules and node circuit breaker rules, wherein the cloud host circuit breaker rules include at least a first circuit breaker rule and a second circuit breaker rule, and the node circuit breaker rules include at least a third circuit breaker rule, a fourth circuit breaker rule, and a fifth circuit breaker rule;

[0026] The first circuit breaker rule is that when the number of cloud server recovery tasks that fail to recover exceeds a first threshold, the cloud server corresponding to the current recovery is obtained, and the remaining unrecovered cloud servers are determined based on the cloud server corresponding to the current recovery, and the remaining unrecovered cloud servers are controlled not to be processed for recovery.

[0027] The second circuit breaker rule is that when the number of times the current cloud host recovery task exceeds a second threshold within a preset time, the cloud host corresponding to the cloud host recovery task will not be processed for recovery.

[0028] The third circuit breaker rule is that when the number of faulty cloud nodes exceeds a third threshold, the faulty cloud nodes will not be restored.

[0029] The fourth circuit breaker rule is that when the number of failed node recovery tasks of the faulty cloud node exceeds the fourth threshold, the faulty cloud node corresponding to the current recovery is obtained, and the faulty cloud node to be recovered is determined based on the faulty cloud node corresponding to the current recovery, and the rule of controlling the faulty cloud node to be recovered is not to perform recovery processing.

[0030] The fifth circuit breaker rule is that if the number of times the node recovery task of the current faulty cloud node is restored exceeds the fifth threshold within a predetermined time, then the current faulty cloud node will not be restored.

[0031] Preferably, the rate limiting rule is formulated through the following steps:

[0032] The rate limiting rules include cloud host rate limiting rules and node rate limiting rules;

[0033] The cloud host rate limiting rule is to obtain the number of cloud hosts corresponding to the current faulty cloud node within a specified time. If the number of cloud hosts exceeds the first rate limiting threshold, the rule is to evacuate them according to the priority of the recovery tasks of each cloud host corresponding to the current faulty cloud node.

[0034] The node rate limiting rule is a rule that, when the number of multiple faulty cloud nodes exceeds the second rate limiting threshold within a specific time period, the nodes are evacuated according to the priority of the recovery tasks corresponding to the multiple faulty cloud nodes.

[0035] To address the aforementioned technical problems, this application also provides a cloud computing node control device, comprising:

[0036] The determination module is used to determine the target cloud host corresponding to each faulty node when a faulty node of the cloud computing platform is detected.

[0037] The acquisition module has the ability to prioritize each recovery task to obtain a priority queue, wherein the recovery tasks include cloud node recovery tasks and cloud host recovery tasks;

[0038] The judgment module is used to execute each of the recovery tasks in the priority queue and determine whether each of the recovery tasks in the priority queue has been successfully executed.

[0039] The priority reduction module is used to reduce the priority of the failed task and add the failed task to the priority queue if the recovery task fails to execute successfully, so that the recovery tasks in the priority queue can be executed again.

[0040] To address the aforementioned technical problems, this application also provides a cloud computing node control device, including a memory for storing computer programs;

[0041] A processor is used to implement the steps of the cloud computing node control method when executing the computer program.

[0042] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the cloud computing node control method.

[0043] This application provides a cloud computing node control method, comprising: when a faulty node is detected in the cloud computing platform, determining the target cloud host corresponding to each faulty node; prioritizing each recovery task to obtain a priority queue, wherein the recovery tasks include cloud node recovery tasks and cloud host recovery tasks; executing each recovery task in the priority queue and determining whether each recovery task in the priority queue has been successfully executed; if a recovery task has not been successfully executed, reducing the priority of the failed task and adding the failed task to the priority queue so that the recovery tasks in the priority queue can be executed again. In the event of a large-scale host failure, priority sorting is first used to determine the order of cloud host recovery after the failure, and recovery tasks are executed in priority order. When a recovery task is detected to have failed, the priority of the failed task is reduced and the failed task is added back to the task queue to ensure stable and orderly recovery of cloud hosts. It can be seen that this application improves the reliability and stability of the cloud platform by reducing the priority of failed tasks and re-executing them.

[0044] In addition, this application also provides a cloud computing node control device and medium, which correspond to the above method and have the same effect. Attached Figure Description

[0045] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a cloud computing node control method provided in an embodiment of this application;

[0047] Figure 2 This is a structural diagram of a cloud computing node control device provided in an embodiment of this application;

[0048] Figure 3 This is a structural diagram of another cloud computing node control device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0050] The core of this application is to provide a cloud computing node control method, device, and medium to prevent the cloud computing platform from crashing due to the failure of recovery tasks when a large-scale node failure occurs, thereby ensuring the normal operation of the cloud platform.

[0051] In an application scenario of a cloud computing node control method, when multiple faulty nodes are detected in the cloud computing platform, the target cloud host corresponding to each faulty node is determined. Recovery tasks are prioritized to obtain a priority queue, which includes cloud node recovery tasks and cloud host recovery tasks. Each recovery task in the priority queue is executed, and its success is determined. If a recovery task fails, its priority is lowered, and it is added back to the priority queue for re-execution. In the event of a large-scale host failure, priority sorting is used to determine the order of cloud host recovery after a failure, and recovery tasks are executed according to priority. When a recovery task fails, its priority is lowered, and it is added back to the task queue to ensure stable and orderly recovery of the cloud hosts. It can be seen that this application improves the reliability and stability of the cloud platform by lowering the priority of failed tasks and re-executing them.

[0052] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Figure 1 A flowchart of a cloud computing node control method provided in an embodiment of this application is shown below. Figure 1 As shown, the method includes:

[0054] S10: When a faulty node of the cloud computing platform is detected, determine the target cloud host corresponding to each faulty node.

[0055] It is understood that a faulty cloud node is a compute node that has failed, and each compute node can host multiple cloud hosts. The host failure handling method provided by this invention can be applied to multi-node failure handling as well as single-node failure handling. The basic idea of ​​high availability implementation includes three steps: monitoring, isolation, and recovery; monitoring is used to detect faulty cloud nodes, isolation is used to isolate faulty cloud nodes, and recovery is used to evacuate cloud hosts on faulty cloud nodes to other healthy host nodes. This invention only addresses the recovery process for high availability. Existing vendors have basically similar recovery processes, simply mechanically evacuating cloud hosts from faulty cloud nodes to other healthy host nodes without effective control over the recovery process. For example, evacuating multiple cloud hosts simultaneously can cause congestion in the evacuation channel, leading to packet loss or failure to evacuate, affecting the normal use of the cloud platform.

[0056] S11: Prioritize each recovery task to obtain a priority queue. Recovery tasks include cloud node recovery tasks and cloud host recovery tasks.

[0057] In practice, cloud node recovery tasks and cloud host recovery tasks can be merged into a single queue, or two parallel priority queues can be set up: a node recovery task queue and a cloud host recovery task queue. It's understandable that priorities can be determined based on resource type; for example, the priority of each cloud host recovery task can be set higher than that of each cloud node recovery task, and the two types of recovery tasks can be sorted according to their respective priorities to obtain the corresponding priority queues. Alternatively, the type of recovery task can be ignored, and the recovery tasks can be prioritized only according to their individual priorities.

[0058] Priority sorting can be performed by calculating the priority of each recovery task using a specific algorithm and queuing them in a task queue for processing; alternatively, it can be sorted based on specific attributes, such as evacuation capacity priority. This invention does not impose any specific limitations, as long as the priority order can be achieved. Higher-priority recovery tasks are processed first, while lower-priority recovery tasks are processed later.

[0059] S12: Execute each recovery task in the priority queue and determine whether each recovery task in the priority queue has been executed successfully;

[0060] S13: If a recovery task fails to execute, lower the priority of the failed task and add it to the priority queue so that the recovery tasks in the priority queue can be executed again.

[0061] It is understandable that when using a certain recovery strategy for fault recovery, the host and cloud hosts may fail to recover. However, when using another recovery strategy, the host and cloud hosts may recover successfully. For example, if all cloud hosts on a failed cloud node are centrally redistributed to a single healthy host node, some cloud hosts may fail to recover successfully due to insufficient resources on the target node. However, if the cloud hosts that failed to recover from the failed cloud node are freely redistributed to all other healthy host nodes, the failed cloud hosts may continue to recover because there are more resources available. In this case, it is still necessary to ensure that the high-priority hosts and cloud hosts that failed to recover receive a chance to retry the recovery.

[0062] Specifically, when a failed recovery task is detected, detailed information about the failed task (including its priority, number of times it was executed, etc.) is obtained, the priority of the failed task is reduced, and the failed task is added back to the priority queue for re-execution.

[0063] This application provides a cloud computing node control method, comprising: when a faulty node is detected in the cloud computing platform, determining the target cloud host corresponding to each faulty node; prioritizing each recovery task to obtain a priority queue, wherein the recovery tasks include cloud node recovery tasks and cloud host recovery tasks; executing each recovery task in the priority queue and determining whether each recovery task in the priority queue has been successfully executed; if a recovery task has not been successfully executed, reducing the priority of the failed task and adding the failed task to the priority queue so that the recovery tasks in the priority queue can be executed again. In the event of a large-scale host failure, priority sorting is first used to determine the order of cloud host recovery after the failure, and recovery tasks are executed in priority order. When a recovery task is detected to have failed, the priority of the failed task is reduced and the failed task is added back to the task queue to ensure stable and orderly recovery of cloud hosts. It can be seen that this application improves the reliability and stability of the cloud platform by reducing the priority of failed tasks and re-executing them.

[0064] In practice, to prevent system crashes caused by too many faulty nodes in the cloud platform when executing recovery tasks in the priority queue, specific rules are needed to control the recovery tasks. For example, a rate-limiting mechanism can ensure that cloud hosts and physical hosts can recover stably and orderly within the platform's capacity when a large-scale host node failure occurs. A circuit breaker mechanism can promptly terminate the host failure recovery process when the cloud platform's fault handling capacity is insufficient or malfunctions, preventing greater danger to the entire cloud platform if a faulty host cannot be recovered.

[0065] As a preferred embodiment, based on the above embodiments, executing each recovery task in the priority queue includes:

[0066] Determine whether the recovery tasks in the priority queue meet the circuit breaker rules;

[0067] If the circuit breaker rule is met, then determine whether the recovery task in the priority queue meets the rate limiting rule;

[0068] If the rate limiting rules are met, the recovery task will be processed.

[0069] If any of the rate limiting rules are not met, the status of the corresponding recovery task will be set to waiting and the process will return to the step of prioritizing the recovery tasks corresponding to the faulty cloud node and each cloud host to obtain a priority queue.

[0070] If any of the circuit breaker rules are not met, the corresponding recovery task will not be processed.

[0071] In a preferred embodiment, the circuit breaker rules are formulated through the following steps:

[0072] The circuit breaker rules include cloud host circuit breaker rules and node circuit breaker rules. The cloud host circuit breaker rules include at least the first circuit breaker rule and the second circuit breaker rule, and the node circuit breaker rules include at least the third circuit breaker rule, the fourth circuit breaker rule, and the fifth circuit breaker rule.

[0073] Among them, the first circuit breaker rule is that when the number of cloud host recovery tasks that fail to recover exceeds the first threshold, the cloud host corresponding to the current recovery is obtained and the remaining unrecovered cloud hosts are determined based on the cloud host corresponding to the current recovery, and the remaining unrecovered cloud hosts are controlled not to be processed for recovery.

[0074] The second circuit breaker rule is that if the number of times the current cloud server recovery task exceeds the second threshold within a preset time, the cloud server corresponding to the cloud server recovery task will not be processed for recovery.

[0075] The third circuit breaker rule is that if the number of faulty cloud nodes exceeds the third threshold, the faulty cloud nodes will not be restored.

[0076] The fourth circuit breaker rule is that when the number of failed node recovery tasks of a faulty cloud node exceeds the fourth threshold, the faulty cloud node corresponding to the current recovery is obtained, and the faulty cloud node to be recovered is determined based on the faulty cloud node corresponding to the current recovery, and the rule that the faulty cloud node to be recovered is not processed is controlled.

[0077] The fifth circuit breaker rule is that if the number of times the node recovery task of the current faulty cloud node is attempted to recover exceeds the fifth threshold within a predetermined time, then the current faulty cloud node will not be processed for recovery.

[0078] Specifically, the circuit breaker rules are divided into two types based on the resource type of the recovery task: one is the circuit breaker rule based on cloud hosts, and the other is the circuit breaker rule based on faulty cloud nodes. That is, the cloud host circuit breaker rule and the node circuit breaker rule. The cloud host circuit breaker rule includes at least the first and second circuit breaker rules, and the node circuit breaker rule includes at least the third, fourth and fifth circuit breaker rules. In other words, it can also include other rules.

[0079] The first circuit breaker rule is that if the number of cloud hosts requiring evacuation during recovery on a single faulty cloud node exceeds a first threshold, it indicates that the remaining cloud hosts on that faulty cloud node no longer have the ability to continue recovery, and the remaining unevacuated cloud hosts will no longer be evacuated. Each cloud host corresponds to a faulty cloud node. When the number of recovery failures of the corresponding cloud host recovery task exceeds the first threshold, the remaining unrecovered cloud hosts are determined based on the currently recovering cloud host, and the remaining unrecovered cloud hosts are prevented from being recovered.

[0080] The second circuit breaker rule allows a single cloud host to perform no more than a set number of recoveries within a preset time. If the current recovery task exceeds the second threshold, no recovery will be initiated. For example, if the set recovery number for a cloud host per day is 5, and the actual number of recoveries exceeds 5, then the cloud host will not be recovered.

[0081] The third circuit breaker rule determines that the high-availability cluster is no longer capable of recovery when the number of faulty cloud nodes exceeds a set value within a certain time period. In this case, the faulty cloud nodes will not be restored. Specifically, if the number of faulty cloud nodes exceeds the third threshold, recovery will not be initiated. This indicates that there are too many faulty cloud nodes, and the recovery process needs to be stopped.

[0082] The fourth circuit breaker rule determines that if the number of data failures caused by a faulty cloud node repeatedly exceeds a set value within a certain time period, the high-availability cluster is deemed to have lost its ability to recover, and the faulty cloud node awaiting recovery will no longer be attempted. If the number of failed recovery tasks for a faulty cloud node exceeds the fourth threshold, the faulty cloud node currently being recovered will be identified, and based on this, the faulty cloud node awaiting recovery will be determined and prevented from undergoing recovery processing. The description method is the same as the first circuit breaker rule and will not be elaborated further here.

[0083] The fifth circuit breaker rule allows a single host node to perform no more than a set number of recovery attempts within a certain time period. If the current recovery tasks exceed the set value, the faulty cloud node will no longer undergo fault recovery. If the number of recovery attempts for the currently faulty cloud node exceeds the fifth threshold within the predetermined time, the currently faulty cloud node will not be processed for recovery. The description method is the same as the second circuit breaker rule, and will not be elaborated further here.

[0084] It should be noted that the first threshold, second threshold, third threshold, fourth threshold, and fifth threshold can be the same or different, without specific restrictions.

[0085] As a preferred embodiment, the rate limiting rules are formulated through the following steps:

[0086] Rate limiting rules include cloud server rate limiting rules and node rate limiting rules;

[0087] The cloud server rate limiting rule is to obtain the number of cloud servers corresponding to the current faulty cloud node within a specified time. If the number of cloud servers exceeds the first rate limiting threshold, the evacuation rule is based on the priority of the recovery tasks of each cloud server corresponding to the current faulty cloud node.

[0088] The node rate limiting rule is a rule that, when the number of multiple faulty cloud nodes exceeds the second rate limiting threshold within a specific time period, the nodes are evacuated according to the priority of the recovery tasks corresponding to the multiple faulty cloud nodes.

[0089] Specifically, the rate limiting rules are divided into two types based on the resource type of the recovery task: one is the rate limiting rule based on cloud hosts, and the other is the rate limiting rule based on faulty host nodes, namely, cloud host rate limiting rules and cloud node rate limiting rules.

[0090] Cloud node rate limiting rules define the number of failed host nodes allowed to be processed at a specific point in time or within a specific time period when multiple host nodes on the platform fail simultaneously. If the rate limiting setting is exceeded, lower-priority failed host nodes will wait for recovery. In other words, if the number of multiple failed host nodes exceeds the second rate limiting threshold within a specific time period, priority evacuation will be performed based on the corresponding recovery tasks. The priority order is similar to that of cloud host rate limiting rules, so it will not be elaborated upon here.

[0091] It should be noted that the first and second rate limiting thresholds can be the same or different, and there are no specific restrictions.

[0092] In practice, to prevent failed tasks from being executed repeatedly, the failure task information can be evaluated, and the priority of the failed task can be reduced if and only if the failed task meets the preset conditions.

[0093] Based on the above embodiments, reducing the priority of failed tasks includes: determining whether the failed task meets preset conditions; if the preset conditions are met, then reducing the priority of the failed task. The preset conditions include: the priority of the failed task is greater than a preset priority, and the number of times the failed task has been executed is less than a threshold number.

[0094] It is understandable that if the recovery task returns an incorrect value or times out, the recovery is considered to have failed.

[0095] In practice, when a failed recovery task is detected, the priority and execution count of the failed task are obtained and compared with a preset priority that allows retries. If the priority of the failed task is less than or equal to the preset priority, the failed task is not allowed to be downgraded for retry recovery. If the priority of the failed task is greater than the minimum priority that allows retries, it is determined whether the execution count of the failed task is less than a threshold. If it is less, the priority of the failed task is reduced, the execution count of the failed task is incremented by one, and the failed task is added back to the queue for re-execution.

[0096] The cloud computing node control method has been described in detail in the above embodiments. This application also provides embodiments corresponding to the cloud computing node control device. It should be noted that this application describes the embodiments of the device part from two perspectives: one is based on the functional modules, and the other is based on the hardware.

[0097] Figure 2 This is a structural diagram of a cloud computing node control device provided in an embodiment of this application, as shown below. Figure 2 As shown, the device includes:

[0098] The determination module 10 is used to determine the target cloud host corresponding to each fault node when a fault node of the cloud computing platform is detected.

[0099] Module 11 is responsible for prioritizing each recovery task to obtain a priority queue. The recovery tasks include cloud node recovery tasks and cloud host recovery tasks.

[0100] The judgment module 12 is used to execute each recovery task in the priority queue and determine whether each recovery task in the priority queue has been successfully executed.

[0101] The priority reduction module 13 is used to reduce the priority of failed tasks and add them to the priority queue if a recovery task fails to execute successfully, so that each recovery task in the priority queue can be executed again.

[0102] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0103] This application provides a cloud computing node control device, comprising: when a faulty node of a cloud computing platform is detected, determining the target cloud host corresponding to each faulty node; prioritizing each recovery task to obtain a priority queue, wherein the recovery tasks include cloud node recovery tasks and cloud host recovery tasks; executing each recovery task in the priority queue and determining whether each recovery task in the priority queue has been successfully executed; if a recovery task has not been successfully executed, reducing the priority of the failed task and adding the failed task to the priority queue so that the recovery tasks in the priority queue can be executed again. In the event of a large-scale host failure, priority sorting is first used to determine the order of cloud host recovery after the failure, and recovery tasks are executed according to priority order. When a recovery task is detected to have failed, the priority of the failed task is reduced and the failed task is added back to the task queue to ensure stable and orderly recovery of cloud hosts. It can be seen that this application improves the reliability and stability of the cloud platform by reducing the priority of failed tasks and re-executing them.

[0104] Figure 3 A structural diagram of another cloud computing node control device provided in the embodiments of this application is shown below. Figure 3 As shown, the cloud computing node control device includes: a memory 20 for storing computer programs;

[0105] The processor 21 is used to implement the steps of the cloud computing node control method as described in the above embodiments when executing a computer program.

[0106] The terminal devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0107] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0108] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the cloud computing node control method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, restoring task priorities.

[0109] In some embodiments, the cloud computing node control device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0110] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the cloud computing node control device and may include more or fewer components than shown.

[0111] The cloud computing node control device provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method:

[0112] When a faulty node is detected on the cloud computing platform, the target cloud host corresponding to each faulty node is determined.

[0113] Prioritize each recovery task to obtain a priority queue. Recovery tasks include cloud node recovery tasks and cloud host recovery tasks.

[0114] Execute each recovery task in the priority queue and determine whether each recovery task in the priority queue has been executed successfully;

[0115] If a recovery task fails to execute, the priority of the failed task is reduced, and the failed task is added to the priority queue so that the recovery tasks in the priority queue can be executed again.

[0116] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0117] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The cloud node control method, apparatus, and medium provided in this application have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0119] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A cloud computing node control method, characterized in that, include: When a faulty node is detected in the cloud computing platform, the target cloud host corresponding to each faulty node is determined. Prioritize each recovery task to obtain a priority queue, wherein the recovery tasks include cloud node recovery tasks and cloud host recovery tasks; Execute each of the recovery tasks in the priority queue, and determine whether each of the recovery tasks in the priority queue has been successfully executed; If the recovery task fails to execute, the priority of the failed task is reduced, and the failed task is added to the priority queue so that the recovery tasks in the priority queue can be executed again. The execution of each of the recovery tasks in the priority queue includes: Determine whether the recovery task in the priority queue meets the circuit breaker rules; If the circuit breaker rule is met, then determine whether the recovery task of the priority queue meets the rate limiting rule; If the rate limiting rules are met, the recovery task will be restored. If any of the rate limiting rules are not met, the status of the corresponding recovery task is set to waiting and the process returns to the step of prioritizing the recovery tasks corresponding to the faulty cloud node and each cloud host to obtain a priority queue. If any of the circuit breaker rules are not met, the corresponding recovery task will not be processed. The formulation of the circuit breaker rules includes: The circuit breaker rules include cloud host circuit breaker rules and node circuit breaker rules, wherein the cloud host circuit breaker rules include at least a first circuit breaker rule and a second circuit breaker rule, and the node circuit breaker rules include at least a third circuit breaker rule, a fourth circuit breaker rule, and a fifth circuit breaker rule; The first circuit breaker rule is that when the number of cloud server recovery tasks that fail to recover exceeds a first threshold, the cloud server corresponding to the current recovery is obtained, and the remaining unrecovered cloud servers are determined based on the cloud server corresponding to the current recovery, and the remaining unrecovered cloud servers are controlled not to be processed for recovery. The second circuit breaker rule is that when the number of times the current cloud host recovery task exceeds a second threshold within a preset time, the cloud host corresponding to the cloud host recovery task will not be processed for recovery. The third circuit breaker rule is that when the number of faulty cloud nodes exceeds a third threshold, the faulty cloud nodes will not be restored. The fourth circuit breaker rule is that when the number of failed node recovery tasks of the faulty cloud node exceeds the fourth threshold, the faulty cloud node corresponding to the current recovery is obtained, and the faulty cloud node to be recovered is determined based on the faulty cloud node corresponding to the current recovery, and the rule of controlling the faulty cloud node to be recovered is not to perform recovery processing. The fifth circuit breaker rule is that if the number of times the node recovery task of the current faulty cloud node is restored exceeds the fifth threshold within a predetermined time, then the current faulty cloud node will not be restored. The rate limiting rules are formulated through the following steps: The rate limiting rules include cloud host rate limiting rules and node rate limiting rules; The cloud host rate limiting rule is to obtain the number of cloud hosts corresponding to the current faulty cloud node within a specified time. If the number of cloud hosts exceeds the first rate limiting threshold, the rule is to evacuate them according to the priority of the recovery tasks of each cloud host corresponding to the current faulty cloud node. The node rate limiting rule is a rule that, when the number of multiple faulty cloud nodes exceeds the second rate limiting threshold within a specific time period, the nodes are evacuated according to the priority of the recovery tasks corresponding to the multiple faulty cloud nodes.

2. The cloud computing node control method according to claim 1, characterized in that, The step of prioritizing each recovery task to obtain a priority queue includes: The cloud node recovery tasks and the cloud host recovery tasks are prioritized to obtain the node recovery task queue and the cloud host recovery task queue, respectively.

3. The cloud computing node control method according to claim 1, characterized in that, The reduction of the priority of failed tasks includes: Determine whether the failed task meets the preset conditions; If the preset conditions are met, the priority of the failed task is reduced.

4. The cloud computing node control method according to claim 3, characterized in that, The preset conditions include: The failed task has a priority higher than a preset priority, and the number of times the failed task is executed is less than a threshold.

5. A cloud computing node control device, characterized in that, include: The determination module is used to determine the target cloud host corresponding to each faulty node when a faulty node of the cloud computing platform is detected. The acquisition module has the ability to prioritize each recovery task to obtain a priority queue, wherein the recovery tasks include cloud node recovery tasks and cloud host recovery tasks; The judgment module is used to execute each of the recovery tasks in the priority queue and determine whether each of the recovery tasks in the priority queue has been successfully executed. The priority reduction module is used to reduce the priority of the failed task and add the failed task to the priority queue if the recovery task fails to be executed successfully, so as to execute each of the recovery tasks in the priority queue again. The execution of each of the recovery tasks in the priority queue includes: Determine whether the recovery task in the priority queue meets the circuit breaker rules; If the circuit breaker rule is met, then determine whether the recovery task of the priority queue meets the rate limiting rule; If the rate limiting rules are met, the recovery task will be restored. If any of the rate limiting rules are not met, the status of the corresponding recovery task is set to waiting and the process returns to the step of prioritizing the recovery tasks corresponding to the faulty cloud node and each cloud host to obtain a priority queue. If any of the circuit breaker rules are not met, the corresponding recovery task will not be processed. The formulation of the circuit breaker rules includes: The circuit breaker rules include cloud host circuit breaker rules and node circuit breaker rules, wherein the cloud host circuit breaker rules include at least a first circuit breaker rule and a second circuit breaker rule, and the node circuit breaker rules include at least a third circuit breaker rule, a fourth circuit breaker rule, and a fifth circuit breaker rule; The first circuit breaker rule is that when the number of cloud server recovery tasks that fail to recover exceeds a first threshold, the cloud server corresponding to the current recovery is obtained, and the remaining unrecovered cloud servers are determined based on the cloud server corresponding to the current recovery, and the remaining unrecovered cloud servers are controlled not to be processed for recovery. The second circuit breaker rule is that when the number of times the current cloud host recovery task exceeds a second threshold within a preset time, the cloud host corresponding to the cloud host recovery task will not be processed for recovery. The third circuit breaker rule is that when the number of faulty cloud nodes exceeds a third threshold, the faulty cloud nodes will not be restored. The fourth circuit breaker rule is that when the number of failed node recovery tasks of the faulty cloud node exceeds the fourth threshold, the faulty cloud node corresponding to the current recovery is obtained, and the faulty cloud node to be recovered is determined based on the faulty cloud node corresponding to the current recovery, and the rule of controlling the faulty cloud node to be recovered is not to perform recovery processing. The fifth circuit breaker rule is that if the number of times the node recovery task of the current faulty cloud node is restored exceeds the fifth threshold within a predetermined time, then the current faulty cloud node will not be restored. The rate limiting rules are formulated through the following steps: The rate limiting rules include cloud host rate limiting rules and node rate limiting rules; The cloud host rate limiting rule is to obtain the number of cloud hosts corresponding to the current faulty cloud node within a specified time. If the number of cloud hosts exceeds the first rate limiting threshold, the rule is to evacuate them according to the priority of the recovery tasks of each cloud host corresponding to the current faulty cloud node. The node rate limiting rule is a rule that, when the number of multiple faulty cloud nodes exceeds the second rate limiting threshold within a specific time period, the nodes are evacuated according to the priority of the recovery tasks corresponding to the multiple faulty cloud nodes.

6. A cloud computing node control device, characterized in that, Includes memory used to store computer programs; A processor, configured to implement the steps of the cloud computing node control method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the cloud computing node control method as described in any one of claims 1 to 4.

Images