High-availability task distribution method, system and device

Abstract

The invention discloses a high-availability task distribution method, system and device, and the method comprises the steps: obtaining a to-be-distributed task; using a consistent Hash algorithm to distribute task Hash identifiers for the to-be-distributed tasks, and mapping the task Hash identifiers to a Hash ring; node hash values of all available nodes in the node cluster are mapped in the hash ring; according to the node hash values of all available nodes, searching a target node of which the node resource can process the to-be-distributed task and the node hash value is closest to the task hash identifier in the hash ring; and distributing the to-be-distributed task to the target node. All available nodes in a node cluster and tasks to be distributed are mapped to the same Hash ring by using a consistent Hash algorithm, so that the tasks can be distributed to the available nodes, the processing efficiency can be improved, the stability of overall task distribution is not influenced when any node fails, and the overall performance of task distribution is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to a high-availability task distribution method, system and device. Background Art

[0002] In modern computing environments, task distribution systems coordinate task execution across multiple nodes to improve resource utilization and processing efficiency.

[0003] Traditional task distribution systems distribute computing tasks to multiple computing nodes for processing. Each node is responsible for executing a part of the task and aggregating the results after completion, thereby improving computing speed and resource utilization. The core of this model is parallel processing. Multiple nodes can execute tasks at the same time to speed up computing. However, if the distribution is not appropriate, some nodes may be overloaded while other nodes are idle, thus affecting overall performance. Summary of the invention

[0004] The embodiments of the present invention provide a highly available task distribution method, system and device to ensure the overall performance of task distribution.

[0005] In a first aspect, an embodiment of the present invention provides a high-availability task distribution method, the method comprising:

[0006] Get the tasks to be distributed;

[0007] Using a consistent hashing algorithm, assigning a task hash identifier to the task to be distributed, and mapping the task hash identifier to a hash ring; the hash ring is mapped with node hash values ​​of all available nodes in the node cluster;

[0008] According to the node hash values ​​of all available nodes, searching in the hash ring for a target node whose node resources can process the task to be distributed and whose node hash value is closest to the task hash identifier;

[0009] Allocate the task to be distributed to the target node.

[0010] Furthermore, the using of a consistent hashing algorithm to assign a task hash identifier to the task to be distributed includes:

[0011] Use the MD5 algorithm or the SHA-1 algorithm to assign a task hash identifier to the task to be distributed;

[0012] The node hash values ​​of all available nodes are generated using the MD5 algorithm or the SHA-1 algorithm.

[0013] Furthermore, before receiving the task to be distributed, the method further includes:

[0014] Register all nodes in the node cluster locally and obtain the node resources and node status of each node; the node resources include CPU resources, memory resources and network delay parameters; the node status is normal, waiting for maintenance and offline;

[0015] For each of the nodes, if the node status of the node is normal, the node is regarded as an available node;

[0016] Using the consistent hashing algorithm, generating a node hash value for each of the available nodes;

[0017] The node hash value of each of the available nodes is mapped to the hash ring.

[0018] Furthermore, the step of searching the hash ring for a target node whose node resource can process the task to be distributed and whose node hash value is closest to the task hash identifier according to the node hash values ​​of all available nodes includes:

[0019] According to the node hash values ​​of all available nodes, searching clockwise in the hash ring for a candidate node whose node hash value is closest to the task hash identifier;

[0020] If the node resources of the candidate node are capable of processing the task to be distributed, determining the candidate node as the target node;

[0021] If the node resources of the candidate node cannot process the task to be distributed, continue to search for the next candidate node clockwise in the hash ring until the node resources of the next candidate node can process the task to be distributed, and determine the next candidate node as the target node.

[0022] Further, each node in the node cluster includes a plurality of virtual nodes created for each physical node in the cluster;

[0023] The allocating the to-be-distributed task to the target node comprises:

[0024] Allocate the task to be distributed to the physical node where the target node is located.

[0025] Further, after mapping the node hash value of each of the available nodes to the hash ring, the method further includes:

[0026] When a new physical node comes online, a new virtual node is created for the new physical node; a node hash value is generated for the new virtual node using the consistent hashing algorithm, and the node hash value of the new virtual node is mapped to the hash ring.

[0027] Furthermore, after allocating the task to be distributed to the target node, the method further includes:

[0028] The distributed snapshot technology is used to regularly copy the task execution process of the task to be distributed to the redundant server.

[0029] Furthermore, after the distributed snapshot technology is used to periodically copy the task execution process of the task to be distributed to the redundant server, the method further includes:

[0030] If the target node fails, the task execution process of the task to be distributed is read from the redundant server;

[0031] Determining whether the task to be distributed has been completed according to the task execution process of the task to be distributed;

[0032] If not, the task to be distributed is reallocated to other nodes.

[0033] In a second aspect, an embodiment of the present invention provides a highly available task distribution system, characterized in that the system includes a node cluster and a task scheduler;

[0034] The node cluster is used to register all nodes in the node cluster to the task scheduler;

[0035] The task scheduler is used to implement the steps of the high-availability task distribution method described in the first aspect above.

[0036] In a third aspect, an embodiment of the present invention provides a highly available task distribution device, the device comprising:

[0037] The acquisition module is used to obtain tasks to be distributed;

[0038] A hash identifier allocation module is used to use a consistent hashing algorithm to allocate a task hash identifier to the task to be distributed, and map the task hash identifier to a hash ring; the hash ring is mapped with node hash values ​​of all available nodes in the node cluster;

[0039] A node search module, used to search in the hash ring for a target node whose node resources can process the task to be distributed and whose node hash value is closest to the task hash identifier according to the node hash values ​​of all available nodes;

[0040] The task allocation module is used to allocate the task to be distributed to the target node.

[0041] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to implement the steps of the above-mentioned high-availability task distribution method when executing the computer program stored in the memory.

[0042] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the above-mentioned high-availability task distribution method when executed by a processor.

[0043] In the embodiment of the present invention, tasks to be distributed are obtained; a consistent hashing algorithm is used to assign a task hash identifier to the tasks to be distributed, and the task hash identifier is mapped to a hash ring; the hash ring is mapped with node hash values ​​of all available nodes in the node cluster; according to the node hash values ​​of all available nodes, a target node whose node resources can process the tasks to be distributed and whose node hash value is closest to the task hash identifier is searched in the hash ring; and the tasks to be distributed are distributed to the target node. In the invention, a consistent hashing algorithm is used to map all available nodes and tasks to be distributed in the node cluster to the same hash ring, ensuring that tasks can be distributed to available nodes, improving processing efficiency, and not affecting the stability of the overall task distribution when any node fails, thereby ensuring the overall performance of task distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A process diagram of a highly available task distribution method provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the structure of a highly available task distribution system provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the structure of a highly available task distribution device provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to ensure the overall performance of task allocation, the embodiments of the present invention provide a highly available task distribution method, system and device.

[0051] Embodiment 1:

[0052] Figure 1 A process diagram of a highly available task distribution method provided by an embodiment of the present invention, the process includes the following steps:

[0053] S101: Obtain tasks to be distributed.

[0054] S102: Use a consistent hashing algorithm to assign a task hash identifier to the task to be distributed, and map the task hash identifier to a hash ring; the hash ring is mapped with node hash values ​​of all available nodes in the node cluster.

[0055] S103: According to the node hash values ​​of all available nodes, search in the hash ring for a target node whose node resources can process the task to be distributed and whose node hash value is closest to the task hash identifier.

[0056] S104: Allocate the tasks to be distributed to the target node.

[0057] Since the consistent hashing algorithm is used in the embodiment of the present invention, all available nodes and tasks to be distributed in the node cluster are mapped to the same hash ring, ensuring that the tasks can be distributed to the available nodes, the processing efficiency can be improved, and the stability of the overall task distribution is not affected when any node fails, thereby ensuring the overall performance of task distribution.

[0058] The highly available task distribution method provided in the embodiment of the present invention is applied to an electronic device, which may specifically be a task scheduler or a central task scheduler (hereinafter referred to as a scheduler). The electronic device is deployed on a highly available task distribution system. In addition to the scheduler, the task distribution system may also include a node cluster, and the available nodes in the node cluster can process the distributed tasks.

[0059] The tasks to be distributed obtained in the above S101 may be one or more. In the embodiment of the present invention, one task to be distributed is mainly used for description. The processing process of multiple tasks to be distributed is similar to the processing process of the one task to be distributed, and will not be repeated.

[0060] The node hash values ​​of all available nodes in the node cluster and the task hash identifiers of the tasks to be distributed are generated using the same hash algorithm, such as the MD5 algorithm or the SHA-1 algorithm. Specifically, the above-mentioned S102 uses a consistent hash algorithm. When assigning task hash identifiers to tasks to be distributed, the MD5 algorithm or the SHA-1 algorithm can be used to assign task hash identifiers to tasks to be distributed; wherein the node hash values ​​of all available nodes are generated using the MD5 algorithm or the SHA-1 algorithm. The hash ring is a logical ring structure, that is, tasks and nodes are mapped based on a ring structure, and optionally, the ring range can be 0 to a maximum value (for example, 2^32 or 2^64). Distributing nodes on the hash ring can increase the granularity of the mapping, ensure that the load can be more evenly distributed to each physical node, and use a distributed task distribution architecture to support multi-node collaboration. Using a consistent hashing algorithm for task distribution can support efficient fault tolerance.

[0061] All nodes can be registered on the scheduler. During the processing, the node can provide its processing capacity and initial status. After registration, the scheduler can try to monitor each node. For example, before the above S101, all nodes in the node cluster can also be registered locally, and the node resources and node status of each node can be obtained; node resources include CPU (Central Processing Unit) resources, memory resources and network delay parameters; node status is normal, waiting for maintenance and offline; for each node, if the node status of the node is normal, the node is used as an available node; use the consistent hashing algorithm to generate a node hash value for each available node; map the node hash value of each available node to the hash ring. The scheduler maintains the hash ring based on the node resources and node status of the node obtained in real time.

[0062] In one implementation, when the system starts, the available nodes (servers) in the cluster are first identified. In order to reduce task migration caused by node fluctuations, multiple virtual nodes (i.e., points in the hash ring) can be created for each physical node. For example, if there are 5 physical nodes, 10 virtual nodes can be created for each node, thus forming a total of 50 virtual nodes. All nodes will be registered with the central task scheduler, and each node will provide its processing capacity and initial status. The central task scheduler will maintain a task-node mapping based on a ring structure, which enables each virtual node to be connected to its adjacent nodes, making it easy to find the node corresponding to the task in a clockwise manner. Then the central task scheduler will monitor the CPU, memory, network latency and other parameters of each node in real time. Each node is equipped with a health check module to report its status to the scheduler regularly. If an abnormality is found, the scheduler will mark the node as "pending maintenance" and adjust the task allocation in time to avoid assigning tasks to the node. For "offline" nodes, the subsequent scheduler will not assign tasks to the node. This implementation can monitor the node status in real time and automatically adjust the task allocation.

[0063] In one implementation, in the above S103, the candidate node whose node hash value is closest to the task hash identifier can be searched clockwise in the hash ring according to the node hash values ​​of all available nodes; if the node resources of the candidate node can process the task to be distributed, the candidate node is determined as the target node; if the node resources of the candidate node cannot process the task to be distributed, the next candidate node is searched clockwise in the hash ring until the node resources of the next candidate node can process the task to be distributed, and the next candidate node is determined as the target node. Specifically, starting from the task hash identifier of the task to be distributed, the hash ring can be searched clockwise until the first node is found, which is the distribution target of the task. If the task hash identifier happens to fall on the hash value of a node, the node can be directly used as the distribution target. If no suitable node is found, the clockwise search continues, and it may eventually return to the starting position of the ring.

[0064] In some cases, each node in the node cluster includes multiple virtual nodes created for each physical node in the cluster. In this case, in the above S104, the task to be distributed can be assigned to the physical node where the target node is located. For example, assume that the virtual nodes A1 (hash value 10) and A2 (hash value 20) of physical node A, the virtual nodes B1 (hash value 30) and B2 (hash value 40) of physical node B, and the virtual node C1 (hash value 50) of physical node C. If the hash value of the task is 35, start searching clockwise from 35 on the hash ring. The first virtual node that exceeds 35 is B2 (hash value 40), so the task will be assigned to physical node B.

[0065] In the embodiment of the present invention, a node change strategy is also set. For example, when a new node is added, a virtual node is created for the new node and its hash value is calculated, and the tasks to be received are reallocated. After the node hash value of each available node is mapped to the hash ring, the example also includes: when a new physical node is online, a new virtual node is created for the new physical node; a consistent hashing algorithm is used to generate a node hash value for the new virtual node, and the node hash value of the new virtual node is mapped to the hash ring. These tasks were originally allocated by nodes with higher hash values. After being processed by the new nodes, a smaller number of tasks will be reallocated due to the addition of nodes, which can ensure improved processing efficiency.

[0066] For example, when there is a node offline, if the offline node is a physical node, it is necessary to find all virtual nodes of the node and assign its tasks to the next node in a clockwise direction to ensure the continuity and stability of the task. The node change strategy can ensure the stability of task allocation. Even if the number of nodes changes, only a small number of tasks need to be reallocated. When distributing tasks, tasks are preferentially assigned to nodes with low load and healthy status to improve processing efficiency. If the physical node of the corresponding virtual node has a high load, continue to search for the next virtual node in a clockwise direction until a suitable node is found.

[0067] In the embodiment of the present invention, a data redundancy and fault tolerance mechanism is also provided. For example, after allocating the task to be distributed to the target node, the following is also included:

[0068] Using distributed snapshot technology, the task execution process of the tasks to be distributed is regularly copied to redundant servers. The data redundancy mechanism is used for state recovery to prevent data loss.

[0069] In the embodiment of the present invention, a fault recovery mechanism is also provided, for example, using a distributed snapshot technology to regularly copy the task execution process of the task to be distributed to the redundant server, and further comprising:

[0070] If the target node fails, the task execution process of the task to be distributed is read from the redundant server;

[0071] Determine whether the task to be distributed has been completed according to the task execution process of the task to be distributed;

[0072] If not, the tasks to be distributed will be reallocated to other nodes. When using the fault recovery mechanism, the system is compatible with rolling upgrades to ensure that the execution of existing tasks is not affected during the upgrade.

[0073] The embodiment of the present invention is described below with a specific embodiment:

[0074] 1. System initialization

[0075] When the system starts, the available nodes (servers) in the cluster are first identified. In order to reduce task migration caused by node fluctuations, multiple virtual nodes (i.e., points in the hash ring) can be created for each physical node. For example, if there are 5 physical nodes, 10 virtual nodes can be created for each node, resulting in a total of 50 virtual nodes. All nodes will be registered with the central task scheduler, and each node will provide its processing power and initial status. The central task scheduler will maintain a task and node mapping based on a ring structure, which enables each virtual node to be connected to its adjacent nodes, making it easy to find the corresponding node for the task in a clockwise direction.

[0076] 2. Node status monitoring

[0077] The central task scheduler will monitor the CPU, memory, network latency and other parameters of each node in real time. Each node is equipped with a health check module to regularly report its status to the scheduler. If an abnormality is found, the scheduler will mark the node as "pending maintenance" and adjust the task allocation in time to avoid assigning tasks to the node.

[0078] 3. Task distribution strategy (Use the consistent hashing algorithm to assign a hash identifier to each task. This algorithm ensures the stability of task distribution, that is, when the number of nodes changes, only a few tasks need to be reallocated. When distributing tasks, tasks are preferentially assigned to nodes with low load and healthy status to improve processing efficiency.)

[0079] 3.1 Task hash value calculation

[0080] Before assigning a task, a certain hash function (such as MD5 or SHA-1) is used to calculate the hash value of the task. The hash value indicates the position of the task on the hash ring.

[0081] 3.2 Hash Ring Structure

[0082] In the consistent hashing architecture, the hash values ​​of nodes and tasks are mapped to a logical ring structure (hash ring). The range of the ring is generally 0 to a maximum value (for example, 2^32 or 2^64).

[0083] 3.3 Virtual Nodes

[0084] Each physical node will create multiple virtual nodes and calculate hash values ​​for them. These virtual nodes are distributed on the hash ring to increase the granularity of the mapping and ensure that the load can be more evenly distributed to each physical node.

[0085] 3.4 Find the target node

[0086] After the task hash value calculation is completed, it is necessary to determine which node the task should be assigned to: starting from the hash value of the task, search clockwise on the hash ring until the first virtual node is found. The physical node associated with this node is the assignment target of the task. Handling edge cases: If the hash value of the task happens to fall on the hash value of a virtual node, the task is directly assigned to the physical node corresponding to the virtual node. If no suitable virtual node is found, it is necessary to continue searching clockwise, and may eventually return to the starting position of the ring.

[0087] 3.5 Return the allocation node

[0088] Once the target virtual node is determined, its corresponding physical node is the processing node of the task. For example: suppose physical node A has virtual nodes A1 (hash value 10) and A2 (hash value 20), physical node B has virtual nodes B1 (hash value 30) and B2 (hash value 40), and physical node C has virtual node C1 (hash value 50). If the hash value of the task is 35, start searching clockwise from 35 on the hash ring. The first virtual node that exceeds 35 is B2 (hash value 40), so the task will be assigned to physical node B.

[0089] 3.6 Node Change Strategy

[0090] When a new node is added, a virtual node is created for the new node and its hash value is calculated, and the tasks to be received are redistributed. These tasks were originally assigned to nodes with higher hash values, and a smaller number of tasks will be redistributed due to the addition of the node. If a node goes offline, all virtual nodes of the node need to be found, and its tasks are assigned to the next node in a clockwise direction to ensure the continuity and stability of the tasks. This algorithm ensures the stability of task allocation, and even if the number of nodes changes, only a small number of tasks need to be redistributed. When distributing tasks, tasks are preferentially assigned to nodes with lower load and healthy status to improve processing efficiency. If the physical node load of the corresponding virtual node is high, continue to search for the next virtual node clockwise until a suitable node is found.

[0091] 4. Data redundancy and fault tolerance

[0092] Regularly replicate task execution status and intermediate results to redundant servers to ensure rapid recovery in the event of a node failure.

[0093] Use distributed snapshot technology to store snapshots of the entire system status at regular intervals.

[0094] 5. Failure recovery

[0095] If node A fails, the scheduler will trigger the fault recovery process, read the task status from the redundant server, and reallocate the unfinished tasks to other nodes.

[0096] The system is compatible with rolling upgrades to ensure that the execution of existing tasks is not affected during the upgrade.

[0097] In modern computing environments, task distribution systems coordinate task execution between multiple nodes to improve resource utilization and processing efficiency. However, traditional task distribution mechanisms are prone to node failures, resulting in uneven or failed task distribution. High-availability task distribution mechanisms are designed to address these problems and ensure that the system can still reliably allocate and execute tasks in the face of node failures. The present invention proposes a highly available task distribution method that uses a distributed architecture and a fault-tolerant algorithm to ensure that the stability of the overall task distribution is not affected when any node fails. The innovation is that the system can dynamically detect the node status and adjust the task distribution strategy according to the real-time load and health status of the node.

[0098] Embodiment 2:

[0099] Based on the above embodiments, Figure 2 A structural diagram of a highly available task distribution system provided by an embodiment of the present invention, including a node cluster and a task scheduler;

[0100] The node cluster is used to register all nodes in the node cluster to the task scheduler;

[0101] The task scheduler is used to implement the steps of the high-availability task distribution method in the above embodiment.

[0102] Optionally, the system may further include a redundant server for periodically replicating the task execution process of the task to be distributed using a distributed snapshot technology.

[0103] Embodiment 3:

[0104] Based on the above embodiments, Figure 3 A schematic diagram of a highly available task distribution device provided by an embodiment of the present invention, the device comprising:

[0105] The acquisition module 301 is used to acquire tasks to be distributed;

[0106] The hash identifier allocation module 302 is used to use a consistent hashing algorithm to allocate a task hash identifier to the task to be distributed, and map the task hash identifier to a hash ring; the hash ring is mapped with node hash values ​​of all available nodes in the node cluster;

[0107] The node search module 303 is used to search, in the hash ring, for a target node whose node resources can process the task to be distributed and whose node hash value is closest to the task hash identifier according to the node hash values ​​of all available nodes;

[0108] The task allocation module 304 is used to allocate the task to be distributed to the target node.

[0109] Furthermore, the hash identifier allocation module 302 is specifically used to use the MD5 algorithm or the SHA-1 algorithm to allocate a task hash identifier for the task to be distributed; wherein the node hash values ​​of all available nodes are generated using the MD5 algorithm or the SHA-1 algorithm.

[0110] Furthermore, it also includes a registration module, which is used to locally register all nodes in the node cluster and obtain the node resources and node status of each node; the node resources include CPU resources, memory resources and network delay parameters; the node status is normal, waiting for maintenance and offline; for each of the nodes, if the node status of the node is normal, the node is used as an available node; using the consistent hashing algorithm, a node hash value is generated for each of the available nodes; and the node hash value of each of the available nodes is mapped to the hash ring.

[0111] Furthermore, the node search module 303 is specifically used to search for a candidate node whose node hash value is closest to the task hash identifier in a clockwise manner in the hash ring according to the node hash values ​​of all available nodes; if the node resources of the candidate node can process the task to be distributed, the candidate node is determined as the target node; if the node resources of the candidate node cannot process the task to be distributed, the next candidate node is continued to be searched clockwise in the hash ring until the node resources of the next candidate node can process the task to be distributed, and the next candidate node is determined as the target node.

[0112] Furthermore, each node in the node cluster includes multiple virtual nodes created for each physical node in the cluster; the task allocation module 304 is specifically used to allocate the task to be distributed to the physical node where the target node is located.

[0113] Furthermore, it also includes an update module, which is used to create a new virtual node for a new physical node when the new physical node comes online; generate a node hash value for the new virtual node using the consistent hashing algorithm, and map the node hash value of the new virtual node to the hash ring.

[0114] Furthermore, a redundancy module is included, which is used to use distributed snapshot technology to regularly copy the task execution process of the task to be distributed to the redundant server.

[0115] Furthermore, it also includes a fault recovery module, which is used to read the task execution process of the task to be distributed from the redundant server if the target node fails; determine whether the task to be distributed is completed based on the task execution process of the task to be distributed; if not, reallocate the task to be distributed to other nodes.

[0116] Embodiment 4:

[0117] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Based on the above embodiments, an electronic device is further provided in an embodiment of the present invention, including a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404;

[0118] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 executes the steps of implementing the high-availability task distribution method in the above embodiment.

[0119] For the specific implementation process, please refer to the above embodiment, and the similarities are not repeated here.

[0120] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0121] The communication interface 402 is used for communication between the electronic device and other devices.

[0122] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0123] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital instruction processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0124] Embodiment 5:

[0125] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of the high-availability task distribution method in the above embodiments.

[0126] For the specific implementation process, please refer to the above embodiment, and the similarities are not repeated here.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A highly available task distribution method, characterized in that: The method comprises: Get the tasks to be distributed; Using a consistent hashing algorithm, assigning a task hash identifier to the task to be distributed, and mapping the task hash identifier to a hash ring; the hash ring is mapped with node hash values ​​of all available nodes in the node cluster; According to the node hash values ​​of all available nodes, searching in the hash ring for a target node whose node resources can process the task to be distributed and whose node hash value is closest to the task hash identifier; Allocate the task to be distributed to the target node.

2. The method according to claim 1, characterized in that The using of the consistent hashing algorithm to assign a task hash identifier to the task to be distributed includes: Use the MD5 algorithm or the SHA-1 algorithm to assign a task hash identifier to the task to be distributed; The node hash values ​​of all available nodes are generated using the MD5 algorithm or the SHA-1 algorithm.

3. The method according to claim 1, characterized in that Before receiving the task to be distributed, the method further includes: Register all nodes in the node cluster locally and obtain the node resources and node status of each node; the node resources include CPU resources, memory resources and network delay parameters; the node status is normal, waiting for maintenance and offline; For each of the nodes, if the node status of the node is normal, the node is regarded as an available node; Using the consistent hashing algorithm, generating a node hash value for each of the available nodes; The node hash value of each of the available nodes is mapped to the hash ring.

4. The method according to claim 3, characterized in that The node resources that can process the task to be distributed and the target node whose node hash value is closest to the task hash identifier in the hash ring according to the node hash values ​​of all available nodes include: According to the node hash values ​​of all available nodes, searching clockwise in the hash ring for a candidate node whose node hash value is closest to the task hash identifier; If the node resources of the candidate node are capable of processing the task to be distributed, determining the candidate node as the target node; If the node resources of the candidate node cannot process the task to be distributed, continue to search for the next candidate node clockwise in the hash ring until the node resources of the next candidate node can process the task to be distributed, and determine the next candidate node as the target node.

5. The method according to claim 3, characterized in that Each node in the node cluster includes a plurality of virtual nodes created for each physical node in the cluster; The allocating the to-be-distributed task to the target node comprises: Allocate the task to be distributed to the physical node where the target node is located.

6. The method according to claim 3, characterized in that After mapping the node hash value of each of the available nodes to the hash ring, the method further includes: When a new physical node comes online, a new virtual node is created for the new physical node; a node hash value is generated for the new virtual node using the consistent hashing algorithm, and the node hash value of the new virtual node is mapped to the hash ring.

7. The method according to any one of claims 1 to 6, characterized in that: After allocating the task to be distributed to the target node, the method further includes: The distributed snapshot technology is used to regularly copy the task execution process of the task to be distributed to the redundant server.

8. The method according to claim 7, characterized in that After the distributed snapshot technology is used to periodically copy the task execution process of the task to be distributed to the redundant server, the method further includes: If the target node fails, the task execution process of the task to be distributed is read from the redundant server; Determining whether the task to be distributed has been completed according to the task execution process of the task to be distributed; If not, the task to be distributed is reallocated to other nodes.

9. A highly available task distribution system, characterized in that: The system includes a node cluster and a task scheduler; The node cluster is used to register all nodes in the node cluster to the task scheduler; The task scheduler is used to implement the steps of the high-availability task distribution method described in any one of claims 1 to 8 above.

10. A highly available task distribution device, characterized in that: The device comprises: The acquisition module is used to obtain tasks to be distributed; A hash identifier allocation module is used to use a consistent hashing algorithm to allocate a task hash identifier to the task to be distributed, and map the task hash identifier to a hash ring; the hash ring is mapped with node hash values ​​of all available nodes in the node cluster; A node search module, used to search in the hash ring for a target node whose node resources can process the task to be distributed and whose node hash value is closest to the task hash identifier according to the node hash values ​​of all available nodes; The task allocation module is used to allocate the task to be distributed to the target node.

Images