Task allocation method and device, equipment and storage medium

Abstract

The invention provides a task allocation method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps that node information in a cluster is obtained, whether the cluster is abnormal or not is determined according to the node information, tasks are redistributed on the basis of a preset algorithm under the condition that the cluster is abnormal, and the stability of a system is improved.

Description

Background Art

[0002] With the development of the times, the scale of intelligent models has gradually increased and the complexity has gradually increased. With the development of intelligent models, the computing resources of a single computing node can no longer meet the training needs of intelligent models. In related technologies, distributed training methods are used to complete the training of intelligent models. The task is divided into multiple subtasks, and then different subtasks are trained using different points. However, the stability of current distributed training is poor. Summary of the invention

[0003] The present disclosure provides a task allocation method, apparatus, device and storage medium, which improve the stability of distributed training at least to a certain extent.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0005] According to one aspect of the present disclosure, a task allocation method is provided, comprising:

[0006] Get node information in the cluster;

[0007] Determine whether the cluster is abnormal based on the node information, where cluster abnormalities include adding new nodes to the cluster and failure of nodes executing tasks in the cluster;

[0008] In the event of cluster anomalies, tasks are reallocated based on a preset algorithm.

[0009] In one embodiment of the present disclosure, the cluster anomaly includes a failure of a node executing a task in the cluster;

[0010] In the event of cluster anomalies, tasks are reallocated based on preset algorithms, including:

[0011] In the event of a failure of a node executing a task in the cluster, the task is assigned to the node executing the task and the redundant node based on a redundancy algorithm;

[0012] Reassign tasks based on preset algorithms.

[0013] In one embodiment of the present disclosure, tasks are allocated to nodes that execute tasks and / or redundant nodes based on a redundancy algorithm, including:

[0014] Determine the failure probability of the nodes performing the task;

[0015] Tasks are allocated to nodes that perform tasks and / or redundant nodes based on the failure probability of the nodes that perform tasks and a preset fault tolerance threshold.

[0016] In one embodiment of the present disclosure, allocating tasks to nodes that perform tasks and / or redundant nodes based on the failure probability of the nodes that perform tasks and a preset fault tolerance threshold includes:

[0017] When the failure probability is less than a first fault tolerance threshold, assigning the task to a node that executes the task;

[0018] When the failure probability is not less than the first fault tolerance threshold and less than the second fault tolerance threshold, allocating tasks based on the output capability data of the task execution node and the output capability data of the redundant node;

[0019] When the failure probability is not less than the second fault tolerance threshold, the task is allocated to the redundant node.

[0020] In one embodiment of the present disclosure, the method further includes:

[0021] According to the preset cycle, tasks are reallocated based on node information and task scheduling model. Node information includes node load information, node hardware health data, and node network connection quality data.

[0022] In one embodiment of the present disclosure, the cluster anomaly includes adding a new node;

[0023] In the event of cluster anomalies, tasks are reallocated based on preset algorithms, including:

[0024] When adding new nodes to the cluster, determine the load balancing factor for each node;

[0025] Tasks are reallocated based on at least one of a load balancing factor of each node, node information, and a Lagrange multiplier algorithm.

[0026] In one embodiment of the present disclosure, the method further includes:

[0027] Store data in the node according to a preset period;

[0028] In case of node failure in the cluster, the task is completed based on the stored data and the time when the data was stored.

[0029] According to another aspect of the present disclosure, there is provided a task allocation device, comprising:

[0030] The acquisition module is used to obtain node information in the cluster;

[0031] A determination module is used to determine whether the cluster is abnormal based on the node information, wherein the cluster abnormality includes the addition of a new node in the cluster and the failure of a node executing a task in the cluster;

[0032] The allocation module is used to reallocate tasks based on a preset algorithm when a cluster is abnormal.

[0033] In one embodiment of the present disclosure, the cluster anomaly includes a failure of a node executing a task in the cluster;

[0034] Distribution module, including:

[0035] A first allocation unit, configured to allocate tasks to nodes executing tasks and / or redundant nodes based on a redundancy algorithm when a node executing tasks in the cluster fails;

[0036] The second allocating unit is used to reallocate tasks based on a preset algorithm.

[0037] In one embodiment of the present disclosure, the first allocation unit includes:

[0038] A determination element for determining a failure probability of a node executing a task;

[0039] The allocation element is used to allocate tasks to nodes that perform tasks and / or redundant nodes based on the failure probability of the nodes that perform tasks and a preset fault tolerance threshold.

[0040] In one embodiment of the present disclosure, a dispensing element comprises:

[0041] A first allocation sub-element, configured to allocate the task to a node that executes the task when the failure probability is less than a first fault tolerance threshold;

[0042] A second allocation sub-element, configured to allocate tasks based on the output capability data of the task execution node and the output capability data of the redundant node when the failure probability is not less than the first fault tolerance threshold and less than the second fault tolerance threshold;

[0043] The third allocation subcomponent is used to allocate the task to the redundant node when the failure probability is not less than the second fault tolerance threshold.

[0044] In one embodiment of the present disclosure, the device further includes:

[0045] The second allocation module is used to reallocate tasks according to a preset period, based on node information and a task scheduling model. The node information includes node load information, node hardware health data, and node network connection quality data.

[0046] In one embodiment of the present disclosure, the cluster anomaly includes adding a new node;

[0047] Distribution module, including:

[0048] A second determination unit is used to determine the load balancing factor of each node when a new node is added to the cluster;

[0049] The second allocating unit is used to reallocate tasks based on at least one of a load balancing factor of each node, node information, and a Lagrange multiplier algorithm.

[0050] In one embodiment of the present disclosure, the device further includes:

[0051] A storage module, used to store data in the node according to a preset period;

[0052] The completion module is used to complete the task based on the stored data and the time when the data was stored in the event of a node failure in the cluster.

[0053] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned task allocation method by executing the executable instructions.

[0054] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned task allocation method is implemented.

[0055] The task allocation method, apparatus, device and storage medium provided by the embodiments of the present disclosure obtain node information in a cluster, determine whether the cluster is abnormal based on the node information, and reallocate tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0058] Figure 1 A flowchart of a task allocation method in an embodiment of the present disclosure is shown;

[0059] Figure 2 Another task allocation method flow chart in the embodiment of the present disclosure is shown;

[0060] Figure 3 A flowchart of another task allocation method in an embodiment of the present disclosure is shown;

[0061] Figure 4 A flowchart of another task allocation method in an embodiment of the present disclosure is shown;

[0062] Figure 5 A flowchart of another task allocation method in an embodiment of the present disclosure is shown;

[0063] Figure 6 A flowchart of another task allocation method in an embodiment of the present disclosure is shown;

[0064] Figure 7 A task allocation device architecture diagram is shown in an embodiment of the present disclosure;

[0065] Figure 8 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0066] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0067] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0068] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0069] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0070] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0071] With the rapid development of deep learning technology, especially in the fields of natural language processing, computer vision and recommendation systems, the scale and complexity of models continue to grow, and the computing resources of a single node can no longer meet the training needs. The training of large-scale models such as GPT and BERT usually requires distributed computing, dividing the task into multiple subtasks for parallel execution. However, the complexity of distributed training also brings new challenges, especially node failures, network delays and load imbalances, which can cause interruptions and delays in the training process, affecting the efficiency and performance of the model.

[0072] The current distributed training technology has weak flexibility and fault tolerance in dealing with system failures and task scheduling, and cannot effectively guarantee the continuity and stability of training. Especially in large-scale model training, any node failure may cause the entire system to be interrupted, wasting a lot of computing resources. Therefore, how to improve the fault tolerance of distributed systems, optimize task allocation, and ensure data consistency has become an important issue that needs to be solved in the field of deep learning.

[0073] It should be noted that the present disclosure can be applied to the training scenarios of intelligent models. For example, in the model training process of the autonomous driving system, the autonomous driving technology relies on complex deep learning models, and the training of these models requires the collaborative work of a large number of nodes. The fault-tolerant mechanism of the present disclosure ensures that the training process is not interrupted when a node fails, significantly improves the stability and efficiency of model training, and meets the high requirements of the autonomous driving system for safety and continuity. For example, in the large-scale language model training process, the present disclosure is applicable to the distributed training of large-scale language models such as GPT and BERT, which can ensure data consistency and dynamic task scheduling, improve the efficiency and stability of model training, and ensure the efficient implementation of the model training process in a complex environment.

[0074] In order to solve the above problems, the embodiments of the present disclosure provide a task allocation method, apparatus, device and storage medium.

[0075] Figure 1 A flow chart of a task allocation method in an embodiment of the present disclosure is shown.

[0076] like Figure 1 As shown, the method may include:

[0077] S110, obtaining node information in the cluster.

[0078] In some embodiments, the cluster may be a distributed cluster.

[0079] In some embodiments, any node in the cluster can complete the training of the model.

[0080] S120, determining whether the cluster is abnormal according to the node information, wherein the cluster abnormality includes adding a new node in the cluster and a failure of a node executing a task in the cluster.

[0081] In some embodiments, the node information may include a node identifier. When it is determined that the node identifier in the cluster has changed, it can be determined that a new node has been added to the cluster and an abnormality has occurred in the cluster.

[0082] In some embodiments, the node information may include node parameter information, and the node parameter information may be converted into a multi-dimensional vector, and whether a node fails may be determined based on a comparison between the multi-dimensional vector and a preset threshold.

[0083] Exemplarily, the node parameter information is Si(t), and it is checked whether Si(t) exceeds a threshold value tau. If Si(t) does not exceed the threshold value, the node information is continuously acquired. If Si(t) exceeds the threshold value, it is determined that the node is faulty.

[0084] S130, in case of cluster abnormality, reallocate tasks based on a preset algorithm.

[0085] In some embodiments, the preset algorithm may include an intelligent model and a Lagrange multiplier method.

[0086] The task allocation method provided by the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0087] Figure 2 A flow chart of a task allocation method in an embodiment of the present disclosure is shown.

[0088] like Figure 2 As shown, the method may include:

[0089] S210, obtaining node information in the cluster;

[0090] S220: Determine, based on the node information, that a node executing a task in the cluster fails.

[0091] In some embodiments, whether a node fails may be determined based on a probability of the node failing and a preset threshold.

[0092] In some embodiments, the node that performs the task may be a master node.

[0093] S230 , when a node executing a task in the cluster fails, assigning the task to the node executing the task and / or redundant nodes based on a redundancy algorithm.

[0094] In some embodiments, the node that executes the task may be a primary node or a redundant node. The redundant nodes may be configured in advance, and one primary node may correspond to multiple redundant nodes.

[0095] S240, reallocate tasks based on a preset algorithm.

[0096] The task allocation method provided by the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0097] Figure 3 A flow chart of another task allocation method in an embodiment of the present disclosure is shown.

[0098] like Figure 3 As shown, the method may include:

[0099] S310, obtaining node information in the cluster;

[0100] S320: Determine, based on the node information, that a node executing a task in the cluster fails.

[0101] S330, determining the failure probability of the node executing the task.

[0102] S340, allocating tasks to nodes that execute tasks and / or redundant nodes based on the failure probability of the nodes that execute tasks and a preset fault tolerance threshold.

[0103] In some embodiments, allocating tasks to nodes performing tasks and / or redundant nodes based on the failure probability of the nodes performing tasks and a preset fault tolerance threshold includes:

[0104] When the failure probability is less than a first fault tolerance threshold, assigning the task to a node that executes the task;

[0105] When the failure probability is not less than the first fault tolerance threshold and less than the second fault tolerance threshold, allocating tasks based on the output capability data of the task execution node and the output capability data of the redundant node;

[0106] When the failure probability is not less than the second fault tolerance threshold, the task is allocated to the redundant node.

[0107] For example, the output of the task execution node is R_primary, and the output of the redundant node is R_redundant. Under normal working conditions, the system uses R_primary to update the model. The switching mechanism can be expressed by the following formula:

[0108] R_output=

[0109] When P_fail_primary < threshold_1, use R_primary;

[0110] When threshold_1 ≤ P_fail_primary < threshold_2, use the weighted average of R_primary and R_redundant, w_1 * R_primary + w_2 * R_redundant;

[0111] When P_fail_primary ≥ threshold_2, completely switch to R_redundant.

[0112] Among them, P_fail_primary is the failure probability of the primary node, threshold_1 and threshold_2 are preset fault tolerance thresholds, and w_1 and w_2 are weighting coefficients, satisfying w_1 + w_2 = 1. This mechanism ensures the continuity of training in different fault risks by dynamically adjusting the weights of node outputs.

[0113] S350, re - allocate tasks based on a preset algorithm.

[0114] The task allocation method provided by the embodiments of the present disclosure improves the stability of the system by obtaining node information in the cluster, determining whether the cluster is abnormal according to the node information, and re - allocating tasks based on a preset algorithm in the case of cluster abnormality.

[0115] Figure 4 Shows another flowchart of the task allocation method in the embodiments of the present disclosure.

[0116] As Figure 4 shown, the method may include:

[0117] S410, obtain node information in the cluster;

[0118] S420, determine whether the cluster is abnormal according to the node information, where cluster abnormality includes adding new nodes in the cluster and the nodes executing tasks in the cluster failing;

[0119] S430, in the case of cluster abnormality, re - allocate tasks based on a preset algorithm.

[0120] S440, re - allocate tasks according to the node information and the task scheduling model at a preset period, where the node information includes node load information, node hardware health data, and node network connection quality data.

[0121] In some embodiments, tasks can be re - allocated based on an optimization model. Exemplarily, the optimization goal of task allocation is to minimize the total computational overhead C(t) of the system, defined as:

[0122] C(t)=sum(w_1*L_i(t)+w_2*H_i(t)+w_3*N_i(t)), i ranges from 1 to N.

[0123] in:

[0124] L_i(t) is the current computing load of node i, which can be expressed as L_i(t)=L_0+delta_L, where L_0 is the basic load and delta_L is the fluctuating load.

[0125] H_i(t) is the hardware health status of node i, defined as H_i(t)=1 / (1+exp(-alpha_h*(H_0-H_i))), where H_0 is the ideal hardware status and alpha_h is the sensitivity coefficient of the health status.

[0126] N_i(t) is the network connection status of node i, expressed as N_i(t)=exp(-beta_n*D_i(t)), where D_i(t) is the network delay of the node and beta_n is the sensitivity coefficient of the network status.

[0127] By solving the above optimization model, the system can dynamically adjust the distribution of tasks under different conditions to avoid a single node becoming a bottleneck, thereby ensuring the overall stability of the system.

[0128] The task allocation method provided by the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0129] Figure 5 A flow chart of another task allocation method in an embodiment of the present disclosure is shown.

[0130] like Figure 5 As shown, the method may include:

[0131] S510, obtaining node information in the cluster;

[0132] S520: Determine whether to add a new node according to the node information.

[0133] In some embodiments, adding a new node may include completing repair of a failed node.

[0134] S530: When a new node is added to the cluster, determine a load balancing factor of each node.

[0135] In some embodiments, the load balancing factor of each node may be determined based on the load of each node, the total system load, and the total number of nodes.

[0136] Exemplarily, the load balancing factor may be determined in the following manner:

[0137] beta_i(t)=(L_i(t) / L_total(t))-(1 / (N+1)),

[0138] Where L_i(t) is the current load of node i, L_total(t) is the total system load, N is the total number of current nodes, and beta_i(t) is the load balancing factor.

[0139] S540: reallocate tasks based on at least one of a load balancing factor of each node, node information, and a Lagrange multiplier algorithm.

[0140] Exemplarily, the Lagrange multiplier method recalculates the task allocation X_i.

[0141] Minimize L(X_i)=C(X_i)+lambda*(sum(X_i)-T_total) to find the optimal task allocation solution. The Lagrange multiplier method is a commonly used algorithm in this field and is not specifically limited in this disclosure.

[0142] In some embodiments, the task data allocated to each node may be determined based on a load balancing factor corresponding to each node.

[0143] The task allocation method provided by the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0144] Figure 6 A flow chart of another task allocation method in an embodiment of the present disclosure is shown.

[0145] like Figure 6 As shown, the method may include:

[0146] S610, obtaining node information in the cluster;

[0147] S620, determining whether the cluster is abnormal according to the node information, wherein the cluster abnormality includes adding a new node to the cluster and a node executing a task in the cluster failing;

[0148] S630: In case of cluster abnormality, reallocate tasks based on a preset algorithm.

[0149] S640: Store the data in the node according to a preset period.

[0150] In some embodiments, data within a node may be stored based on data snapshots.

[0151] S650, when a node in the cluster fails, complete the task based on the stored data and the time when the data was stored.

[0152] In some embodiments, completing a task based on the stored data and the time at which the data was stored may include:

[0153] The storage data with the storage data time closest to the current time is taken as the target data;

[0154] Complete training based on target data.

[0155] Exemplarily, D_recovered(t)=max(S_i(T)), where T≤t.

[0156] D_recovered(t) is the target data, S_i(T) is the data snapshot of node i at time T. By finding the data snapshot corresponding to the maximum timestamp, the system can quickly recover to the state before the failure and reduce the risk of data inconsistency.

[0157] The task allocation method provided by the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0158] Based on the same inventive concept, the present disclosure also provides a task allocation device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0159] Figure 7 A structural diagram of a task allocation device in an embodiment of the present disclosure is shown.

[0160] like Figure 7 As shown, the apparatus 700 may include:

[0161] The acquisition module 710 is used to obtain node information in the cluster;

[0162] A determination module 720 is used to determine whether the cluster is abnormal according to the node information, wherein the cluster abnormality includes the addition of a new node in the cluster and the failure of a node executing a task in the cluster;

[0163] The allocation module 730 is used to reallocate tasks based on a preset algorithm when a cluster is abnormal.

[0164] The task allocation device provided in the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0165] In one embodiment of the present disclosure, the cluster anomaly includes a failure of a node executing a task in the cluster;

[0166] Distribution module, including:

[0167] A first allocation unit, configured to allocate tasks to nodes executing tasks and / or redundant nodes based on a redundancy algorithm when a node executing tasks in the cluster fails;

[0168] The second allocating unit is used to reallocate tasks based on a preset algorithm.

[0169] The task allocation device provided in the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0170] In one embodiment of the present disclosure, the first allocation unit includes:

[0171] A determination element for determining a failure probability of a node executing a task;

[0172] The allocation element is used to allocate tasks to nodes that perform tasks and / or redundant nodes based on the failure probability of the nodes that perform tasks and a preset fault tolerance threshold.

[0173] The task allocation device provided in the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0174] In one embodiment of the present disclosure, a dispensing element comprises:

[0175] A first allocation sub-element, configured to allocate the task to a node that executes the task when the failure probability is less than a first fault tolerance threshold;

[0176] A second allocation sub-element, configured to allocate tasks based on the output capability data of the task execution node and the output capability data of the redundant node when the failure probability is not less than the first fault tolerance threshold and less than the second fault tolerance threshold;

[0177] The third allocation subcomponent is used to allocate the task to the redundant node when the failure probability is not less than the second fault tolerance threshold.

[0178] The task allocation device provided in the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0179] In one embodiment of the present disclosure, the device further includes:

[0180] The second allocation module is used to reallocate tasks according to a preset period, based on node information and a task scheduling model. The node information includes node load information, node hardware health data, and node network connection quality data.

[0181] In one embodiment of the present disclosure, the cluster anomaly includes adding a new node;

[0182] Distribution module, including:

[0183] A second determination unit is used to determine the load balancing factor of each node when a new node is added to the cluster;

[0184] The second allocating unit is used to reallocate tasks based on at least one of a load balancing factor of each node, node information, and a Lagrange multiplier algorithm.

[0185] The task allocation device provided in the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0186] In one embodiment of the present disclosure, the device further includes:

[0187] A storage module, used to store data in the node according to a preset period;

[0188] The completion module is used to complete the task based on the stored data and the time when the data was stored in the event of a node failure in the cluster.

[0189] The task allocation device provided in the embodiment of the present disclosure can be used to execute the positioning methods provided in the above-mentioned method embodiments. The implementation principles and technical effects are similar and will not be described herein for the sake of simplicity.

[0190] The task allocation device provided in the embodiment of the present disclosure obtains node information in the cluster, determines whether the cluster is abnormal according to the node information, and reallocates tasks based on a preset algorithm when the cluster is abnormal, thereby improving the stability of the system.

[0191] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0192] Refer to the following Figure 8The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0193] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 88, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 88).

[0194] The storage unit stores program codes, which can be executed by the processing unit 88, so that the processing unit 88 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 88 can perform the following steps of the above method embodiment:

[0195] Get node information in the cluster;

[0196] Determine whether the cluster is abnormal based on the node information, where cluster abnormalities include adding new nodes to the cluster and failure of nodes executing tasks in the cluster;

[0197] In the event of cluster anomalies, tasks are reallocated based on a preset algorithm.

[0198] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0199] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0200] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0201] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0202] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0203] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0204] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0205] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0206] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0207] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0208] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0209] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0210] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the example implementation modes described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation mode of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation mode of the present disclosure.

[0211] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A task allocation method, characterized in that: include: Get node information in the cluster; Determine whether the cluster is abnormal according to the node information, wherein the cluster abnormality includes adding a new node to the cluster and a node executing a task in the cluster failing; In the event of cluster anomalies, tasks are reallocated based on a preset algorithm.

2. The method according to claim 1, characterized in that The cluster anomaly includes a failure of a node executing a task in the cluster; In the case of cluster abnormality, reallocating tasks based on a preset algorithm includes: In the event that a node executing a task in the cluster fails, allocating the task to the node executing the task and / or a redundant node based on a redundancy algorithm; Reassign tasks based on preset algorithms.

3. The method according to claim 2, characterized in that The allocating the task to the node executing the task and / or the redundant node based on the redundancy algorithm comprises: Determining a failure probability of a node performing the task; The task is allocated to the node performing the task and / or the redundant node based on the failure probability of the node performing the task and a preset fault tolerance threshold.

4. The method according to claim 3, characterized in that The allocating the task to the node performing the task and / or the redundant node based on the failure probability of the node performing the task and a preset fault tolerance threshold comprises: When the failure probability is less than a first fault tolerance threshold, allocating the task to the node that executes the task; When the failure probability is not less than a first fault tolerance threshold and less than a second fault tolerance threshold, allocating the task based on the output capability data of the task execution node and the output capability data of the redundant node; When the failure probability is not less than a second fault tolerance threshold, the task is allocated to the redundant node.

5. The method according to claim 1, characterized in that The method further comprises: According to a preset cycle, tasks are reallocated based on the node information and a task scheduling model, wherein the node information includes node load information, node hardware health data, and node network connection quality data.

6. The method according to claim 1, characterized in that The cluster anomaly includes adding a new node; In the case of cluster abnormality, reallocating tasks based on a preset algorithm includes: In case a new node is added to the cluster, determining a load balancing factor for each node; Tasks are reallocated based on at least one of a load balancing factor of each node, node information, and a Lagrange multiplier algorithm.

7. The method according to claim 1, characterized in that The method further comprises: Store data in the node according to a preset period; In case of node failure in the cluster, the task is completed based on the stored data and the time when the data was stored.

8. A task allocation device, characterized in that: include: The acquisition module is used to obtain node information in the cluster; A determination module, configured to determine whether the cluster is abnormal according to the node information, wherein the cluster abnormality includes the addition of a new node in the cluster and the failure of a node executing a task in the cluster; The allocation module is used to reallocate tasks based on a preset algorithm when a cluster is abnormal.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the task allocation method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task allocation method according to any one of claims 1 to 7 is implemented.

Images