Task allocation method, device and equipment based on distributed cluster

Abstract

The invention provides a task allocation method, device and equipment based on a distributed cluster and a computer readable storage medium. The method comprises the following steps: acquiring to-be-allocated tasks and determining priorities among the to-be-allocated tasks; selecting a to-be-allocated task node from the low-load nodes according to the task processing capability and the task processing type of the low-load nodes; according to the task number and the priority of the to-be-allocated tasks and the task processing capability and the task processing type of the to-be-allocated task nodes, estimating whether the load of the to-be-allocated task nodes after task allocation exceeds a preset load threshold value or not; if the load does not exceed the preset load threshold, establishing a distribution relationship between the to-be-distributed task and the to-be-distributed task node; and distributing the to-be-distributed task according to the distribution relation. According to the embodiment of the invention, tasks can be distributed more accurately, node overload and software crash are avoided, and load balancing is realized.

Description

Technical Field

[0001] The present application belongs to the technical field of task allocation, and in particular, relates to a distributed cluster-based task allocation method, device, equipment and computer-readable storage medium. Background Art

[0002] When performing tasks, the robot system software can easily exceed its own load capacity, causing the software to crash.

[0003] At present, the task allocation in the related technology is: tasks are allocated to idle nodes in order of priority, which easily leads to overload of some nodes and causes software crash.

[0004] Therefore, how to allocate tasks more accurately, avoid node overload and software crash, and achieve load balancing is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The embodiments of the present application provide a distributed cluster-based task allocation method, device, equipment and computer-readable storage medium, which can more accurately allocate tasks, avoid node overload and software crash, and achieve load balancing.

[0006] In a first aspect, an embodiment of the present application provides a distributed cluster-based task allocation method, characterized by comprising:

[0007] Obtain tasks to be assigned and determine the priorities among the tasks to be assigned;

[0008] Selecting nodes to be assigned tasks from low-load nodes according to the processing capacity and type of tasks of the low-load nodes;

[0009] According to the number and priority of the tasks to be assigned and the processing capability and type of the tasks of the task nodes to be assigned, it is estimated whether the load of the task nodes to be assigned after the task assignment exceeds a preset load threshold;

[0010] If the load does not exceed the preset load threshold, an allocation relationship between the tasks to be allocated and the task nodes to be allocated is established;

[0011] Allocate pending tasks according to the allocation relationship.

[0012] Optionally, obtain the tasks to be assigned and determine the priorities among the tasks to be assigned, including:

[0013] Categorize the tasks to be assigned according to their nature, complexity or relevance;

[0014] Determine the priority among the classified tasks according to their importance, resource requirements, risks and dependencies.

[0015] Optionally, selecting a task node to be assigned from the low-load nodes according to the processing capability of the low-load nodes and the type of the processing tasks includes:

[0016] Determine low-load nodes based on the resource utilization of the nodes;

[0017] Determine the processing capacity of low-load nodes based on their CPU, memory resources, network bandwidth, and storage capacity;

[0018] Determine the type of processing tasks for low-load nodes based on computing requirements, storage needs, or data transfer volume;

[0019] The nodes to be assigned tasks are selected from the low-load nodes according to the processing capacity of the low-load nodes and the types of the processing tasks.

[0020] Optionally, based on the number and priority of the tasks to be assigned and the processing capability and type of the tasks of the task nodes to be assigned, estimating whether the load of the task nodes to be assigned after the task assignment exceeds a preset load threshold includes:

[0021] Calculate the estimated CPU and memory resources consumed by each task on the task node to be assigned based on the number and priority of the tasks to be assigned and the processing capacity and type of the tasks on the task node to be assigned.

[0022] Count the CPU and memory resources that each task is expected to consume on the task node to be assigned, and determine the load of the task node to be assigned after task assignment;

[0023] Determine whether the load of the task node to be assigned after the task is assigned exceeds a preset load threshold; wherein the preset load threshold is pre-set according to the performance of the node and the stability requirements of the system.

[0024] Optionally, also include:

[0025] Evaluate the node computing power of each node in the distributed cluster, and divide the nodes into multiple subclusters according to the order of the node computing power.

[0026] Optionally, also include:

[0027] If there are tasks to be assigned but no allocation relationship has been established, the number of tasks to be assigned that have not established an allocation relationship will be used as the new number of tasks. Starting from selecting low-load nodes from each sub-cluster of the distributed cluster according to the number of tasks of the nodes, the task allocation calculation will be performed again until the allocation relationship of all tasks to be assigned is determined.

[0028] Optionally, also include:

[0029] During the task allocation process, the node load and task execution are continuously monitored. If the node load is close to the preset threshold or there is a shortage of resources, the allocation strategy is adjusted or node resources are increased in a timely manner.

[0030] After allocating the tasks to be allocated according to the allocation relationship, the allocation results are counted and verified.

[0031] In a second aspect, an embodiment of the present application provides a task allocation device based on a distributed cluster, the device comprising:

[0032] A task acquisition module is used to acquire tasks to be assigned and determine the priorities among the tasks to be assigned;

[0033] The task node selection module to be assigned is used to select the task node to be assigned from the low-load nodes according to the processing capacity and type of the task of the low-load nodes;

[0034] The load estimation module is used to estimate whether the load of the task node to be assigned after the task is assigned exceeds a preset load threshold according to the number and priority of the task to be assigned and the processing capacity and type of the task of the task node to be assigned;

[0035] An allocation relationship establishment module, used to establish an allocation relationship between the tasks to be allocated and the task nodes to be allocated if the load does not exceed a preset load threshold;

[0036] The task allocation module is used to allocate the tasks to be allocated according to the allocation relationship.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0038] When the processor executes the computer program instructions, a distributed cluster-based task allocation method is implemented.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, a distributed cluster-based task allocation method is implemented.

[0040] The embodiments of the present application provide a distributed cluster-based task allocation method, device, equipment and computer-readable storage medium, which can more accurately allocate tasks, avoid node overload and software crash, and achieve load balancing.

[0041] The present application embodiment provides a distributed cluster-based task allocation method, which is characterized by comprising:

[0042] Obtain tasks to be assigned and determine the priorities among the tasks to be assigned;

[0043] Selecting nodes to be assigned tasks from low-load nodes according to the processing capacity and type of tasks of the low-load nodes;

[0044] According to the number and priority of the tasks to be assigned and the processing capability and type of the tasks of the task nodes to be assigned, it is estimated whether the load of the task nodes to be assigned after the task assignment exceeds a preset load threshold;

[0045] If the load does not exceed the preset load threshold, an allocation relationship between the tasks to be allocated and the task nodes to be allocated is established;

[0046] Allocate pending tasks according to the allocation relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flowchart of a distributed cluster-based task allocation method provided by an embodiment of the present application;

[0049] Figure 2 It is a structural diagram of a distributed cluster-based task allocation device provided by an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0052] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0053] In order to solve the problems of the prior art, the embodiments of the present application provide a distributed cluster-based task allocation method, apparatus, device and computer-readable storage medium. The following first introduces the distributed cluster-based task allocation method provided by the embodiments of the present application.

[0054] Figure 1 FIG. 1 is a flow chart of a distributed cluster-based task allocation method provided by an embodiment of the present application. Figure 1 As shown, the distributed cluster-based task allocation method is characterized by comprising:

[0055] S101, obtaining tasks to be assigned and determining the priorities among the tasks to be assigned;

[0056] S102, selecting a task node to be assigned from the low-load nodes according to the processing capacity of the low-load nodes and the type of the processing tasks;

[0057] S103, estimating whether the load of the task node to be assigned after the task is assigned exceeds a preset load threshold according to the number of tasks to be assigned, the priority of the task to be assigned, and the processing capacity and type of the task of the task node to be assigned;

[0058] S104: If the load does not exceed the preset load threshold, establish an allocation relationship between the tasks to be allocated and the task nodes to be allocated;

[0059] S105. Allocate the tasks to be allocated according to the allocation relationship.

[0060] In one embodiment, obtaining tasks to be assigned and determining priorities among the tasks to be assigned include:

[0061] Categorize the tasks to be assigned according to their nature, complexity or relevance;

[0062] Determine the priority among the classified tasks according to their importance, resource requirements, risks and dependencies.

[0063] In one embodiment, selecting a task node to be assigned from the low-load nodes according to the processing capability of the low-load nodes and the type of the processing tasks includes:

[0064] Determine low-load nodes based on the resource utilization of the nodes;

[0065] Determine the processing capacity of low-load nodes based on their CPU, memory resources, network bandwidth, and storage capacity;

[0066] Determine the type of processing tasks for low-load nodes based on computing requirements, storage needs, or data transfer volume;

[0067] The nodes to be assigned tasks are selected from the low-load nodes according to the processing capacity of the low-load nodes and the types of the processing tasks.

[0068] In one embodiment, based on the number and priority of the tasks to be assigned and the processing capability and type of the tasks of the task nodes to be assigned, estimating whether the load of the task nodes to be assigned after the task assignment exceeds a preset load threshold includes:

[0069] Calculate the estimated CPU and memory resources consumed by each task on the task node to be assigned based on the number and priority of the tasks to be assigned and the processing capacity and type of the tasks on the task node to be assigned.

[0070] Count the CPU and memory resources that each task is expected to consume on the task node to be assigned, and determine the load of the task node to be assigned after task assignment;

[0071] Determine whether the load of the task node to be assigned after the task is assigned exceeds a preset load threshold; wherein the preset load threshold is pre-set according to the performance of the node and the stability requirements of the system.

[0072] In one embodiment, it further includes:

[0073] Evaluate the node computing power of each node in the distributed cluster, and divide the nodes into multiple subclusters according to the order of the node computing power.

[0074] In one embodiment, it further includes:

[0075] If there are tasks to be assigned but no allocation relationship has been established, the number of tasks to be assigned that have not established an allocation relationship will be used as the new number of tasks. Starting from selecting low-load nodes from each sub-cluster of the distributed cluster according to the number of tasks of the nodes, the task allocation calculation will be performed again until the allocation relationship of all tasks to be assigned is determined.

[0076] In one embodiment, it further includes:

[0077] During the task allocation process, the node load and task execution are continuously monitored. If the node load is close to the preset threshold or there is a shortage of resources, the allocation strategy is adjusted or node resources are increased in a timely manner.

[0078] After allocating the tasks to be allocated according to the allocation relationship, the allocation results are counted and verified.

[0079] Figure 2 1 is a schematic diagram of a distributed cluster-based task allocation device provided by an embodiment of the present application; the distributed cluster-based task allocation device includes:

[0080] The task acquisition module 201 is used to acquire tasks to be assigned and determine the priorities among the tasks to be assigned;

[0081] The task node selection module 202 is used to select the task node to be assigned from the low-load nodes according to the processing capacity and type of the task of the low-load nodes;

[0082] The load estimation module 203 is used to estimate whether the load of the task node to be assigned after the task is assigned exceeds a preset load threshold according to the number and priority of the task to be assigned and the processing capacity and type of the task of the task node to be assigned;

[0083] The allocation relationship establishing module 204 is used to establish an allocation relationship between the tasks to be allocated and the nodes to be allocated if the load does not exceed a preset load threshold;

[0084] The task allocation module 205 is used to allocate tasks to be allocated according to the allocation relationship.

[0085] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.

[0086] The electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0087] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0088] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 302 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 302 may be inside or outside the electronic device. In a particular embodiment, the memory 302 may be a non-volatile solid-state memory.

[0089] In one embodiment, the memory 302 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0090] The processor 301 reads and executes computer program instructions stored in the memory 302 to implement any one of the distributed cluster-based task allocation methods in the above embodiments.

[0091] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0092] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0093] Bus 310 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0094] In addition, in combination with the distributed cluster-based task allocation method in the above embodiment, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the distributed cluster-based task allocation methods in the above embodiment is implemented.

[0095] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0096] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0097] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0098] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0099] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A task allocation method based on distributed cluster, characterized in that: include: Obtain tasks to be assigned and determine the priorities among the tasks to be assigned; Selecting nodes to be assigned tasks from low-load nodes according to the processing capacity and type of tasks of the low-load nodes; According to the number and priority of the tasks to be assigned and the processing capability and type of the tasks of the task nodes to be assigned, it is estimated whether the load of the task nodes to be assigned after the task assignment exceeds a preset load threshold; If the load does not exceed the preset load threshold, an allocation relationship between the tasks to be allocated and the task nodes to be allocated is established; Allocate pending tasks according to the allocation relationship.

2. The distributed cluster-based task allocation method according to claim 1, characterized in that: Obtain tasks to be assigned and determine the priorities among the tasks to be assigned, including: Categorize the tasks to be assigned according to their nature, complexity or relevance; Determine the priority among the classified tasks according to their importance, resource requirements, risks and dependencies.

3. The distributed cluster-based task allocation method according to claim 1, characterized in that: According to the processing capacity and type of tasks of the low-load nodes, the nodes to be assigned tasks are selected from the low-load nodes, including: Determine low-load nodes based on the resource utilization of the nodes; Determine the processing capacity of low-load nodes based on their CPU, memory resources, network bandwidth, and storage capacity; Determine the type of processing tasks for low-load nodes based on computing requirements, storage needs, or data transfer volume; The nodes to be assigned tasks are selected from the low-load nodes according to the processing capacity of the low-load nodes and the types of the processing tasks.

4. The distributed cluster-based task allocation method according to claim 1, characterized in that: According to the number and priority of the tasks to be assigned and the processing capability and type of the tasks of the task nodes to be assigned, it is estimated whether the load of the task nodes to be assigned after the task assignment exceeds the preset load threshold, including: Calculate the estimated CPU and memory resources consumed by each task on the task node to be assigned based on the number and priority of the tasks to be assigned and the processing capacity and type of the tasks on the task node to be assigned. Count the CPU and memory resources that each task is expected to consume on the task node to be assigned, and determine the load of the task node to be assigned after task assignment; Determine whether the load of the task node to be assigned after the task is assigned exceeds a preset load threshold; wherein the preset load threshold is pre-set according to the performance of the node and the stability requirements of the system.

5. The distributed cluster-based task allocation method according to claim 1, characterized in that: Also includes: Evaluate the node computing power of each node in the distributed cluster, and divide the nodes into multiple subclusters according to the order of the node computing power.

6. The distributed cluster-based task allocation method according to claim 5, characterized in that: Also includes: If there are tasks to be assigned but no allocation relationship has been established, the number of tasks to be assigned that have not established an allocation relationship will be used as the new number of tasks. Starting from selecting low-load nodes from each sub-cluster of the distributed cluster according to the number of tasks of the nodes, the task allocation calculation will be performed again until the allocation relationship of all tasks to be assigned is determined.

7. The distributed cluster-based task allocation method according to claim 1, characterized in that: Also includes: During the task allocation process, the node load and task execution are continuously monitored. If the node load is close to the preset threshold or there is a shortage of resources, the allocation strategy is adjusted or node resources are increased in a timely manner. After allocating the tasks to be allocated according to the allocation relationship, the allocation results are counted and verified.

8. A task allocation device based on distributed cluster, characterized in that: The device includes: A task acquisition module is used to acquire tasks to be assigned and determine the priorities among the tasks to be assigned; The task node selection module to be assigned is used to select the task node to be assigned from the low-load nodes according to the processing capacity and type of the task of the low-load nodes; The load estimation module is used to estimate whether the load of the task node to be assigned after the task is assigned exceeds a preset load threshold according to the number and priority of the task to be assigned and the processing capacity and type of the task of the task node to be assigned; An allocation relationship establishment module, used to establish an allocation relationship between the tasks to be allocated and the task nodes to be allocated if the load does not exceed a preset load threshold; The task allocation module is used to allocate the tasks to be allocated according to the allocation relationship.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the distributed cluster-based task allocation method as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: Computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by the processor, the distributed cluster-based task allocation method according to any one of claims 1 to 7 is implemented.

Images