Task processing method, device and storage medium

Abstract

The present application provides a task processing method, device and storage medium, which are applied to the field of big data, including: when the queuing time of the target task in the queue to be scheduled is greater than a threshold, N target computing nodes are determined from multiple computing nodes; when the target task is scheduled, the N target computing nodes are used to execute the target task. The method of the present application determines whether the target task needs to be scheduled first by judging the queuing time of the target task. When the target task needs to be scheduled, the target computing node is selected and the target task is executed based on the target computing node, thereby reducing the "queue jumping" of other tasks and also reducing the queuing time of the target task.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a task processing method, device and storage medium. Background Art

[0002] With the development of big data technology, the scale of neural network parameters is getting larger and larger, and the cycle of deep learning training is getting longer and longer. When performing training tasks, multiple graphics processing units (GPUs) can be used to accelerate training to meet the training needs of big data and large models.

[0003] Currently, Kubernetes (K8s) clusters can implement GPU card accelerated training. For example, GPU servers can be deployed on a large scale in K8s clusters to perform deep learning tasks. However, when using K8s clusters for task training, some training task anomalies often occur. Summary of the invention

[0004] The present application provides a task processing method, device and storage medium, which are used to solve the problem of long queuing time when multi-machine and multi-card training tasks are to be scheduled.

[0005] In a first aspect, the present application provides a task processing method, which is applied to a computing cluster, where the computing cluster includes multiple computing nodes; the task processing method includes:

[0006] When the queuing time of the target task in the to-be-scheduled queue is greater than a threshold, N target computing nodes are determined from multiple computing nodes; the target task is a task that requires computing power of multiple computing nodes when executed, and N is an integer greater than or equal to 2;

[0007] When the target task is scheduled, N target computing nodes are used to execute the target task.

[0008] Optionally, the resource manifest file of the target task includes tolerance parameters and also includes:

[0009] Add taint marks to N target computing nodes; the taint marks are related to the tolerance parameters.

[0010] Optionally, when the target task is scheduled, N target computing nodes are used to execute the target task, including:

[0011] When the target task is scheduled, find the N target computing nodes corresponding to the taint marks that match the tolerance parameters;

[0012] Utilize N target computing nodes to execute the target task.

[0013] Optionally, the task processing method further includes:

[0014] When the target tasks are scheduled, the taint marks of the N target computing nodes are deleted.

[0015] Optionally, both tolerance parameters and taint tags include a key name, a key value, and an effect.

[0016] Optionally, the task processing method further includes:

[0017] Get the task with the earliest submission time and tolerance parameters in the queue to be scheduled, and get the target task.

[0018] Optionally, the resource list file of the target task includes a task scale; determining N target computing nodes from the multiple computing nodes includes:

[0019] Determine the number of cabinets M required according to the task scale; wherein the cabinet includes multiple computing nodes;

[0020] Calculate the sum of GPU resource allocation rates of the computing nodes of any cabinet in the computing cluster respectively;

[0021] Determine the cabinet with the smallest sum of M GPU resource allocation rates, and obtain N target computing nodes in the M cabinets.

[0022] Optionally, the task scale includes the number K of GPU cards required to execute the target task, and M is related to K, the number of computing nodes in the cabinet, and the number of GPU cards included in the computing nodes in the cabinet.

[0023] Optionally, the resource list file of the target task includes the task scale; determining N target computing nodes from the multiple computing nodes also includes:

[0024] Preempt N target computing nodes in multiple cabinets according to the task scale.

[0025] In a second aspect, the present application provides a task processing device, comprising:

[0026] The task scheduling module is used to determine N target computing nodes from multiple computing nodes when the queuing time of the target task in the queue to be scheduled is greater than a threshold; the target task is a task that requires the computing power of multiple computing nodes when executed, and N is an integer greater than or equal to 2.

[0027] The task execution module is used to execute the target task using N target computing nodes when the target task is scheduled.

[0028] In a third aspect, an embodiment of the present application provides a terminal device, including: a memory and a processor;

[0029] The memory is used to store computer instructions; the processor is used to execute the computer instructions stored in the memory to implement any method in the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement any one of the methods in the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any method in the first aspect when executed by a processor.

[0032] The task processing method, device and storage medium provided by the present application determine N target computing nodes from multiple computing nodes when the queuing time of the target task in the queue to be scheduled is greater than a threshold; when the target task is scheduled, the N target computing nodes are used to execute the target task. In this way, when the waiting time of the target task is too long, the central node can schedule the designated computing node to execute the target task, and the machine is dedicated to the specific purpose, and other tasks cannot occupy the computing power resources of the target computing node, thereby reducing the "queue jumping" of other tasks and reducing the waiting scheduling time of multi-machine and multi-card training tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a scenario of a task processing method provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of a task processing method provided in an embodiment of the present application;

[0035] Figure 3 A flowchart of a task processing method provided in an embodiment of the present application;

[0036] Figure 4 A flowchart of a task processing method provided in an embodiment of the present application;

[0037] Figure 5 A flowchart of a task processing method provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of the structure of a task processing device provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of a task processing terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0041] First, the terms involved in this application are explained:

[0042] Graphics processing unit (GPU): A processor that performs image processing tasks. The GPU can store images and process graphics, reducing the computing burden of the central processing unit (CPU) and improving the system's display function and display speed.

[0043] Container orchestration management system (Kubernetes, K8s): An open source application for managing containers on multiple hosts in a cloud platform. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0044] Pod: It is the basis of all business types in the K8s cluster system and the smallest unit that can be scheduled by K8s.

[0045] Compute Node: A working host in a K8s cluster system. It can be a physical node or a virtual machine. Pods need to run on specific Nodes. A Node can include multiple GPU cards.

[0046] Rack: refers to the cabinet where the server is placed. Several servers can be placed in a rack. A switch can be installed on the top of the rack to realize the network interconnection between the servers inside the rack. A rack can include multiple nodes.

[0047] Taint: The K8s cluster system can set taints on computing nodes, making it impossible for Pods to be scheduled to the computing nodes.

[0048] Toleration: Pods with tolerance can be scheduled to compute nodes with taints, achieving the purpose of dedicated machines for dedicated use.

[0049] The task processing method provided in the embodiment of the present application can be applied to Figure 1 The K8s cluster system architecture diagram is shown in the figure. Figure 1As shown, the computing cluster 10 of the K8s cluster system includes: a central node 101 and multiple cabinets 102. The central node 101 is used to obtain training tasks and schedule and allocate computing nodes according to the computing power resource requirements of the training tasks. The cabinet 102 is used to manage multiple computing nodes 103 in the cabinet, wherein multiple computing nodes 103 in the same cabinet 102 can communicate quickly using switches and other methods. The computing node 103 can be used to execute the training tasks scheduled by the central node 101. The central node can be a service plug-in deployed to a server or other device through software coding, or it can be a host, a physical server, a virtual machine, and other forms of equipment that can execute training tasks. The computing node 103 can be a host, a physical server, a virtual machine, and other forms of equipment that can execute training tasks.

[0050] In the big data scenario where K8s computing clusters deploy GPU servers on a large scale to perform deep learning tasks, multi-GPU card accelerated training has become the current mainstream deep learning training method. The system's GPU computing resources are limited. When there are many training tasks, the K8s central node can prioritize the scheduling of training tasks with low computing resource requirements. However, multi-machine and multi-card training tasks have a large demand for computing resources. It is possible that the multi-machine and multi-card training tasks submitted first are always "queued" by small tasks submitted later, causing the multi-machine and multi-card training tasks to be "starved" in a waiting state.

[0051] In view of this, the task processing method provided by the embodiment of the present application sets tolerance parameters for the training tasks by the central node when the waiting time of the training tasks to be scheduled is too long. The central node determines the cabinet to be scheduled according to the number of GPU cards required for the training task, and sets a taint mark on the computing nodes in the cabinet; the computing nodes with taint marks can execute the training tasks with set tolerance parameters, and refuse to execute the training tasks without set tolerance parameters. This method avoids the "queue jumping" of non-tolerant training tasks by setting tolerance and taint mechanisms, thereby reducing the waiting scheduling time of multi-machine and multi-card tasks.

[0052] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0053] Figure 2 The following is a flow chart of the task processing method provided by the embodiment of the present application. Figure 2 As shown:

[0054] S201. When the queuing time of the target task in the queue to be scheduled is greater than a threshold, N target computing nodes are determined from multiple computing nodes; the target task is a task that requires computing power of multiple computing nodes when executed, and N is an integer greater than or equal to 2.

[0055] The target task is a pending training task that requires the computing power of multiple computing nodes when executed. The target task can be a multi-machine multi-card training task. The threshold is the upper limit of the time for the target task to wait for scheduling, which can be customized. The computing cluster includes multiple computing nodes. The computing cluster can receive training tasks. When executing training tasks, the computing cluster allows the execution of multiple training tasks within the computing power resource range. The remaining training tasks enter the queue to be scheduled according to the task submission time, waiting to be scheduled by the central node.

[0056] For multi-machine and multi-card training tasks that require large computing resources, when the task to be scheduled in the queue is the target task, some computing nodes in the computing cluster may be occupied by other training tasks that have not been completed, resulting in the computing resources of the remaining computing nodes being unable to meet the target task. However, some training tasks that are queued after the target task require less computing resources. The central node can give priority to scheduling training tasks that can be met by the computing resources of the remaining computing nodes. The target task is "cut in line" by the training task with less computing resource requirements, which prolongs the waiting time for the target task to be scheduled.

[0057] If the target task waits too long in the queue, a small task may jump in line. When the target task's waiting time in the queue exceeds the upper limit of the waiting time, the central node selects N target computing nodes from the computing nodes as the "dedicated machine" to execute the target task.

[0058] S202: When the target task is scheduled, use N target computing nodes to execute the target task.

[0059] When the target task is scheduled, the central node can schedule N target computing nodes to train the target task, achieving “dedicated machine for dedicated purpose”.

[0060] The task processing method provided in the embodiment of the present application determines N target computing nodes from multiple computing nodes when the queuing time of the target task in the queue to be scheduled is greater than a threshold; when the target task is scheduled, the N target computing nodes are used to execute the target task. In this way, when the waiting time of the target task is too long, the central node can schedule the designated computing node to execute the target task, and the machine is dedicated to the specific purpose, and other tasks cannot occupy the computing power resources of the target computing node, thereby reducing the "queue jumping" of other tasks and reducing the waiting scheduling time of multi-machine and multi-card training tasks.

[0061] Combine the following Figure 3The task processing method provided in the embodiment of the present application is described. Figure 3 FIG. 1 is a flow chart of a task processing method provided by an embodiment of the present application. Figure 3 As shown:

[0062] S301. Obtain a resource list file for a training task.

[0063] The resource manifest file is used to record the orchestration and deployment of resource management and resource objects in the K8s cluster. For example, the resource manifest file can be a YAML (yet another markup language) file.

[0064] The central node may obtain the resource inventory file based on itself, the central node may also obtain the resource inventory file from other computing nodes in the computing cluster, or the central node may also obtain the resource inventory file from a third-party data source outside the computing cluster.

[0065] For example, a user can submit a YAML file on the user interface of a terminal device or edit a YAML file online on the user interface. The central node can receive the YAML file uploaded by the terminal device. The terminal device can be a central node, a computing node, and a third-party data source device.

[0066] For another example, the resource list file can be stored in the local storage of the terminal device. When the central node receives a task instruction to execute multiple machines and multiple cards, the central node obtains the resource list file from the local storage of the corresponding terminal device according to the task instruction. This application does not limit the method of obtaining the training task resource list file.

[0067] S302: Add a tolerance parameter toleration in the resource list file of the training task.

[0068] The tolerance parameters include key name key, key value value, effect effect and operator operator. When the training task is submitted to the K8s cluster, the resource list file of the training task includes the task scale, which is used to indicate the computing resources required to execute the training task. When the central node obtains the task scale of the training task, it determines whether the training task is a multi-machine multi-card training task based on the task scale. For example, the task scale can be the number of GPU cards required to execute the training task. If it is a multi-machine multi-card training task, the central node adds the tolerance parameters to the resource list file of the multi-machine multi-card training task. If not, no processing is performed.

[0069] The central node can define the task scale of multiple machines and multiple cards as a key, and the key can be used as a label for the taint mark and tolerance parameter. The task scale set by the key in the tolerance parameter can be consistent with the task scale in the taint mark.

[0070] The central node can set the operator to Equal or Exists. When the operator is Equal, the central node can set the value; when the operator is Exists, the central node can ignore the value.

[0071] The central node can define value as big as a label for the taint marker and tolerance parameter.

[0072] The effect can describe the role of tolerance parameters. Tolerance parameters can support multiple effects. Effects may include NoSchedule, PreferNoSchedule, and NoExecute. Among them, NoSchedule can instruct the central node not to schedule training tasks that do not add tolerance parameters to computing nodes with taint marks; PreferNoSchedule can instruct the central node to avoid scheduling training tasks that do not add tolerance parameters to computing nodes with taint marks as much as possible; NoExecute can instruct the central node not to schedule training tasks that do not add tolerance parameters to computing nodes with taint marks, and terminate the execution of existing training tasks in computing nodes with taint marks. This application does not limit the type of effect, and different effects can be selected to mark multi-machine and multi-card training tasks to achieve dedicated use of multiple machines and multi-card training tasks, or resource priority preemption.

[0073] The central node can set tolerance parameters based on the taint mark in the computing cluster. The taint mark can make the computing node exclude pods with a certain type of characteristics. When a computing node is tainted, normal training tasks cannot use the computing node, but training tasks that can tolerate such characteristics can use the computing node.

[0074] The above embodiments exemplarily illustrate the task processing method provided in the embodiments of the present application. It can be understood that other related or adaptively modified schemes that can implement the task processing method of the embodiments of the present application are within the protection scope of the embodiments of the present application.

[0075] S303: Submit the multi-machine multi-card training task to the queue to be scheduled.

[0076] The queue to be scheduled may include multiple multi-machine multi-card training tasks. After the central node sets tolerance parameters for the multi-machine multi-card training tasks, it submits the multi-machine multi-card training tasks with tolerance parameters to the queue to be scheduled, waiting for scheduling.

[0077] It is understandable that the queue to be scheduled also includes other training tasks, among which the multi-machine multi-card training tasks and other tasks can be arranged according to the submission time and wait for scheduling in the queue to be scheduled.

[0078] The task processing method provided in the embodiment of the present application distinguishes the multi-machine multi-card training task from other training tasks by adding tolerance parameters related to the stain mark in the computing cluster to the multi-machine multi-card training task. The central node submits the multi-machine multi-card training task to the queue to be scheduled. When the queuing time of the multi-machine multi-card training task in the queue to be scheduled is greater than the threshold, the central node determines N target computing nodes from multiple computing nodes; and when the multi-machine multi-card training task is scheduled, the N target computing nodes are used to execute the task. In this way, when the waiting time of the multi-machine multi-card training task is too long, the central node can schedule the designated computing node to execute the multi-machine multi-card training task, and the dedicated machine is dedicated, and other tasks cannot occupy the computing power resources of the target computing node, thereby reducing the "queue jumping" of other tasks and reducing the waiting scheduling time of the multi-machine multi-card training task.

[0079] Combine the following Figure 4 The method of executing multi-machine and multi-card training tasks with tolerance parameters on N computing nodes is further explained. Figure 4 FIG. 1 is a flow chart of a task processing method provided by an embodiment of the present application. Figure 4 As shown:

[0080] S401, obtaining the task with the earliest submission time and tolerance parameters in the queue to be scheduled, and obtaining the target task.

[0081] The central node obtains the multi-machine and multi-card training tasks with tolerance parameters in the queue to be scheduled, and takes the multi-machine and multi-card training tasks with the earliest submission time as the target tasks.

[0082] S402: Monitor the queuing time of the target task in the queue to be scheduled.

[0083] The central node may monitor the waiting scheduling time of the target task in the queue to be scheduled. For example, the queuing time may be the time interval between the monitoring time of the central node and the submission time of the target task. It is understandable that in the queue to be scheduled, other training tasks to be executed or being executed may be arranged before the target task. These other training tasks may affect the queuing time of the target task.

[0084] S403: If the queuing time is greater than the threshold, the required number of cabinets is determined according to the task scale of the target task.

[0085] The threshold can be used as a criterion for determining whether to reserve resources for the target task. The central node can set the threshold through a configuration file, and the threshold can be customized. For example, the threshold can be set according to the execution time of the target task that the user wants, the threshold can be set according to the estimated execution time of the training task before the target task in the queue, and the threshold can also be set according to other principles, which is not restricted in this application. Optionally, the central node can monitor the executing training tasks in real time. If the central node does not detect that the target task is scheduled within the threshold time, it determines that the queuing time of the target task exceeds the threshold. The central node can allocate a "dedicated plane" for the target task.

[0086] The computing cluster includes multiple cabinets, each of which includes multiple computing nodes. The computing nodes in any cabinet can communicate with each other based on a switch. When the central node is scheduling a target task, it determines the number of cabinets M required based on the task scale of the target task. It can be understood that a cabinet can be a connection relationship between virtual computing nodes or a physical space where computing nodes are placed.

[0087] For example, the task scale may be the number K of GPU cards required to execute the target task, where M is related to K, the number a of GPU cards included in any computing node, and the number b of computing nodes included in any cabinet. The specific relationship is as follows:

[0088] The number of cabinets M=K / a / b; when M is not an integer, M may be rounded up.

[0089] For example: If the target task requires 32 GPU cards, any cabinet includes 2 computing nodes, and any computing node is configured with 8 GPU cards, then M=32 / 8 / 2=2, and the computing power resources of 2 cabinets are required to execute the target task.

[0090] For another example, if the target task requires 34 GPU cards, any cabinet includes 2 computing nodes, and any computing node is configured with 8 GPU cards, then M = 34 / 8 / 2 = 2.125. If M is not an integer, M is rounded up, and the computing power resources of 3 cabinets are required to execute the target task.

[0091] Optionally, if the queuing time does not exceed the threshold, the target task continues to be queued. The central node can continue to monitor whether the target task is scheduled. Until the target task is scheduled, the central node determines the required number of cabinets according to the task scale of the target task.

[0092] S404: Calculate the sum of GPU resource allocation rates of computing nodes of any cabinet in the plurality of cabinets of the computing cluster.

[0093] The GPU resource allocation rate can indicate the occupancy of the GPU card. The central node can obtain the GPU resource allocation rate of each computing node, and add up the GPU allocation rates of the computing nodes belonging to the same cabinet to obtain the sum of the GPU resource allocation rates of the computing nodes in the cabinet. Among them, the higher the sum of the GPU resource allocation rates, the less computing power resources the central node can use to schedule the target task. It is understandable that the central node can sort the GPU resource allocation rates of the cabinets.

[0094] S405 . Determine the cabinet with the smallest sum of M GPU resource allocation rates, and obtain N target computing nodes in the M cabinets.

[0095] The central node selects M cabinets with the smallest values ​​from the sum of the GPU resource allocation rates of the sorted cabinets to obtain N target computing nodes in the M cabinets. M is related to N, for example, M = N / the number of computing nodes included in any cabinet, b. If N / b is not an integer, N / b is rounded up.

[0096] S406: Add taint marks to N target computing nodes.

[0097] The M cabinets can be regarded as "victims". After the central node selects the M cabinets, it adds taint marks to the N target computing nodes among the "victims".

[0098] The central node can define nodeX as the name of all computing nodes in the "victim". The central node adds taint marks to N target computing nodes. For example, X can be 1-N. The process of adding taint marks is as follows:

[0099] The central node adds a taint mark to node1.

[0100] The central node adds a taint mark to node2.

[0101] …

[0102] The central node adds a taint mark to nodeN-1.

[0103] The central node adds a taint mark to nodeN.

[0104] It is understandable that a taint mechanism can be set in the computing cluster, and the central node can add taint marks to N target computing nodes in turn according to the taint mechanism. When the central node adds a taint mark to the target computing node, the central node can set the taint mark corresponding to the tolerance parameter based on the taint mechanism, such as: task-scale, big, and NoSchedule. When the task scale of the training task is a multi-machine multi-card training task, the central node adds tolerance parameters for the multi-machine multi-card training task. Among them, the setting of the tolerance parameters can refer to the taint mechanism.

[0105] For example, the taint marks of nodeX in the taint mechanism include task-scale, big, and NoSchedule, which can correspond to the key, value, and effect of the tolerance parameter, indicating that nodeX can execute the training task with the tolerance parameter set and refuse to execute other training tasks. This application can also use other parameters as taint marks and tolerance parameters, and this application does not limit this.

[0106] It should be noted that the central node in the embodiment of the present application can add taint marks to the computing nodes in the cabinet. There is a network topology relationship between the computing nodes. In a scenario where multi-machine communication is required, the task processing method provided in the embodiment of the present application can automatically select the server with the least network congestion for scheduling. Exemplarily, the computing nodes in the same cabinet can communicate based on a switch. When the computing nodes in different cabinets communicate with each other, the information is transmitted to their respective switches respectively, and the communication is completed by multiple switches. Optionally, a third-party communication device can also be added to communicate between multiple switches.

[0107] The embodiment of the present application can be set to preferentially use the computing nodes of the same cabinet to perform the target task, so as to reduce the network interconnection between switches, thereby speeding up the training efficiency of multi-machine multi-card training tasks. It is understandable that when the GPU computing power resources in the computing cluster are insufficient, the embodiment of the present application can also use computing nodes of different cabinets to perform multi-machine multi-card training tasks, and the present application does not limit this.

[0108] For example, cabinet 1 includes computing nodes A and B, and cabinet 2 includes computing nodes C and D. A and B can communicate with each other based on switch 1, and C and D can communicate with each other based on switch 2. When A and C communicate with each other, the information can be transmitted via A-switch 1-switch 2-C, and the time consumed exceeds the time consumed between A and B. Therefore, illustratively, when the target task requires the computing power resources of two computing nodes, the central node can give priority to scheduling computing nodes A and B in cabinet 1, or the central node can give priority to scheduling computing nodes C and D in cabinet 2. In this way, the scheduling time and execution time of multi-machine and multi-card training tasks can be improved. It can be understood that in the above situation, if the computing power resources of A and C are occupied, the embodiment of the present application can also use B and D in the two cabinets to perform multi-machine and multi-card training tasks. This application does not limit this.

[0109] The task processing method provided in the embodiment of the present application sets a stain mark on the cabinet and adds a tolerance parameter to the multi-machine multi-card training task, and selects the target computing node through a specific strategy. In this way, the "queue jumping" of tasks with smaller computing power resource requirements is avoided, thereby reducing the waiting scheduling time of multi-machine multi-card tasks.

[0110] In the task processing method provided in the embodiment of the present application, a computing node with a taint mark can execute a target task with a tolerance parameter. When the target task is scheduled, if the computing node is set with a taint mark, it may cause other subsequent tasks to be unable to use the computing node. Figure 5 The method for deleting stain marks provided in an embodiment of the present application is described.

[0111] S501: Check whether the target task has been scheduled.

[0112] The central node may poll whether the target task has been scheduled. When the central node detects that the target task has been scheduled, step S502 may be executed.

[0113] Optionally, if the target task queried by the central node is to be scheduled or has not been fully scheduled, the central node waits for the target computing node to execute the target task and enters the next round of query.

[0114] S502: When the target task is scheduled, delete the taint marks of the N target computing nodes.

[0115] After the target task is scheduled, the central node will delete the "victim" identity of the cabinet, and all N target computing nodes need to delete the taint mark.

[0116] The central node can define nodeX as the name of all computing nodes in the "victim". The central node deletes the taint marks of N target computing nodes in turn. For example, X can be 1-N. Deleting taint marks is similar to adding taint marks. The exemplary process of deleting taint marks is as follows:

[0117] The central node deletes the taint mark of node1. The central node may add a minus sign "-" to the end of the taint mark of the computing node to make the taint mark invalid.

[0118] The central node deletes the taint mark of node2.

[0119] …

[0120] The central node deletes the taint mark of nodeN-1.

[0121] The central node deletes the taint mark of nodeN.

[0122] The order of adding the taint marks can be consistent with the order of deleting the taint marks. After the above instructions, the identities of all "victims" are cancelled and the computing nodes can return to normal. Exemplarily, the computing nodes that have deleted the taint marks can re-execute other training tasks.

[0123] The task processing method provided by the embodiment of the present application is to poll whether the target task has been scheduled. If so, the stain marks of N target computing nodes are deleted. In this way, the scheduling of the target task is monitored in real time by the central node, and the stain mark of the target computing node is deleted when the target task is scheduled, which reduces the situation where other subsequent tasks cannot schedule the target computing node, and improves the accuracy of the task processing method of the present application.

[0124] It is understandable that the task processing method provided in the embodiment of the present application adds a stain mark to the "victim" and adds a tolerance parameter to the multi-machine multi-card training task. Through this processing method, the original running task on the target computing node in the "victim" will not be preempted. When the original task of the target computing node in the original cabinet is completed, the multi-machine multi-card training task will be scheduled to the "victim", thereby retaining the integrity of the original task.

[0125] Optionally, the embodiment of the present application may also provide a multi-machine multi-card preemption mechanism, wherein N target computing nodes are determined from the multiple computing nodes, and further comprising:

[0126] Preempt N target computing nodes in multiple cabinets according to the task scale.

[0127] When the central node queries the GPU resource allocation rate and finds that it does not meet the computing power resources of the target task, the central node can preempt N target computing nodes in multiple cabinets according to the task scale.

[0128] Exemplarily, the central node can realize the preemption of computing resources by modifying the stain mark and tolerance parameters. For example, the central node can evict the training task occupying the target computing node based on the modified effect to realize the preemption of computing resources. The central node can also realize the preemption of computing resources by adding a preemption mechanism in the service plug-in. This application does not limit this.

[0129] The task processing method provided in the embodiment of the present application sets a preemption mechanism. When available computing resources are insufficient, the central node gives priority to scheduling multi-machine and multi-card training tasks, thereby reducing the waiting scheduling time for multi-machine and multi-card training tasks.

[0130] Combined with the above Figure 1-Figure 5 , the method provided in the embodiment of the present application is explained, and the related device for executing the above method provided in the embodiment of the present application is described below.

[0131] Figure 6 The structure diagram of the task processing device provided in the embodiment of the present application is shown as follows: Figure 6 As shown:

[0132] The task processing device 60 provided in the embodiment of the present application is applied to a computing cluster, the computing cluster includes multiple computing nodes, and the task processing device includes:

[0133] The task scheduling module 601 is used to determine N target computing nodes from multiple computing nodes when the queuing time of the target task in the queue to be scheduled is greater than a threshold; the target task is a task that requires the computing power of multiple computing nodes when executed, and N is an integer greater than or equal to 2.

[0134] The task execution module 602 is used to execute the target task using N target computing nodes when the target task is scheduled.

[0135] Optionally, the task scheduling module 601 further includes:

[0136] The tolerance setting module is used to add tolerance parameters to the resource manifest file of the target task.

[0137] The taint marking module is used to add taint marks to N target computing nodes; the taint marks are related to the tolerance parameters.

[0138] Optionally, the task execution module is used to execute the target task using N target computing nodes when the target task is scheduled, specifically including:

[0139] When the target task is scheduled, the task execution module searches for N target computing nodes corresponding to the taint marks that match the tolerance parameters; and uses the N target computing nodes to execute the target task.

[0140] Optionally, the task processing device further includes:

[0141] The taint marking module is also used to delete the taint marks of N target computing nodes when the target task is scheduled.

[0142] Optionally, both tolerance parameters and taint tags include a key name, a key value, and an effect.

[0143] Optionally, the task scheduling module 601 further includes: a timing module for acquiring the task with the earliest submission time and tolerance parameters in the queue to be scheduled, and obtaining the target task.

[0144] Optionally, the resource list file of the target task includes the task scale; the task scheduling module 601 also includes a calculation module, which is specifically used for:

[0145] The computing module is used to determine the required number of cabinets M according to the task scale; wherein the cabinet includes multiple computing nodes.

[0146] The calculation module is also used to respectively calculate the sum of the GPU resource allocation rates of the computing nodes of any cabinet in the computing cluster.

[0147] The computing module is also used to determine the cabinet with the smallest sum of M GPU resource allocation rates, and obtain N target computing nodes in the M cabinets.

[0148] Optionally, the task scale includes the number K of GPU cards required to execute the target task, and M is related to K, the number of computing nodes in the cabinet, and the number of GPU cards included in the computing nodes in the cabinet.

[0149] Optionally, the resource list file of the target task includes the task scale; the task scheduling module 601 further includes:

[0150] The resource preemption module is used to preempt N target computing nodes in multiple cabinets according to the task scale.

[0151] The task processing device of the embodiment of the present application can be used to execute the technical solution of the above-mentioned task processing method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0152] Figure 7 This is a schematic diagram of the structure of a task processing terminal device provided in an embodiment of the present application. Figure 7 As shown, the task processing terminal device 70 provided in this embodiment may include:

[0153] Processor 701; and memory 702, used to store executable instructions of the terminal device.

[0154] Among them, the processor is configured to execute the technical solution of the above-mentioned task processing method embodiment by executing executable instructions. Its implementation principle and technical effect are similar and will not be repeated here.

[0155] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the technical solution of the above-mentioned task processing method embodiment is implemented. The implementation principle and technical effect are similar and will not be repeated here.

[0156] A computer program product is also provided in an embodiment of the present application, including a computer program. When the computer program is executed by a processor, the technical solution of the above-mentioned task processing method embodiment is implemented. Its implementation principle and technical effect are similar and will not be repeated here.

[0157] In the specific implementation of the above-mentioned terminal device or server, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0158] Those skilled in the art will appreciate that all or part of the steps of any of the above method embodiments may be completed by hardware associated with program instructions. The aforementioned program may be stored in a computer-readable storage medium, and when the program is executed, all or part of the steps of the above method embodiments are executed.

[0159] If the technical solution of the present application is implemented in the form of software and sold or used as a product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a computer program or a number of instructions. The computer software product enables a computer device (which can be a personal computer, a server, a network device, or a similar electronic device) to perform all or part of the steps of the method of embodiment 1 of the present application. The aforementioned storage medium can be a U disk, a mobile hard disk, a ROM, a random access memory (RAM), a disk, or an optical disk, etc., which can store program codes.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A task processing method, characterized in that: Applied to a computing cluster, the computing cluster includes a plurality of computing nodes; the method includes: Obtain a resource list file for the training task, wherein the resource list file includes a task scale, and the task scale is used to indicate the computing power resources required to execute the training task; According to the task scale, determining the training task as a target task, wherein the target task is a task that requires computing power of multiple computing nodes when executed; Add tolerance parameters to the resource list file of the target task; Submit the target task to the queue to be scheduled; Monitoring the queuing time of the target task in the queue to be scheduled, and when the queuing time of the target task in the queue to be scheduled is greater than a threshold, determining N target computing nodes from the multiple computing nodes; wherein N is an integer greater than or equal to 2; When the target task is scheduled, the target task is executed using the N target computing nodes, where the N target computing nodes are computing nodes to which taint marks are added.

2. The method according to claim 1, characterized in that The method further comprises: Adding taint marks to the N target computing nodes; the taint marks are related to the tolerance parameters.

3. The method according to claim 2, characterized in that When the target task is scheduled, using the N target computing nodes to execute the target task includes: When the target task is scheduled, searching for N target computing nodes corresponding to the taint marks matching the tolerance parameter; The target task is executed using the N target computing nodes.

4. The method according to claim 3, characterized in that: The method further comprises: When the target task is scheduled, the taint marks of the N target computing nodes are deleted.

5. The method according to any one of claims 2 to 4, characterized in that: The tolerance parameter and the stain mark both include a key name key, a key value value and an effect effect.

6. The method according to claim 5, characterized in that The method further comprises: The task with the earliest submission time and tolerance parameters in the queue to be scheduled is obtained to obtain the target task.

7. The method according to any one of claims 1 to 4, characterized in that: Determining N target computing nodes from the plurality of computing nodes includes: Determine the required number of cabinets M according to the task scale; wherein the cabinet includes multiple computing nodes; Calculate the sum of the GPU resource allocation rates of the computing nodes of any cabinet in the computing cluster respectively; Determine the cabinet with the smallest sum of the GPU resource allocation rates, and obtain N target computing nodes in the M cabinets.

8. The method according to claim 7, characterized in that The task scale includes the number K of GPU cards required to execute the target task, and M is related to K, the number of computing nodes in the cabinet, and the number of GPU cards included in the computing nodes in the cabinet.

9. The method according to any one of claims 1 to 4, characterized in that: Determining N target computing nodes from the plurality of computing nodes further includes: N target computing nodes are seized in the multiple cabinets according to the task scale.

10. A task processing device, characterized in that: Applied to a computing cluster, the computing cluster includes a plurality of computing nodes; the device includes: A task scheduling module is used to obtain a resource list file of a training task, wherein the resource list file includes a task scale, and the task scale is used to indicate the computing resources required to execute the training task; According to the task scale, determining the training task as a target task, wherein the target task is a task that requires computing power of multiple computing nodes when executed; Add tolerance parameters to the resource list file of the target task; Submit the target task to the queue to be scheduled; Monitoring the queuing time of the target task in the queue to be scheduled, and when the queuing time of the target task in the queue to be scheduled is greater than a threshold, determining N target computing nodes from the multiple computing nodes; wherein N is an integer greater than or equal to 2; The task execution module is used to execute the target task using the N target computing nodes when the target task is scheduled, and the N target computing nodes are computing nodes with taint marks added.

11. A terminal device, characterized in that: include: Memory and processor; The memory is used to store computer instructions; the processor is used to execute the computer instructions stored in the memory to implement the method described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 9.

Images