Resource scheduling methods, devices, electronic equipment and storage media

Abstract

This application provides a resource scheduling method, apparatus, electronic device, and storage medium. The method includes: determining the target number of cores required by a queued task waiting to be computed by a second hardware resource platform; obtaining the number of idle residual cores of each first hardware processing node in a first hardware resource platform with idle residual cores; matching the queued task with each of the first hardware processing nodes based on the target number of cores and the number of residual cores to obtain a hardware resource matching result; and pushing the queued task to the first hardware resource platform based on the hardware resource matching result, instructing the first hardware processing node with a residual core count greater than or equal to the target number of cores to compute the queued task. Embodiments of this application can improve the overall resource utilization and overall computing efficiency of the hardware resource platform.

Description

Technical Field

[0001] This application relates to the field of resource management, specifically to a resource scheduling method, apparatus, electronic device, and storage medium. Background Technology

[0002] In many fields, data processing tasks involving computation require the allocation of certain hardware resources to provide necessary support. These hardware resources typically refer to the hardware cores of a hardware resource platform; for example, a CPU platform allocates its CPU cores as hardware resources to tasks to support computation. In current technologies, the scheduling and allocation of hardware cores within a hardware resource platform often results in some cores being wasted, leading to low resource utilization and computational efficiency. Summary of the Invention

[0003] One objective of this application is to provide a resource scheduling method, apparatus, electronic device, and storage medium that can improve the overall resource utilization and overall computing efficiency of a hardware resource platform.

[0004] According to one aspect of the embodiments of this application, a resource scheduling method is disclosed, the method comprising:

[0005] For queued tasks waiting to be computed by the second hardware resource platform, determine the target number of cores required by the queued tasks;

[0006] For a first hardware resource platform with idle and residual cores, obtain the number of idle and residual cores of each first hardware processing node in the first hardware resource platform.

[0007] Based on the target number of cores and the remaining number of cores, the queued tasks are matched with each of the first hardware processing nodes to obtain hardware resource matching results.

[0008] Based on the hardware resource matching result, the queued task is pushed to the first hardware resource platform, instructing the first hardware processing node, whose remaining core count is greater than or equal to the target core count, to calculate the queued task.

[0009] According to one aspect of the embodiments of this application, a resource scheduling apparatus is disclosed, the apparatus comprising:

[0010] The target core determination module is configured to determine the number of target cores required by a queued task waiting for computation by a second hardware resource platform.

[0011] The residual core determination module is configured to, for a first hardware resource platform with idle residual cores, obtain the number of idle residual cores of each first hardware processing node in the first hardware resource platform.

[0012] The hardware resource matching module is configured to match the queued tasks with each of the first hardware processing nodes based on the target number of cores and the remaining number of cores, so as to obtain the hardware resource matching result.

[0013] The queued task push module is configured to push the queued task to the first hardware resource platform based on the hardware resource matching result, and instruct the first hardware processing node whose remaining core count is greater than or equal to the target core count to calculate the queued task.

[0014] In one exemplary embodiment of this application, the device is configured as follows:

[0015] Determine the software certificate required for the queued task;

[0016] When an idle software certificate is detected on the software resource platform, and a first hardware processing node with a residual core count greater than or equal to the target core count is detected based on the hardware resource matching result, the software certificate is assigned to the queued task, and the queued task is pushed to the first hardware resource platform.

[0017] In one exemplary embodiment of this application, the device is configured as follows:

[0018] After the queuing task is completed, the software certificate in the software resource platform is released, and the remaining cores in the first hardware resource platform occupied by the queuing task are released.

[0019] In one exemplary embodiment of this application, the device is configured as follows:

[0020] The queue members whose tags are bound to the second hardware resource platform are filtered out to obtain the queued tasks;

[0021] When the hardware resource matching result determines that the first hardware resource platform has hardware processing nodes with a residual core count greater than or equal to the target core count, the tag of the queued task is modified to be bound to the first hardware resource platform, and the queued task is pushed to the first hardware resource platform.

[0022] In one exemplary embodiment of this application, the device is configured as follows:

[0023] When it is determined, based on the hardware resource matching result, that there are hardware processing nodes in the first hardware resource platform with a residual core count greater than or equal to the target core count, the queued task is bound to the tag of the target hardware processing node in the first hardware resource platform used to calculate the queued task.

[0024] After the queuing task calculation is completed, the remaining cores in the target hardware processing node occupied by the queuing task are released, and the queuing task is unbound from the tag of the target hardware processing node.

[0025] In one exemplary embodiment of this application, the device is configured as follows:

[0026] Select the first hardware processing node whose number of residual cores is greater than or equal to the target number of cores;

[0027] The target hardware processing node with the smallest number of residual cores among the first selected hardware processing nodes is identified, and the target hardware processing node is matched and bound with the queued task to obtain the hardware resource matching result.

[0028] In one exemplary embodiment of this application, the device is configured as follows:

[0029] For at least two queued tasks waiting to be computed by the second hardware resource platform, determine the target number of cores required by each of the at least two queued tasks.

[0030] The target core counts required by the at least two queued tasks are respectively taken as indivisible numbers to fill the remaining core counts of each first hardware processing node, and the total number of remaining cores of the first hardware resource platform after filling is calculated.

[0031] The hardware resource matching results are obtained by matching the at least two queued tasks with each of the first hardware processing nodes according to the filling method of the remaining cores with the smallest value.

[0032] According to one aspect of the embodiments of this application, an electronic device is disclosed, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the methods provided in the various optional implementations described above.

[0033] According to one aspect of the embodiments of this application, a computer program medium is disclosed, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods provided in the various optional implementations described above.

[0034] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0035] In this embodiment, for queued tasks waiting to be computed by the second hardware resource platform, the queued task is matched with each of the first hardware processing nodes of the first hardware resource platform based on the target number of cores required by the queued task and the number of idle residual cores on each of the first hardware processing nodes of the first hardware resource platform. Based on the matching result, the queued task can be computed by a first hardware processing node with a residual core count greater than or equal to the target number of cores. This improves the overall resource utilization and computational efficiency of both the second and first hardware resource platforms by reusing the residual cores of the first hardware resource platform.

[0036] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0037] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0038] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0039] Figure 1 A schematic diagram of a basic resource scheduling strategy according to an embodiment of this application is shown.

[0040] Figure 2 A flowchart of a resource scheduling method according to an embodiment of this application is shown.

[0041] Figure 3 A schematic diagram of hardware resource distribution before resource scheduling is shown according to an embodiment of this application.

[0042] Figure 4 An embodiment according to this application is shown. Figure 3 A schematic diagram of hardware resource distribution after resource scheduling in the embodiment.

[0043] Figure 5 A flowchart illustrating the coupling between the resource scheduling method provided in this application and the basic resource scheduling strategy according to one embodiment of this application is shown.

[0044] Figure 6 A detailed flowchart of resource scheduling according to one embodiment of this application is shown.

[0045] Figure 7 A schematic diagram of the resource scheduling development logic architecture according to an embodiment of this application is shown.

[0046] Figure 8 A block diagram of a resource scheduling apparatus according to an embodiment of this application is shown.

[0047] Figure 9 A hardware diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0048] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0049] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0050] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0051] This application provides a resource scheduling method that can be used in the Portable Batch System (PBS) for high-performance computing (HPC) clusters. As a computing task manager for resource scheduling in HPC, PBS can schedule HPC computing resources using the resource scheduling method provided in this application, thereby improving the overall resource utilization and overall computing efficiency of HPC.

[0052] Figure 1 A schematic diagram of a basic resource scheduling strategy according to an embodiment of this application is shown.

[0053] refer to Figure 1 In this embodiment, after the resource scheduling system PBS receives the task Q to be calculated, it confirms the relevant information of the task Q: 48 cores required, and specified hardware resource platform P.

[0054] The system then checks whether the hardware resource platform P has any idle hardware processing nodes and whether the software resource platform has a license certificate for task Q. If both conditions are met—that is, both idle hardware processing nodes and a certificate are available—task Q is copied from storage to the hardware resource platform P and processed by the idle hardware processing nodes. The two idle hardware processing nodes used to process task Q each have 24 cores occupied and 4 remaining cores idle.

[0055] It should be noted that this embodiment is mainly used to demonstrate the basic strategy logic of resource scheduling. Therefore, both task Q and hardware resource platform P are abstract concepts, and their quantity and type should not be limited.

[0056] Figure 2 A flowchart of a resource scheduling method according to an embodiment of this application is shown. An exemplary execution subject of the method is a resource scheduling system PBS. The method includes:

[0057] Step S110: For queued tasks waiting for computation by the second hardware resource platform, determine the target number of cores required by the queued tasks.

[0058] Step S120: For the first hardware resource platform with idle and residual cores, obtain the number of idle residual cores of each first hardware processing node in the first hardware resource platform.

[0059] Step S130: Based on the target number of cores and the remaining number of cores, match the queued tasks with each first hardware processing node to obtain the hardware resource matching result.

[0060] Step S140: Based on the hardware resource matching results, push the queued tasks to the first hardware resource platform and instruct the first hardware processing node with a remaining core count greater than or equal to the target core count to calculate the queued tasks.

[0061] Specifically, in this embodiment, for queued task Q2 that should be calculated by the second hardware resource platform P2, but which is temporarily unable to be calculated immediately by P2, the required target core count n2 is obtained. The target core count of n2 indicates that calculating queued task Q2 requires n2 cores.

[0062] In this context, the core, as the unit of hardware resource usage within the hardware resource platform, resides in the hardware processing node that serves as the unit of hardware resource scheduling. Cores include, but are not limited to, CPU cores and GPU cores. When the hardware resource platform is a CPU platform built from CPU nodes, its hardware processing nodes are CPU nodes, and its core is a CPU core. When the hardware resource platform is a GPU platform built from GPU nodes, its hardware processing nodes are GPU nodes, and its core is a GPU core.

[0063] It should be noted that when a task is assigned to a hardware processing node for computation using the hardware processing node as the hardware resource scheduling unit, and the hardware processing node's capacity to handle the task is confirmed using the core as the hardware resource occupancy unit, a hardware processing node in the hardware resource platform may still have some idle cores even when it is in computation state. In this embodiment, the idle cores of a hardware processing node in computation state are referred to as residual cores. For example, a hardware processing node C1 in the first hardware resource platform P1 has 28 cores. After assigning a task requiring 20 cores to C1 for processing, although C1 is occupied and in computation state, it still has 8 idle cores. That is, C1 has 8 residual cores.

[0064] Considering that residual cores are generally wasted, reusing them would make more efficient use of hardware resources. Therefore, in this embodiment, to improve the overall resource utilization and computational efficiency of P2 and P1, when a queued task Q2 waiting to be computed by P2 is detected, and idle residual cores are detected in P1, the target number of cores n2 required by the queued task Q2 is obtained, and the number of idle residual cores N1 (N11, N12, ..., N1m) of each first hardware processing node C1 (C11, C12, ..., C1m) in P1 is obtained. Then, based on the target number of cores n2 and the number of residual cores N1 (N11, N12, ..., N1m), the queued task Q2 is matched with each first hardware processing node C1 (C11, C12, ..., C1m) to obtain a hardware resource matching result. Then, based on the hardware resource matching result, the queued task Q2 is pushed to P1, instructing C1 with a number of residual cores greater than or equal to the target number of cores n2 to compute the queued task Q2. Here, m is a positive integer.

[0065] For example, the target number of cores n2 required for queuing task Q2 is 6; the number of idle residual cores N1(N11,N12,N13,N14) of the four first hardware processing nodes C1 (C11,C12,C13,C14) in the first hardware resource platform Q1 is (4,6,8,5). Since the number of residual cores N12 of C12 is 6 and the number of residual cores N13 of C13 is 8, both are greater than or equal to n2, queuing task Q2 can be pushed to the first hardware resource platform P1, instructing the first hardware processing node C12 to calculate queuing task Q2, or instructing the first hardware processing node C13 to calculate queuing task Q2.

[0066] Figure 3 A schematic diagram of hardware resource distribution before resource scheduling is shown according to an embodiment of this application. Figure 4 It shows Figure 3 A schematic diagram of hardware resource distribution after resource scheduling in the embodiment.

[0067] refer to Figure 3 and Figure 4 As can be seen, before resource scheduling, some of the first hardware processing nodes in hardware resource platform P1 that are in computation have idle residual cores. Leaving these residual cores idle would result in a waste of overall resources. Therefore, when there are queued tasks waiting to be computed by hardware resource platform P2, these queued tasks are allocated to the first hardware processing nodes in P1 with idle residual cores, allowing these residual cores to be reused, thereby improving the overall resource utilization and overall computational efficiency of P1 and P2.

[0068] Therefore, in this embodiment, for queued tasks waiting to be computed by the second hardware resource platform, the queued task is matched with each of the first hardware processing nodes of the first hardware resource platform based on the target number of cores required by the queued task and the number of idle residual cores of each first hardware processing node of the first hardware resource platform. Based on the matching result, the queued task can be computed by a first hardware processing node with a residual core count greater than or equal to the target number of cores. This improves the overall resource utilization and computational efficiency of both the second and first hardware resource platforms by reusing the residual cores of the first hardware resource platform.

[0069] In one embodiment, the software certificate required by the queued task is determined. When an idle software certificate is detected on the software resource platform, and a first hardware processing node with a residual core count greater than or equal to the target core count is detected based on the hardware resource matching results, the software certificate is assigned to the queued task, and the queued task is pushed to the first hardware resource platform.

[0070] In this embodiment, in addition to determining the target core count n2 required by the queuing task Q2, the PBS also determines the software certificate L2 required by the queuing task Q2. The determined target core count n2 is used for hardware resource matching, and the determined software certificate L2 is used for software resource matching. When both hardware and software resources are successfully matched, software and hardware resources are then allocated to the queuing task Q2.

[0071] Specifically, PBS can first determine the software certificate L2 required by the queued task Q2. After detecting that there is an idle software certificate L2 on the software resource platform, it then performs hardware resource matching based on the target number of cores n2. Subsequently, after determining that the first hardware resource platform P1 has a first hardware processing node C1 with a residual number of cores N1 greater than or equal to the target number of cores n2 based on the hardware resource matching results, it assigns the software certificate L2 to the queued task Q2 and pushes the queued task Q2 to Q1.

[0072] PBS can also perform hardware resource matching based on the target number of cores n2 while detecting whether there is an idle software certificate L2 on the software resource platform. Then, when it detects that there is an idle software certificate L2 on the software resource platform, and determines, based on the hardware resource matching results, that the first hardware resource platform P1 has a first hardware processing node C1 with a residual number of cores N1 greater than or equal to the target number of cores n2, it assigns the software certificate L2 to the queued task Q2 and pushes the queued task Q2 to Q1.

[0073] Figure 5 A flowchart illustrating the coupling of the resource scheduling method provided in this application with the basic resource scheduling strategy according to an embodiment of this application is shown.

[0074] refer to Figure 5 In this embodiment, after receiving the task Q1 submitted by the user, PBS confirms the information of task Q1: the software name is S1, the specified CPU platform is P1, and the required number of cores is n1.

[0075] PBS searches the software resource platform to detect the certificate L1 of software S1, i.e., the software certificate L1 of task Q1; and searches P1 to detect the number of residual cores N1.

[0076] If there is no available software certificate L1, or the number of remaining cores N1 is less than the target number of cores n1, then task Q1 will be queued and calculated when P1 has enough remaining cores.

[0077] If an idle software certificate L1 exists, and the remaining core count N1 is greater than or equal to the target core count n1, then the task execution script is invoked, task Q1 is pushed to P1, and the remaining core count of P1 is updated to N1-n1. After the remaining core count of P1 is updated to (N1-n1), if there is a queued task Q2 waiting for CPU platform P2 to perform calculations, then the software certificate L2 and target core count n2 of the queued task Q2 can be confirmed. Therefore, if an idle software certificate L2 exists and (N1-n1) is greater than or equal to n2, the queued task Q2 is pushed to P1 for calculation.

[0078] In one embodiment, after the queued task is completed, the software certificate in the software resource platform is released, and the remaining cores occupied by the queued task in the first hardware resource platform are released.

[0079] In this embodiment, after the PBS assigns the software certificate L2 to the queued task Q2 and pushes the queued task Q2 to the first hardware resource platform P1, the first hardware processing node C1, whose remaining core number N1 is greater than or equal to the target core number n2, assigns n2 of the remaining cores to the queued task Q2 and calculates the queued task Q2 using the n2 remaining cores.

[0080] It can be seen that during the calculation of the queued task Q2, while occupying the software certificate L2 in the software resource platform, it also occupies n2 residual cores of C1, where the number of residual cores N1 is greater than or equal to the number of target cores n2.

[0081] Since the occupied hardware and software resources are temporarily unavailable for other tasks, to ensure efficient utilization of these resources, after the queued task Q2 completes its computation, the PBS will release the hardware and software resources occupied by queued task Q2. Specifically, software certificate L2 will be released, and the n² remaining cores in the first hardware resource platform P1 that were occupied by queued task Q2 will be released. The released hardware and software resources will then be restored to a usable, idle state.

[0082] In one embodiment, queue members whose tags are bound to the second hardware resource platform are filtered out to obtain queued tasks. When it is determined from the hardware resource matching results that the first hardware resource platform has hardware processing nodes with a residual core count greater than or equal to the target core count, the tag of the queued task is modified to bind to the first hardware resource platform, and the queued task is pushed to the first hardware resource platform.

[0083] In this embodiment, PBS manages the scheduling direction of tasks through tag binding.

[0084] Specifically, after receiving a task request for computation, PBS determines which hardware resource platform should perform the computation. If the task needs to be performed by the first hardware resource platform P1, the task's tag is bound to P1; if the task needs to be performed by the second hardware resource platform P2, the task's tag is bound to P2.

[0085] During the subsequent resource scheduling of tasks, PBS identifies the corresponding hardware resource platform through the task's tag and then allocates the task to the corresponding hardware resource platform for computation. If, during the task allocation process, it is determined that the corresponding hardware resource platform does not currently have sufficient hardware resources to compute the task, the task is pushed into a queue to wait.

[0086] PBS filters out queue members whose tags are bound to P2, obtaining queued task Q2 waiting for P2 to calculate. After confirming that queued task Q2 will be pushed to P1, PBS changes the tag of queued task Q2 to be bound to P1, and then pushes queued task Q2 to P1 again.

[0087] Figure 6 A detailed flowchart of resource scheduling according to an embodiment of this application is shown.

[0088] refer to Figure 6 In this embodiment, PBS confirms the number of remaining cores N1 of CPU platform P1, and confirms the software name S2 of task Q2 that should be calculated by CPU platform P2, and the required target number of cores n2.

[0089] If P1 has residual cores, i.e., the number of residual cores N1 is greater than 0, and task Q2 is in the queue, then the software resource platform is matched with task Q2 to confirm whether the software certificate L2 of task Q2 is idle, and the hardware resource platform is matched with task Q2 to confirm whether the number of residual cores N1 of P1 is greater than or equal to the target number of cores n2 of task Q2.

[0090] If the software certificate L2 of task Q2 is idle, and the number of residual cores N1 of P1 is greater than or equal to the target number of cores n2 of task Q2, then modify the label of task Q2 to bind it to P1, and then push task Q2 to P1 for calculation.

[0091] After the calculation is complete, PBS releases the software certificate L2 occupied by task Q2 and releases the n2 cores in P1 occupied by task Q2.

[0092] In one embodiment, when it is determined, based on the hardware resource matching results, that the first hardware resource platform has hardware processing nodes with a residual core count greater than or equal to the target core count, the queued task is bound to the tag of the target hardware processing node used to calculate the queued task in the first hardware resource platform. After the queued task calculation is completed, the residual cores occupied by the queued task in the target hardware processing node are released, and the tag of the queued task is unbound from the target hardware processing node.

[0093] In this embodiment, for the queued task Q2 originally bound to the second hardware resource platform P2, after PBS confirms that it will push it to the first hardware resource platform P1 for calculation, it will also bind the tag of the queued task Q2 to the target hardware processing node. The target hardware processing node is the first hardware processing node in P1 used to calculate the queued task Q2.

[0094] After the queued task Q2 completes its calculation, the remaining cores occupied by the queued task Q2 in the target hardware processing node are released, and the tag of the queued task Q2 is unbound from the target hardware processing node.

[0095] Figure 7 A schematic diagram of the resource scheduling development logic architecture according to an embodiment of this application is shown.

[0096] refer to Figure 7 In this embodiment, during the first phase of the resource scheduling development logic, four related databases are established: a hardware / software matching database (Hardware_software), a cluster node information database (Nodes_test), a task operation information database (Job_operation), and a software license information database (Job_lic_data).

[0097] Specifically, by analyzing the characteristics of various simulation software, the optimal CPU platform is matched, and a hardware / software matching database (Hardware_software) is established. Key fields include: CPU platform, number of cores, hard disk, memory, and applicable software.

[0098] Establish a cluster node information database, Nodes_test, to monitor the status of cluster nodes. Key fields include: node name, node status, number of physical cores, amount of physical memory, number of cores in the system, task number in operation, CPU utilization, memory usage, platform name, software name, and core usage.

[0099] Establish a task operation information database, Job_operation, to record task operation logs. Key fields include: node name, node status, number of physical cores, amount of physical memory, number of cores in the system, task number being computed, CPU utilization, memory usage, platform name, software name, and core usage.

[0100] Establish a software license information database, Job_lic_data, to monitor software license status. Key fields include: number of records, time, day of the week, software name, total number of licenses, number of licenses in use, number of tasks in computation, number of tasks in the queue, total number of tasks, number of cores in computation, and number of cores in the queue.

[0101] In the second phase of the resource scheduling development logic, a judgment logic for the secondary utilization of residual cores was established through a self-developed shell + Python automated script.

[0102] Specifically, the PBS command `qstat` can be used to obtain information about queued tasks: task ID, software name, required platform name, and target core count.

[0103] Then, using the information from the queued tasks, the software license information database Job_lic_data is retrieved to view the software license.

[0104] If the license does not meet the requirements of the queuing task, it continues to queue. If the license meets the requirements of the queuing task, the information from the queuing task is used to retrieve the hardware / software matching database Hardware_software to obtain the compatible CPU platform name P1. The cluster node information database Nodes_test is retrieved to confirm the number of remaining cores of P1. If the number of remaining cores of P1 is less than the target number of cores, it continues to queue; if the number of remaining cores of P1 is greater than or equal to the target number of cores, it enters stage three of the resource scheduling development logic.

[0105] In the third phase of the resource scheduling development logic, the mapping relationship between the software and hardware resource platforms and the queued tasks is reconstructed through a self-developed shell + Python automated script.

[0106] Specifically, a software tag Sc is added to the node label of the queued task, and the relevant operations are recorded in the task operation information database Job_operation. The CPU platform information in the tag of the queued task is modified to P1, and the relevant operations are recorded in the task operation information database Job_operation. The PBS command qrun is called through a script to push the queued task to P1 for computation.

[0107] In the fourth stage of the resource scheduling development logic, the mapping relationship between software and hardware resource platforms and queued tasks is restored through a self-developed shell + Python automated script.

[0108] Specifically, after the queued task is completed, computing resources are released. Then, based on the task ID, the task operation information database Job_operation is retrieved to find the software tag information before the CPU platform modification, and the node tag of P1 is restored, thereby restoring the original mapping relationship between the hardware and software resource platform and the queued task.

[0109] In one embodiment, a first hardware processing node with a residual core count greater than or equal to a target core count is selected. The target hardware processing node with the smallest residual core count among the selected first hardware processing nodes is then determined, and this target hardware processing node is matched and bound to a queued task to obtain a hardware resource matching result.

[0110] In this embodiment, PBS selects the first hardware processing node that can take over the queued task Q2 and has the smallest number of remaining cores as the target hardware processing node to be matched and bound to the queued task Q2, and then instructs the target hardware processing node to calculate the queued task Q2.

[0111] Specifically, the PBS selects the first hardware processing node C1 whose remaining core count N1 is greater than or equal to the target core count n2. Then, the C1 node with the smallest N1 is identified as the target hardware processing node, and this target hardware processing node is matched and bound to the queued task Q2. The queued task Q2 is then pushed to the first hardware resource platform P1, instructing the target hardware processing node to compute the queued task Q2.

[0112] For example: the target core count n2 required for queuing task Q2 is 6; the first hardware processing nodes C1 in the first hardware resource platform P1 with a residual core count N1 greater than or equal to n2 include C12 and C13. C12 has a residual core count N12 of 6, and C13 has a residual core count N13 of 8. Since C12 has the smallest N1 among the selected C1 nodes, C12 is matched and bound to queuing task Q2. Then, queuing task Q2 is pushed to P1, instructing C12 to calculate queuing task Q2.

[0113] In one embodiment, for at least two queued tasks awaiting computation by the second hardware resource platform, the target number of cores required by each of the at least two queued tasks is determined. The target number of cores required by each of the at least two queued tasks is treated as an indivisible number and used to fill the remaining core count of each of the first hardware processing nodes. The total number of remaining cores on the first hardware resource platform after filling is then calculated. The at least two queued tasks are matched with each of the first hardware processing nodes according to the filling method with the smallest total number of remaining cores, resulting in a hardware resource matching result.

[0114] In this embodiment, PBS simultaneously matches multiple queued tasks with each first hardware processing node to further improve the overall utilization of the remaining cores of the first hardware resource platform.

[0115] Specifically, for multiple queued tasks Q2(Q21, Q22...Q2k) waiting for computation by the second hardware resource platform P2, PBS determines the target number of cores n2(n21, n22...n2k) required by each of these queued tasks Q2(Q21, Q22...Q2k). Then, n2(n21, n22...n2k) is used to fill the remaining number of cores N1(N11, N12...N1m) of each first hardware processing node C1(C11, C12...C1m). Here, k and m are both positive integers.

[0116] During the filling process, the number of target cores n2 used for filling must be less than or equal to the number of residual cores N1 to be filled; and n21, n22...n2k are indivisible but can be combined with each other. For example: if n21 is less than or equal to N11, n21 can be used to fill N11; if the sum of n21 and n22 is less than or equal to N11, n21 and n22 can be used together to fill N11.

[0117] By iterating through various filling methods, the total number of residual cores of the first hardware resource platform P1 after filling is calculated under each filling method. Then, the residual core total with the smallest value is selected, and according to the corresponding filling method, multiple queued tasks Q2 (Q21, Q22...Q2k) are matched with each first hardware processing node C1 (C11, C12...C1m) to obtain the hardware resource matching result.

[0118] For example: the target number of cores n2(n21,n22,n23) required by the three queuing tasks Q2(Q21,Q22,Q23) is (2,6,7); the number of residual cores N1(N11,N12) of the two first hardware processing nodes C1(C11,C12) in the first hardware resource platform P1 is (2,7).

[0119] Filling method 1: Fill N11 with n21 and fill N12 with n22. The total number of residual cores in P1 after filling is (N11-n21)+(N12-n22)=(2-2)+(7-6)=1.

[0120] Filling method 2: Fill N11 with n21 and fill N12 with n23. The total number of residual cores in P1 after filling is (N11-n21)+(N12-n23)=(2-2)+(7-7)=0.

[0121] Filling method 3: Fill N12 with n21. The total number of residual cores in P1 after filling is N11 + (N12 - n21) = 2 + (7 - 2) = 7.

[0122] Of the three filling methods, filling method 2 results in the smallest total number of residual cores for P1 after filling. Therefore, according to filling method 2, queued task Q21 is matched and bound to the first hardware processing node C11, and queued task Q23 is matched and bound to the first hardware processing node C12. Queued tasks Q21 and Q23 can then be pushed to P1, and C11 is instructed to calculate queued task Q21, while C12 calculates queued task Q23.

[0123] Figure 8 A block diagram of a resource scheduling apparatus according to an embodiment of the present application is shown, the apparatus comprising:

[0124] The target core determination module 210 is configured to determine the number of target cores required by a queued task waiting for computation by a second hardware resource platform.

[0125] The residual core determination module 220 is configured to obtain the number of idle residual cores of each first hardware processing node in the first hardware resource platform for the first hardware resource platform with idle residual cores.

[0126] The hardware resource matching module 230 is configured to match the queued tasks with each of the first hardware processing nodes based on the target number of cores and the remaining number of cores, so as to obtain the hardware resource matching result.

[0127] The queuing task push module 240 is configured to push the queuing task to the first hardware resource platform based on the hardware resource matching result, and instruct the first hardware processing node whose number of residual cores is greater than or equal to the number of target cores to calculate the queuing task.

[0128] In one exemplary embodiment of this application, the device is configured as follows:

[0129] Determine the software certificate required for the queued task;

[0130] When an idle software certificate is detected on the software resource platform, and a first hardware processing node with a residual core count greater than or equal to the target core count is detected based on the hardware resource matching result, the software certificate is assigned to the queued task, and the queued task is pushed to the first hardware resource platform.

[0131] In one exemplary embodiment of this application, the device is configured as follows:

[0132] After the queuing task is completed, the software certificate in the software resource platform is released, and the remaining cores in the first hardware resource platform occupied by the queuing task are released.

[0133] In one exemplary embodiment of this application, the device is configured as follows:

[0134] The queue members whose tags are bound to the second hardware resource platform are filtered out to obtain the queued tasks;

[0135] When the hardware resource matching result determines that the first hardware resource platform has hardware processing nodes with a residual core count greater than or equal to the target core count, the tag of the queued task is modified to be bound to the first hardware resource platform, and the queued task is pushed to the first hardware resource platform.

[0136] In one exemplary embodiment of this application, the device is configured as follows:

[0137] When it is determined, based on the hardware resource matching result, that there are hardware processing nodes in the first hardware resource platform with a residual core count greater than or equal to the target core count, the queued task is bound to the tag of the target hardware processing node in the first hardware resource platform used to calculate the queued task.

[0138] After the queuing task calculation is completed, the remaining cores in the target hardware processing node occupied by the queuing task are released, and the queuing task is unbound from the tag of the target hardware processing node.

[0139] In one exemplary embodiment of this application, the device is configured as follows:

[0140] Select the first hardware processing node whose number of residual cores is greater than or equal to the target number of cores;

[0141] The target hardware processing node with the smallest number of residual cores among the first selected hardware processing nodes is identified, and the target hardware processing node is matched and bound with the queued task to obtain the hardware resource matching result.

[0142] In one exemplary embodiment of this application, the device is configured as follows:

[0143] For at least two queued tasks waiting to be computed by the second hardware resource platform, determine the target number of cores required by each of the at least two queued tasks.

[0144] The target core counts required by the at least two queued tasks are respectively taken as indivisible numbers to fill the remaining core counts of each first hardware processing node, and the total number of remaining cores of the first hardware resource platform after filling is calculated.

[0145] The hardware resource matching results are obtained by matching the at least two queued tasks with each of the first hardware processing nodes according to the filling method of the remaining cores with the smallest value.

[0146] The following is for reference. Figure 9 To describe the electronic device 30 according to an embodiment of this application. Figure 9 The electronic device 30 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0147] like Figure 9 As shown, the electronic device 30 is presented in the form of a general-purpose computing device. The components of the electronic device 30 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, and a bus 330 connecting different system components (including storage unit 320 and processing unit 310).

[0148] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the exemplary method description section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 310 can perform actions such as... Figure 3 The steps shown are as follows.

[0149] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 3201 and / or cache memory 3202, and may further include read-only memory (ROM) 3203.

[0150] Storage unit 320 may also include a program / utility 3204 having a set (at least one) program module 3205, such program module 3205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0151] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0152] Electronic device 30 can also communicate with one or more external devices 400 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 30, and / or with any device that enables electronic device 30 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Input / output (I / O) interface 350 is connected to display unit 340. Furthermore, electronic device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. As shown, network adapter 360 communicates with other modules of electronic device 30 via bus 330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 30, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0153] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the embodiments of this application.

[0154] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0155] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This program product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0156] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0158] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0159] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as JAVA and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0160] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0161] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0162] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0163] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A resource scheduling method, characterized in that, The method includes: For at least two queued tasks waiting to be computed by the second hardware resource platform, determine the target number of cores required by each of the at least two queued tasks. For a first hardware resource platform with idle and residual cores, the number of idle residual cores in each first hardware processing node of the first hardware resource platform is obtained; the residual cores include idle cores in the first hardware processing nodes that are in computing state. The target core counts required by the at least two queued tasks are respectively taken as indivisible numbers to fill the remaining core counts of each first hardware processing node, and the total number of remaining cores of the first hardware resource platform after filling is calculated. According to the filling method of the minimum value of the total number of remaining cores, the at least two queued tasks are matched with each of the first hardware processing nodes to obtain hardware resource matching results; Based on the hardware resource matching result, the at least two queued tasks are pushed to the first hardware resource platform, and the first hardware processing node with a residual core count greater than or equal to the target core count is instructed to calculate the at least two queued tasks.

2. The method of claim 1, wherein, The method further includes: Determine the software certificate required for the queued task; When an idle software certificate is detected on the software resource platform, and a first hardware processing node with a residual core count greater than or equal to the target core count is detected based on the hardware resource matching result, the software certificate is assigned to the queued task, and the queued task is pushed to the first hardware resource platform.

3. The method of claim 2, wherein, The method further includes: After the queuing task is completed, the software certificate in the software resource platform is released, and the remaining cores in the first hardware resource platform occupied by the queuing task are released.

4. The method of claim 1, wherein, The method further includes: filtering out queue members whose tags are bound to the second hardware resource platform in the queuing queue to obtain the queuing task; Based on the hardware resource matching result, pushing the queued task to the first hardware resource platform includes: when it is determined from the hardware resource matching result that the first hardware resource platform has hardware processing nodes with a residual core count greater than or equal to the target core count, modifying the tag of the queued task to bind it to the first hardware resource platform, and pushing the queued task to the first hardware resource platform.

5. The method of claim 4, wherein, The method further includes: When it is determined, based on the hardware resource matching result, that there are hardware processing nodes in the first hardware resource platform with a residual core count greater than or equal to the target core count, the queued task is bound to the tag of the target hardware processing node in the first hardware resource platform used to calculate the queued task. After the queuing task calculation is completed, the remaining cores in the target hardware processing node occupied by the queuing task are released, and the queuing task is unbound from the tag of the target hardware processing node.

6. The method of claim 1, wherein, Based on the target number of cores and the remaining number of cores, the queued tasks are matched with each of the first hardware processing nodes to obtain hardware resource matching results, including: Select the first hardware processing node whose number of residual cores is greater than or equal to the target number of cores; The target hardware processing node with the smallest number of residual cores among the first selected hardware processing nodes is identified, and the target hardware processing node is matched and bound with the queued task to obtain the hardware resource matching result.

7. A resource scheduling apparatus, characterized by comprising: The device includes: The target core determination module is configured to determine the number of target cores required by at least two queued tasks waiting for computation by the second hardware resource platform. The residual core determination module is configured to, for a first hardware resource platform with idle residual cores, obtain the number of idle residual cores in each first hardware processing node of the first hardware resource platform; the residual cores include idle cores in the first hardware processing nodes that are in computing state. The hardware resource matching module is configured to use the target core counts required by the at least two queued tasks as indivisible numbers to fill the remaining core counts of each first hardware processing node, and calculate the total number of remaining cores of the first hardware resource platform after filling; and match the at least two queued tasks with each first hardware processing node according to the filling method with the smallest value of the total number of remaining cores to obtain the hardware resource matching result. The queued task push module is configured to push the at least two queued tasks to the first hardware resource platform based on the hardware resource matching result, and instruct the first hardware processing node, whose number of residual cores is greater than or equal to the number of target cores, to calculate the at least two queued tasks.

8. An electronic device, comprising: include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the processor of a computer, cause the computer to perform the method described in any one of claims 1 to 6.

Images