Multi-queue job scheduling method and system and nonvolatile storage medium

Abstract

The invention discloses a multi-queue job scheduling method and system and a nonvolatile storage medium. The method comprises the steps that operation information of all sub-jobs located in multiple execution queues is obtained, the multiple sub-jobs located in the different execution queues belong to a complete job to be executed, and the operation information comprises priority information and resource operation information; based on the operation information, whether a plurality of target sub-jobs belonging to the same job to be executed meet preemption conditions or not is judged, and the preemption conditions comprise a priority condition and a resource demand condition; under the condition that the multiple target sub-jobs meet the preemption condition, dispatching instructions are issued to controllers corresponding to the multiple execution queues respectively, and the dispatching instructions are used for enabling the multiple target sub-jobs to be preempted to the running positions in the corresponding execution queues. The technical problem that a traditional single-cluster preemption method is prone to causing asynchronous operation of multiple sub-jobs of the same job is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer systems, in particular to a multi-queue job scheduling method and system and a non-volatile storage medium. BACKGROUND

[0002] In a job scenario requiring a large amount of running resources, especially when the job is deployed across multiple intelligence calculation centers, the coordination and allocation of resources become a key challenge. A single job needs to be split into multiple sub-jobs and started in different intelligence calculation centers. In this process, the synchronous start of all sub-jobs and resource readiness are crucial to ensure the efficiency and consistency of distributed training. When resources are insufficient, high-priority jobs need to be prioritized through preemption mechanisms. Traditional single-cluster preemption schemes only focus on their own priority, and the local priority strategies (such as queue weight or resource occupancy threshold) of each intelligence calculation center are independently formulated, lacking a collaborative evaluation mechanism for global job objectives, which can easily lead to different running of sub-jobs belonging to the same job on multiple platforms.

[0003] At present, there is no effective solution to the above problems. SUMMARY

[0004] The embodiments of the present application provide a multi-queue job scheduling method, system and non-volatile storage medium to at least solve the technical problem that the traditional single-cluster preemption method can easily lead to different running of multiple sub-jobs of the same job.

[0005] According to an aspect of the embodiments of the present application, a multi-queue job scheduling method is provided, comprising: obtaining running information of each sub-job located in multiple execution queues, wherein the multiple sub-jobs located in different execution queues belong to a complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the running information includes priority information and resource running information; based on the running information, judging whether multiple target sub-jobs belonging to the same to-be-executed job all meet preemption conditions, wherein the preemption conditions include priority conditions and resource demand conditions; and in the case that the multiple target sub-jobs all meet the preemption conditions, issuing a scheduling instruction to a controller corresponding to each of the multiple execution queues, wherein the scheduling instruction is used to make the multiple target sub-jobs preempt the running positions in the corresponding execution queues.

[0006] Optionally, reading the running information of each sub-job located in the multiple execution queues comprises: reading the running information uploaded based on the local controllers corresponding to the multiple execution queues from a central database.

[0007] Optionally, based on the running information, it is judged whether the multiple target sub-jobs belonging to the same to-be-executed job all satisfy the preemption condition, comprising: in the multiple execution queues, the local priority size relationship between the target sub-job and the running sub-job is compared respectively, wherein the running sub-job is a sub-job located in a running position and being executed; in the case that the local priority of the target sub-job is greater than the local priority of the running sub-job, it is determined that the target sub-job satisfies the priority condition.

[0008] Optionally, in the multiple execution queues, the local priority size relationship between the target sub-job and the running sub-job is compared respectively, comprising: the global priority of the target sub-job and the global priority of the running sub-job are acquired respectively; based on the global priority, the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job are determined respectively; the size relationship between the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job is compared.

[0009] Optionally, based on the running information, it is judged whether the multiple target sub-jobs belonging to the same to-be-executed job all satisfy the preemption condition, comprising: in the multiple execution queues, the resource usage of the running sub-job and the resource demand of the target sub-job are acquired respectively, wherein the running sub-job is a sub-job located in a running position and being executed; in the case that the resource usage is not less than the resource demand, it is determined that the target sub-job satisfies the resource demand condition.

[0010] Optionally, in the case that the multiple target sub-jobs all satisfy the preemption condition, a release instruction is issued to the multiple execution queues, wherein the release instruction is used to make the pre-empted running sub-job and other sub-jobs belonging to the same to-be-executed job as the running sub-job release the occupied running resources, and the running sub-job is a sub-job located in a running position and being executed.

[0011] According to another aspect of the embodiment of the application, a multi-queue job scheduling method is further provided, comprising: acquiring running information of multiple sub-jobs in a local execution queue, wherein the running information comprises priority information and resource running information; sending the running information to a central controller; receiving a scheduling instruction sent by the central controller, wherein the scheduling instruction is determined by the central controller based on the running information; based on the scheduling instruction, scheduling a target sub-job to a running position in the local execution queue, wherein the multiple sub-jobs comprise the target sub-job.

[0012] Optionally, the running state of the pre-empted running sub-job is saved in a local database, wherein the running sub-job is a sub-job located in a running position and being executed, and the running state comprises running parameters and data processing progress.

[0013] Optionally, in the case of re-running the running sub-job back to the running position, based on the running state saved in the local database, the running sub-job is resumed.

[0014] According to another aspect of the embodiments of the present application, a multi-queue job scheduling system is also provided, comprising: a central controller configured to execute any one of the above multi-queue job scheduling methods; and a plurality of local controllers configured to schedule sub-jobs in local execution queues based on scheduling instructions issued by the central controller.

[0015] According to still another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, comprising a stored program, wherein when the program is executed, the non-volatile storage medium controls a device in which the non-volatile storage medium is located to execute any one of the above multi-queue job scheduling methods.

[0016] According to yet another aspect of the embodiments of the present application, a computer device is also provided, comprising a processor configured to execute a program, wherein when the program is executed, any one of the above multi-queue job scheduling methods is executed.

[0017] According to still another aspect of the embodiments of the present application, a computer program product is also provided, comprising a computer program, wherein when the computer program is executed by a processor, any one of the above multi-queue job scheduling methods is implemented.

[0018] In the embodiments of the present application, a multi-queue job scheduling method is adopted, running information of each sub-job located in a plurality of execution queues is read, wherein the plurality of sub-jobs located in different execution queues belong to a complete to-be-executed job, and the running information comprises priority information and resource running information; based on the running information, it is determined whether a plurality of target sub-jobs belonging to the same to-be-executed job all satisfy preemption conditions, wherein the preemption conditions comprise priority conditions and resource demand conditions; in the case where the plurality of target sub-jobs all satisfy the preemption conditions, a scheduling instruction is issued to a controller corresponding to each of the plurality of execution queues, wherein the scheduling instruction is used to make the plurality of target sub-jobs preempt a running position in the corresponding execution queue, so that the purpose that the plurality of sub-jobs of the same job can synchronously obtain running resources is achieved, thereby realizing the technical effects of improving resource utilization and job running efficiency, and further solving the technical problem that the traditional single-cluster preemption method easily causes the plurality of sub-jobs of the same job to run asynchronously. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application. In the drawings:

[0020] Figure 1A hardware structure block diagram of a computer terminal for implementing a multi-queue job scheduling method is shown;

[0021] Figure 2 A flowchart of a multi-queue job scheduling method according to an embodiment of the present application is shown;

[0022] Figure 3 A flowchart of a multi-queue joint preemption process according to an optional embodiment of the present application is shown;

[0023] Figure 4 A flowchart of a multi-queue preemption process according to an optional embodiment of the present application is shown;

[0024] Figure 5 An architecture diagram of a cross-wide-area multi-cluster heterogeneous queue preemption scheduling system according to an optional embodiment of the present application is shown;

[0025] Figure 6 A running diagram of a cross-wide-area multi-cluster heterogeneous queue preemption scheduling system according to an optional embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to an embodiment of the present invention, a multi-queue job scheduling method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a multi-queue job scheduling method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0030] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0031] The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage devices corresponding to the multi-queue job scheduling method in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the multi-queue job scheduling method of the application program described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor, which can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The display can be, for example, a touch screen type liquid crystal display (LCD) that can enable a user to interact with the user interface of the computer terminal 10.

[0033] Figure 2 is a flowchart of the multi-queue job scheduling method provided according to the embodiments of the present application, as shown in Figure 2 The method comprises the following steps:

[0034] Step S201, reading running information of each sub-job located in a plurality of execution queues, wherein the plurality of sub-jobs located in different execution queues respectively belong to a complete to-be-executed job, and the running information comprises priority information and resource running information.

[0035] In this step, in the environment across wide-area multi-intelligent computing centers, a complete to-be-executed job often needs to be split and scheduled to run in a plurality of execution queues. The execution queue here refers to a queue system for managing and scheduling computing tasks within an intelligent computing center, which respectively allocates resources such as GPUs, CPUs, memories, etc. according to local strategies to execute jobs in the task list. Due to limited resources and extremely high resource synchronization requirements of to-be-executed jobs, the priority information and resource running state of each sub-job are crucial for global resource optimization and deployment.

[0036] The priority information defines the execution order and importance of each sub-job, and determines which sub-job has the right to obtain or retain resources in the case of resource shortage. It is usually related to job type, tenant level, job urgency, etc. Through a priority mapping manager, the global priority can be converted into specific priority description of each intelligent computing center, ensuring consistent determination of the priority of cross-cluster jobs.

[0037] The resource running information not only covers the current resource usage of the sub-job, such as the allocated GPU, CPU core number, memory and storage space, but also includes the resource demand, that is, the minimum or ideal amount of resources required for executing the sub-job. The resource usage of the sub-job is an index for measuring the resource consumption of the current sub-job in the intelligent computing center, which directly reflects the actual resource occupation of the sub-job in the execution process. The resource demand represents the resources that the sub-job is expected to need in the next running period. It is calculated based on the computing demand of the job, the expected running efficiency and the possible resource competition and other factors. The resource demand is crucial for predicting and planning resource allocation, and helps the scheduling system to pre-judge which jobs may need additional resources, and when the resources are tight, which jobs can be temporarily interrupted or the resources can be redistributed to ensure that the resource demand of high-priority jobs is met in priority.

[0038] In step S202, based on the running information, it is judged whether the multiple target sub-jobs belonging to the same to-be-executed job all satisfy the preemption condition, wherein the preemption condition includes a priority condition and a resource demand condition.

[0039] In this step, the priority condition is the basis for predicting whether the sub-job can preoccupy resources. Each sub-job is assigned a globally consistent priority, which is uniformly marked by the priority mapping manager between the intelligent computing centers, ensuring that the priority of cross-center jobs is consistently recognized in all participating scheduling queues. When the priority of a sub-job is higher than the job in the running position in the execution queue, it means that it can legally preoccupy the resources occupied by the low-priority job to meet its own resource demand, thereby ensuring the smooth execution of high-priority jobs. The resource demand condition focuses on the current resource state and future demand of the sub-job. The resource usage in the running information can help the scheduling system understand the actual resource occupation of each sub-job, while the resource demand reveals the resources that the sub-job is expected to need for further running. When the resource demand of a sub-job can be satisfied by the available resources of the sub-job running in the current queue, it is considered to satisfy the priority condition and can preoccupy the resources of the running position sub-job.

[0040] When the sub-jobs meet the priority condition and the resource requirement condition at the same time, it can be considered that the current state of the sub-job is a pre-preempted state, that is, a preempted ready state, and the preempted state indicates that the sub-job has been preempted, and the state is synchronized to the global job scheduling, so that the global job scheduling synchronizes the state to the queues of other sub-jobs belonging to the same job. If all sub-jobs are ready, it indicates that the to-be-executed job can execute preemption, and the state of the sub-job is updated to pre-emptable. When the sub-job is pre-emptable, it indicates that the sub-job can preempt the local low-priority job. After the preemption executor obtains the sub-job for executing preemption, the local queue scheduler is triggered to execute preemption. When the sub-job is in the preempted state, the preemption executor will notify the local scheduler to evict the sub-job and protect the preemption site.

[0041] In step S203, when the plurality of target sub-jobs meet the preemption condition, a scheduling instruction is issued to the controllers corresponding to the plurality of execution queues respectively, wherein the scheduling instruction is used to make the plurality of target sub-jobs preempt the running positions in the corresponding execution queues.

[0042] In this step, when it is evaluated that all target sub-jobs meet the preemption condition, that is, these sub-jobs not only have higher priority than other sub-jobs in the execution queue, but also the resource occupation of the running sub-job can meet the resource requirement of the sub-job, which can be met without significantly affecting other jobs. At this time, the controller will execute the next scheduling decision, that is, issue a scheduling instruction to the local controllers of each intelligent computing center.

[0043] The specific function of the scheduling instruction is to instruct the controllers (that is, the preemption executors) of each execution queue to perform preemption operation on the target sub-jobs, so that they obtain the running resources in the corresponding execution queue. In the case of resource limitation, it means that the sub-jobs with lower priority will be paused or terminated to release enough resources for the sub-jobs with higher priority. This process needs strict coordination to ensure that the preemption actions of all sub-jobs are completed almost at the same time window, avoiding the partial sub-jobs starting ahead of or delaying, which destroys the synchronization and overall performance of the job.

[0044] For example, in the cross-domain training scene, Figure 3 is a flowchart of a multi-queue joint preemption process according to an optional embodiment of the present application, as Figure 3 shown, the large model training job is distributed in multiple intelligent computing clusters after scheduling, and when preemption is initiated, the sub-jobs distributed in multiple intelligent computing centers need to meet the preemption condition. When the sub-jobs of a certain intelligent computing center meet the preemption condition, the pre-preempted state needs to be notified to the global job scheduling, and after all sub-jobs meet the preemption condition, the different queues of each intelligent computing center are notified to execute preemption.

[0045] Through the above steps, the purpose that multiple sub-jobs of the same job can synchronously obtain running resources is achieved, thereby realizing the technical effects of improving resource utilization and job running efficiency, and further solving the technical problem that the traditional single cluster preemption method is easy to cause multiple sub-jobs of the same job to run asynchronously.

[0046] As an optional embodiment, reading the running information of each sub-job located in the multiple execution queues comprises: reading the running information uploaded by the local controller corresponding to each of the multiple execution queues from the central database.

[0047] Optionally, the central database refers to a distributed key-value storage system, such as ETCD, used to maintain the state and resource information of cross-cluster jobs. When a job across multiple intelligent algorithm centers in a wide area is split into multiple sub-jobs and scheduled in different execution queues, the running information of each sub-job (including priority information, resource running state, etc.) needs to be monitored and recorded in real time. The local controller (i.e., the preemption executor) of each queue is responsible for collecting and reporting the running information of the sub-jobs in its cluster to the central database. The central controller regularly or event-drivenly obtains these running information from the central database, which is used as the basis for evaluating whether the sub-jobs meet the preemption conditions. By taking the central database as the aggregation point of information, not only the efficient storage and sharing of running information are realized, but also the consistency and real-time of information are ensured, which provides a solid data foundation for the multi-queue joint preemption mechanism.

[0048] As an optional embodiment, based on the running information, judging whether multiple target sub-jobs belonging to the same to-be-executed job all meet the preemption conditions comprises: comparing the local priority size relationship between the target sub-jobs and the running sub-jobs in the multiple execution queues, respectively, wherein the running sub-jobs are sub-jobs located in the running position and being executed; in the case that the local priority of the target sub-job is greater than the local priority of the running sub-job, it is determined that the target sub-job meets the priority condition.

[0049] Optionally, the priority is determined based on a dual system of global priority and local priority. The global priority is a priority standard uniformly set across all intelligence calculation centers, used to coordinate and manage resource allocation between different clusters, ensuring that high-priority cross-cluster jobs can obtain the required resource support. The local priority is separately set in each intelligence calculation center, which reflects the local resource allocation strategy and the priority order of job scheduling. Through the priority mapping manager, a mapping relationship is established between the global priority and the local priority, so that the global priority can be accurately identified and applied in each intelligence calculation center. When the target sub-job needs to preempt the running position in the execution queue, the system compares the local priority of the target sub-job with the local priority of the current running sub-job. If the local priority of the target sub-job (which can be obtained based on the global priority mapping) is greater than the local priority of the running sub-job, the system will determine that the target sub-job meets the priority condition for preemption and is qualified for resource preemption.

[0050] As an optional embodiment, in the plurality of execution queues, the local priority of the target sub-job and the local priority of the running sub-job are compared respectively, including: obtaining the global priority of the target sub-job and the global priority of the running sub-job respectively; based on the global priority, determining the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job respectively; comparing the size relationship between the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job.

[0051] Optionally, the local priority of the sub-job can be mapped from the global priority through a preset mapping table. For example, Table 1 is a mapping table of global priority and queue priority according to an optional embodiment of the present application. As shown in Table 1, the priority mapping table needs to enter information when registering and managing different intelligence calculation centers, and subsequently needs to refer to this table to convert the priority into the priority description of the corresponding intelligence calculation center when rendering the sub-job generated after scheduling. The mapping of global priority and priority of each intelligence calculation center needs to consider the priority between tenants and the priority between jobs to ensure the consistency of global tenant / job priority.

[0052] Global priority Alien queue 1 priority Alien queue 2 priority Alien queue 3 priority Super / Root P1 High Administrator VIP 3 P2 Middle Manager VIP 2 P3 Low NormalUser VIP 1 P4 AnonymousUser VIP 0 P5

[0053] Table 1: Mapping table of global priority and queue priority

[0054] As an optional embodiment, based on the running information, it is determined whether the plurality of target sub-jobs belonging to the same to-be-executed job all meet the preemption condition, including: in the plurality of execution queues, obtaining the resource usage of the running sub-job and the resource demand of the target sub-job respectively, wherein the running sub-job is a sub-job located in the running position and being executed; under the condition that the resource usage is not less than the resource demand, it is determined that the target sub-job meets the resource demand condition.

[0055] Optionally, when evaluating whether the target sub-job meets the resource demand condition, the resource demand amount of the target sub-job and the resource usage amount of the running sub-jobs in all relevant execution queues can be first read from the central database. Then, for each running sub-job in each execution queue, it is checked whether its resource usage amount is sufficient to cover the resource demand amount of the target sub-job. If the resource usage amount of a running sub-job in a certain execution queue is not less than the resource demand amount of the target sub-job, it can be considered that the target sub-job meets the resource demand condition in this execution queue. This means that the target sub-job has sufficient reason to preempt the necessary resources from the running sub-job to meet its own computing demand and advance the execution progress.

[0056] As an optional embodiment, in the case where multiple target sub-jobs all meet the preemption condition, release instructions are issued to multiple execution queues, wherein the release instructions are used to make the preempted running sub-jobs and other sub-jobs belonging to the same to-be-executed job as the running sub-jobs release the occupied running resources, the running sub-jobs being the sub-jobs located at the running position and being executed.

[0057] Optionally, when all target sub-jobs meet the priority and resource demand conditions, release instructions can also be issued to multiple execution queues, instructing each intelligent computing center to stop the low-priority sub-job at the current running position and requiring other sub-jobs belonging to the same job to also release resources. This instruction ensures that the sub-jobs of high-priority jobs can quickly obtain the required resources, while maintaining the integrity of the job through the synchronization mechanism, avoiding resource fragmentation and job startup delay, and improving the efficiency and resource utilization of cross-intelligent computing center large model training.

[0058] For example, in the cross-domain training scenario, Figure 4 is a flowchart of a multi-queue preemption process according to an optional embodiment of the present application, as Figure 4 As shown in the figure, the sub-jobs distributed in each intelligent computing center can preempt low-priority jobs and can also be preempted by higher-priority jobs. When a sub-job in a certain intelligent computing center is preempted, the preempted state needs to be reported to the global job scheduler. When the job scheduler learns that any sub-job is preempted, it needs to notify other sub-jobs belonging to the same job as the sub-job to release resources.

[0059] According to the embodiment of the present application, a multi-queue job scheduling method is also provided, which comprises: obtaining running information of a plurality of sub-jobs in a local execution queue, wherein the running information comprises priority information and resource running information; sending the running information to a central controller; receiving a scheduling instruction sent by the central controller, wherein the scheduling instruction is determined by the central controller based on the running information; and scheduling a target sub-job to a running position in the local execution queue based on the scheduling instruction, wherein the plurality of sub-jobs comprise the target sub-job.

[0060] Optionally, the local controller continuously collects running information of each sub-job in the local execution queue, including priority and resource usage, and particularly focuses on the state of the target sub-job. When the central controller makes a scheduling decision based on the aggregated global running information, the local controller immediately receives a scheduling instruction that explicitly specifies the specific requirements for resource adjustment. Subsequently, the local controller accurately schedules the target sub-job to a running position in the queue according to the scheduling instruction, while releasing resources for low-priority jobs, ensuring that high-priority sub-jobs are executed preferentially. The entire process embodies the close cooperation between the local execution queue and the central controller, and realizes effective management of resources and rapid response of jobs.

[0061] As an optional embodiment, the running state of the preempted running sub-job, which is a sub-job located in a running position and being executed, is saved in a local database, wherein the running state comprises running parameters and data processing progress.

[0062] Optionally, in the multi-queue joint preemption mechanism, when a high-priority sub-job needs to occupy the resources of a low-priority sub-job that is running, the local controller will take a measure called "on-site protection" to save the running state of the running sub-job that is about to be preempted. This process aims to ensure that even if a sub-job is paused due to resource preemption, it can quickly resume its running state before being interrupted when the resource is rescheduled, reducing the loss caused by job interruption.

[0063] For example, in the cross-domain training scenario, when determining that preemption is executable, the scheduler needs to determine the jobs that need to be evicted. The intelligent algorithm center will take a snapshot of the state of these jobs, such as checkpoints of large model training jobs, runtime related parameters, data processing progress, etc. Unlike the current on-site protection, the state data of the related on-site protection needs to have a global data view and cross-domain state data synchronization capability. At the same time, the preempted state needs to be synchronized to the global job scheduling, and then notify the intelligent algorithm centers where other sub-jobs of the same job are deployed to perform cross-domain resource preemption.

[0064] As an optional embodiment, in the case that the running sub-job reenters the running position, the execution of the running sub-job is resumed based on the running state saved in the local database.

[0065] Optionally, when the running sub-job previously preempted reacquires resources and can reenter the running position to continue execution, the local controller uses the running state information previously saved in the local database to quickly resume the execution of the sub-job. This resumption process is a key link to ensure job continuity and efficiency, which enables the system to maintain the integrity of the job and the consistency of the data in a dynamically changing resource environment.

[0066] For example, in a cross-domain training scenario, the preempted job is added to the to-be-scheduled queue by local scheduling to obtain a rescheduling opportunity. When the job reacquires resources, the training is resumed from the snapshot state saved in the live protection phase. If the subsequent job scheduling part implements cross-domain rescheduling logic, the job can obtain a rescheduling opportunity through global job scheduling and be deployed and run at other AI centers.

[0067] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0068] From the above description of the embodiments, those skilled in the art can clearly understand that the multi-queue job scheduling method according to the above embodiments can be implemented by means of software and a general hardware platform as necessary, and of course, it can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method of each embodiment of the present application.

[0069] According to the embodiments of the present application, a multi-queue job scheduling system is also provided, which includes: a central controller configured to execute any one of the multi-queue job scheduling methods described above; and a plurality of local controllers configured to schedule sub-jobs in a local execution queue based on scheduling instructions issued by the central controller.

[0070] Optionally, the multi-queue job scheduling system takes a central controller as the core, comprehensively oversees the global resource view and scheduling decision, and cooperates with multiple distributed local controllers. The latter follows the instructions issued by the center, accurately schedules the sub-jobs in the local execution queue, ensures efficient synchronous operation of the large model training job across the wide area multi-intelligent algorithm center, and simultaneously realizes seamless execution of the optimal allocation and preemption mechanism of resources. The close cooperation between the central controller and the local controller builds an intelligent scheduling network that flexibly responds to resource changes and ensures job continuity.

[0071] As an optional embodiment, a cross-wide-area multi-cluster heterogeneous queue preemption scheduling system is also provided. Figure 5 According to the optional embodiment of the present application, a schematic diagram of the architecture of the cross-wide-area multi-cluster heterogeneous queue preemption scheduling system is provided, as shown in Figure 5 The task scheduling is responsible for maintaining the cross-cluster resource view, storing the real-time resource status of each intelligent algorithm center, implementing the job splitting strategy, generating the sub-job and resource demand description, executing the global preemption decision, and issuing the preemption instruction to the local scheduler. The task agent interfaces with the management and control platform of each intelligent algorithm center, receives the preemption strategy issued by the global scheduler, executes the preemption action during local resource allocation, and feeds back the preemption execution status. To realize the preemption mechanism of the cross-domain multi-heterogeneous queue, the following functional modules need to be implemented in the task scheduling and the task agent respectively:

[0072] The priority mapping manager defines the urgency of the job. High-priority jobs can preempt the resources of low-priority jobs. The priority of the heterogeneous queue is designed independently. To realize the preemption cooperation of the heterogeneous queue, a unified priority specification needs to be established. The function of this functional module is to maintain the global priority mapping table, map the priority of the managed heterogeneous queue with the global priority, and establish a unified priority rule.

[0073] The synchronous preemption controller splits the large model training job into multiple sub-jobs and deploys them in suitable intelligent algorithm centers. The scheduling strategy of each intelligent algorithm center will lead to different states of the sub-jobs belonging to the same job. Distributed in multiple intelligent algorithm centers, the preemption opportunities of each intelligent algorithm center are inconsistent. For the same job, all its sub-jobs need to obtain the required resources for normal training. Therefore, the task scheduling needs to coordinate the preemption opportunities of each heterogeneous queue to realize synchronous preemption and synchronous operation of the job.

[0074] Job state synchronization. This part of the function is implemented on the task agent, which reports the local job state to the task scheduling, and realizes the synchronization of the job state between the global and local intelligent algorithm centers.

[0075] The preemption executor, the synchronization preemption controller as a global preemption coordinator, and the preemption executor as a trigger of actual heterogeneous queue preemption on the task agent. After the preemption state of all sub-jobs is ready, the synchronization preemption controller sets the job preemption state to a pre-emptible state. The preemption executor notifies the pre-emptible state of the sub-job to the scheduler of the heterogeneous queue, and the actual preemption is performed by the heterogeneous queue scheduler.

[0076] Figure 6 According to an optional embodiment of the present application, a running schematic diagram of a cross-wide-area multi-cluster heterogeneous queue preemption scheduling system is provided. Figure 6 As shown in the figure, the task scheduling and the task agent cooperate to complete the running. After the task scheduling distributes the sub-jobs to each intelligent computing center, the task agent of each intelligent computing center will undertake these sub-jobs, manage the life cycle of the sub-jobs, and cooperate with the global task scheduling to make the sub-jobs completed smoothly in the local intelligent computing center. In the process of heterogeneous queue preemption, the functions to be realized are composed of two parts: global / local sub-job state synchronization and heterogeneous multi-queue preemption opportunity synchronization.

[0077] The embodiment of the present application can provide a computer device. Optionally, in the embodiment, the computer device can be located in at least one network device of a plurality of network devices of a computer network. The computer device comprises a memory and a processor.

[0078] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the multi-queue task scheduling method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned multi-queue task scheduling method. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the computer terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0079] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining running information of each sub-job located in a plurality of execution queues, wherein the plurality of sub-jobs located in different execution queues respectively belong to one complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the running information includes priority information and resource running information; judging whether a plurality of target sub-jobs belonging to the same to-be-executed job all meet preemption conditions based on the running information, wherein the preemption conditions include priority conditions and resource demand conditions; and in the case that the plurality of target sub-jobs all meet the preemption conditions, issuing a scheduling instruction to a controller corresponding to each of the plurality of execution queues, wherein the scheduling instruction is used to make the plurality of target sub-jobs preempt the running positions in the corresponding execution queues.

[0080] Optionally, the processor can further execute program codes of the following steps: reading running information of each sub-job located in a plurality of execution queues, including: reading running information uploaded based on a local controller corresponding to each of the plurality of execution queues from a central database.

[0081] Optionally, the processor can further execute program codes of the following steps: judging whether a plurality of target sub-jobs belonging to the same to-be-executed job all meet preemption conditions based on the running information, including: comparing local priority size relationship between the target sub-job and a running sub-job in each of the plurality of execution queues, wherein the running sub-job is a sub-job located in a running position and being executed; and in the case that the local priority of the target sub-job is greater than the local priority of the running sub-job, determining that the target sub-job meets the priority condition.

[0082] Optionally, the processor can further execute program codes of the following steps: comparing local priority size relationship between the target sub-job and a running sub-job in each of the plurality of execution queues, including: respectively acquiring global priority of the target sub-job and global priority of the running sub-job; respectively determining local priority corresponding to the target sub-job and local priority corresponding to the running sub-job based on the global priority; and comparing size relationship between the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job.

[0083] Optionally, the processor can further execute program codes of the following steps: judging whether a plurality of target sub-jobs belonging to the same to-be-executed job all meet preemption conditions based on the running information, including: respectively acquiring resource usage of a running sub-job and resource demand of a target sub-job in each of the plurality of execution queues, wherein the running sub-job is a sub-job located in a running position and being executed; and in the case that the resource usage is not less than the resource demand, determining that the target sub-job meets the resource demand condition.

[0084] Optionally, the processor can further execute program codes of the following steps: in a case that the multiple target sub-jobs all satisfy the preemption condition, issuing a release instruction to the multiple execution queues, wherein the release instruction is used to make the preempted running sub-job and other sub-jobs belonging to the same to-be-executed job as the running sub-job release the occupied running resources, and the running sub-job is a sub-job located at a running position and being executed.

[0085] By adopting the embodiment of the present application, a multi-queue job scheduling method is provided. Running information of each sub-job located in multiple execution queues is acquired, wherein the multiple sub-jobs located in different execution queues belong to one complete to-be-executed job, the sub-job comprises one or more to-be-executed tasks, and the running information comprises priority information and resource running information. Whether the multiple target sub-jobs belonging to the same to-be-executed job all satisfy a preemption condition is judged based on the running information, wherein the preemption condition comprises a priority condition and a resource demand condition. In a case that the multiple target sub-jobs all satisfy the preemption condition, a scheduling instruction is issued to the controllers corresponding to the multiple execution queues respectively, wherein the scheduling instruction is used to make the multiple target sub-jobs preempt the running positions in the corresponding execution queues. The purpose that the multiple sub-jobs of the same job can synchronously obtain running resources is achieved, so that the technical effects of improving resource utilization and job running efficiency are realized, and the technical problem that the multiple sub-jobs of the same job are prone to running asynchronously in the traditional single-cluster preemption method is solved.

[0086] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0087] The embodiment of the present application further provides a non-volatile storage medium. Optionally, in the present embodiment, the non-volatile storage medium can be used to save the program codes executed by the multi-queue job scheduling method provided by the above-mentioned embodiments.

[0088] Optionally, in the present embodiment, the non-volatile storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0089] Optionally, in the embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining running information of each sub-job located in the plurality of execution queues, wherein the plurality of sub-jobs located in different execution queues respectively belong to one complete to-be-executed job, the sub-job comprises one or more to-be-executed tasks, and the running information comprises priority information and resource running information; judging whether the plurality of target sub-jobs belonging to the same to-be-executed job all satisfy preemption conditions based on the running information, wherein the preemption conditions comprise priority conditions and resource demand conditions; and issuing a scheduling instruction to the controller corresponding to each of the plurality of execution queues in the case that the plurality of target sub-jobs all satisfy the preemption conditions, wherein the scheduling instruction is used to make the plurality of target sub-jobs preempt the running positions in the corresponding execution queues.

[0090] Optionally, in the embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: reading running information of each sub-job located in the plurality of execution queues, comprising: reading the running information uploaded by the local controller corresponding to each of the plurality of execution queues from the central database.

[0091] Optionally, in the embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: judging whether the plurality of target sub-jobs belonging to the same to-be-executed job all satisfy preemption conditions based on the running information, comprising: comparing the local priority size relationship between the target sub-job and a running sub-job in each of the plurality of execution queues, wherein the running sub-job is a sub-job located in a running position and being executed; and determining that the target sub-job satisfies the priority condition in the case that the local priority of the target sub-job is greater than the local priority of the running sub-job.

[0092] Optionally, in the embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: comparing the local priority size relationship between the target sub-job and the running sub-job in each of the plurality of execution queues, comprising: respectively obtaining the global priority of the target sub-job and the global priority of the running sub-job; respectively determining the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job based on the global priority; and comparing the size relationship between the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job.

[0093] Optionally, in the embodiment, the nonvolatile storage medium is configured to store program code for performing the following steps: determining, based on the running information, whether the target sub-jobs belonging to the same to-be-executed job all satisfy the preemption condition, including: obtaining, in the multiple execution queues, resource usage of running sub-jobs and resource requirement of the target sub-job respectively, wherein the running sub-jobs are sub-jobs located in the running position and being executed; and determining that the target sub-job satisfies the resource requirement condition under the condition that the resource usage is not less than the resource requirement.

[0094] Optionally, in the embodiment, the nonvolatile storage medium is configured to store program code for performing the following steps: in the case that the multiple target sub-jobs all satisfy the preemption condition, issuing a release instruction to the multiple execution queues, wherein the release instruction is used to make the pre-empted running sub-jobs and other sub-jobs belonging to the same to-be-executed job as the running sub-jobs release the occupied running resources, and the running sub-jobs are sub-jobs located in the running position and being executed.

[0095] The embodiment of the present application also provides a computer program product, including a computer program, and optionally, in the embodiment, the computer program can realize the following when executed by a processor: obtaining running information of each sub-job located in multiple execution queues, wherein multiple sub-jobs located in different execution queues respectively belong to one complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the running information includes priority information and resource running information; determining, based on the running information, whether multiple target sub-jobs belonging to the same to-be-executed job all satisfy a preemption condition, wherein the preemption condition includes a priority condition and a resource requirement condition; and in the case that the multiple target sub-jobs all satisfy the preemption condition, issuing a scheduling instruction to a controller corresponding to each of the multiple execution queues, wherein the scheduling instruction is used to make the multiple target sub-jobs preempt to a running position in the corresponding execution queue.

[0096] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0097] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0098] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other manners. For example, the described unit embodiments can be divided into other ways, for example, the division of units can be a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between units can be indirect coupling or communication connection through some interfaces, and can be electrical or other forms.

[0099] The units described as separate components may or can not be physically separate, and the components shown as units may or can not be physical units, i.e., they can be located in one place or distributed on multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0100] In addition, the functional units in each embodiment of the present application can be integrated into a processing unit, or each unit can be physically present alone, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0101] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solutions of the present application or the essential part or all or part of the technical solutions that contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0102] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for multi-queue job scheduling, the method comprising: The method comprises: obtaining running information of each sub-job in a plurality of execution queues, wherein the plurality of sub-jobs in different execution queues belong to a complete to-be-executed job, the sub-job comprises one or more to-be-executed tasks, and the running information comprises priority information and resource running information; judging whether a plurality of target sub-jobs belonging to the same to-be-executed job all meet preemption conditions based on the running information, wherein the preemption conditions comprise a priority condition and a resource demand condition; in the case that the plurality of target sub-jobs all meet the preemption conditions, issuing a scheduling instruction to a controller corresponding to each of the plurality of execution queues, wherein the scheduling instruction is used to make the plurality of target sub-jobs preempt a running position in the corresponding execution queue.

2. The method according to claim 1, characterized in that, The obtaining of the running information of each sub-job in the plurality of execution queues comprises: reading the running information uploaded by a local controller corresponding to each of the plurality of execution queues from a central database.

3. The method according to claim 1, characterized in that, The judging whether the plurality of target sub-jobs belonging to the same to-be-executed job all meet the preemption conditions based on the running information comprises: comparing a local priority size relationship between the target sub-job and a running sub-job in each of the plurality of execution queues, wherein the running sub-job is a sub-job in the running position and being executed; in the case that the local priority of the target sub-job is greater than the local priority of the running sub-job, determining that the target sub-job meets the priority condition.

4. The method according to claim 3, characterized in that, The comparing of the local priority size relationship between the target sub-job and the running sub-job in each of the plurality of execution queues comprises: obtaining a global priority of the target sub-job and a global priority of the running sub-job respectively; determining a local priority corresponding to the target sub-job and a local priority corresponding to the running sub-job based on the global priority respectively; comparing a size relationship between the local priority corresponding to the target sub-job and the local priority corresponding to the running sub-job.

5. The method of claim 1, characterized in that, The judging whether the plurality of target sub-jobs belonging to the same to-be-executed job all meet the preemption conditions based on the running information comprises: obtaining a resource usage of a running sub-job and a resource demand of the target sub-job in each of the plurality of execution queues, wherein the running sub-job is a sub-job in the running position and being executed; in the case that the resource usage is not less than the resource demand, determining that the target sub-job meets the resource demand condition.

6. The method of claim 1, characterized in that, The method further comprises: in the case that the plurality of target sub-jobs all meet the preemption conditions, issuing a release instruction to the plurality of execution queues, wherein the release instruction is used to make a preempted running sub-job and other sub-jobs belonging to the same to-be-executed job as the running sub-job all release occupied running resources, and the running sub-job is a sub-job in the running position and being executed.

7. A multi-queue job scheduling method, characterized by, The method comprises: Obtaining running information of a plurality of sub-jobs in a local execution queue, wherein the sub-jobs comprise one or more to-be-executed tasks, and the running information comprises priority information and resource running information; Sending the running information to a central controller; Receiving a scheduling instruction sent by the central controller, wherein the scheduling instruction is determined by the central controller based on the running information; Based on the scheduling instruction, scheduling a target sub-job to a running position in the execution queue, wherein the plurality of sub-jobs comprises the target sub-job.

8. The method according to claim 7, characterized in that, Further comprising: Saving a running state of a preempted running sub-job in a local database, wherein the running sub-job is a sub-job located in the running position and being executed, and the running state comprises running parameters and data processing progress.

9. The method according to claim 8, characterized in that, Further comprising: In the case that the running sub-job returns to the running position, resuming execution of the running sub-job based on the running state saved in the local database.

10. A multi-queue job scheduling system, characterized by, Comprise: A central controller for executing the multi-queue job scheduling method of any one of claims 1 to 6; A plurality of local controllers for scheduling sub-jobs in a local execution queue based on scheduling instructions issued by the central controller.

11. A non-volatile storage medium, comprising: A non-volatile storage medium comprises a stored program, wherein when the program is running, the device where the non-volatile storage medium is located executes the multi-queue job scheduling method of any one of claims 1 to 6.

12. A computer device, comprising: Comprise: A memory and a processor, The memory stores a computer program; The processor is configured to execute the computer program stored in the memory, and the computer program makes the processor execute the multi-queue job scheduling method of any one of claims 1 to 6 when running.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the multi-queue job scheduling method of any one of claims 1 to 6.

Images