Multi-queue job processing method and system and nonvolatile storage medium

Abstract

The invention discloses a multi-queue job processing method and system and a nonvolatile storage medium. The method comprises the steps that queue state information of all sub-jobs located in multiple job queues is obtained, the sub-jobs located in the different job queues belong to a complete job to be executed, each sub-job comprises one or more tasks to be executed, and the queue state information comprises an initial queuing sequence and resource information; based on the queue state information, respectively determining a target queuing sequence of each sub-job in the plurality of job queues; based on the target queuing sequence, a sorting instruction is issued to the controllers corresponding to the multiple job queues, and the sorting instruction is used for sorting the multiple sub-jobs according to the target queuing sequence. According to the method and the device, the technical problem of resource interlocking among multiple queues caused by the fact that the sub-jobs of the multiple queues cannot be queued cooperatively is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer task scheduling, in particular to a multi-queue job processing method and system and a nonvolatile storage medium. BACKGROUND

[0002] With the increasing demand for computing power, it is difficult for a single data center to meet the increasing demand for computing power. A new method is proposed in the industry, which is to jointly use multi-party computing power resources to carry large jobs. Specifically, work is no longer limited to a single intelligent computing center, but is cleverly divided into multiple sub-jobs, which are executed in parallel in multiple intelligent computing centers distributed in different geographical locations. However, this raises a new problem: how to ensure that these sub-jobs can be synchronized to obtain scheduling in their respective computing power centers, thereby efficiently utilizing dispersed resources and avoiding resource idling and mutual locking.

[0003] The traditional solution is to build a multi-data center network managed by a single operating entity, which provides unified job queue services to the outside through a centralized platform, simplifying the complexity of cross-cluster scheduling. However, this method does not address the core contradiction, that is, when a training job needs to be coordinated across multiple heterogeneous job queues, how to ensure the synchronization of each sub-job and the effective allocation of resources.

[0004] At present, there is no effective solution to the above problems. SUMMARY

[0005] The embodiments of the present application provide a multi-queue job processing method, system and nonvolatile storage medium to at least solve the technical problem that the sub-jobs of multiple queues cannot be cooperatively queued, resulting in resource locking between multiple queues.

[0006] According to an aspect of an embodiment of the present application, a multi-queue job processing method is provided, comprising: obtaining queue state information of each sub-job located in multiple job queues, wherein the multiple sub-jobs located in different job queues belong to a complete job to be executed, the sub-job includes one or more tasks to be executed, and the queue state information includes initial queuing order and resource information; based on the queue state information, determining a target queuing order of each sub-job in the multiple job queues; and based on the target queuing order, issuing a sorting instruction to a controller corresponding to each of the multiple job queues, wherein the sorting instruction is used to sort the multiple sub-jobs according to the target queuing order.

[0007] Optionally, obtaining the queue state information of each sub-job located in the multiple job queues comprises: reading the queue state information uploaded by the local controller corresponding to each of the multiple job queues from a central database.

[0008] Optionally, based on the queue state information, the target queuing order of each sub-job in the plurality of job queues is determined respectively, including: obtaining an initial global order of the plurality of to-be-executed jobs; based on the queue state information, adjusting the initial global order according to a preset global strategy target, to obtain a target global order of the plurality of to-be-executed jobs; based on the target global order, adjusting an initial queuing order of each sub-job in the plurality of job queues to obtain the target queuing order.

[0009] Optionally, based on the queue state information, the target global order of the plurality of to-be-executed jobs is obtained by adjusting the initial global order according to a preset global strategy target, including: performing permutation and combination on the plurality of to-be-executed jobs to obtain a plurality of original global orders; constructing a first scoring function with the global strategy target as the target, wherein the first scoring function is used to quantify the matching degree between the original global order and the global strategy target; based on the first scoring function, scoring the plurality of original global orders respectively to obtain a first scoring result corresponding to each of the plurality of original global orders; based on the first scoring result, adjusting the initial global order to obtain the target global order.

[0010] According to another aspect of the embodiment of the application, a multi-queue job processing method is also provided, including: obtaining queue state information of a plurality of sub-jobs in a local job queue, wherein the sub-job includes one or more to-be-executed tasks, and the queue state information includes an initial queuing order and resource information; sending the queue state information to a central controller; receiving an ordering instruction sent by the central controller, wherein the ordering instruction includes a target queuing order, and the target queuing order is determined by the central controller based on the queue state information; and based on the ordering instruction, ordering the plurality of sub-jobs according to the target queuing order.

[0011] Optionally, obtaining the queue state information of the plurality of sub-jobs in the job queue includes: obtaining queue information of the job queue, job information of each of the plurality of sub-jobs, and an overall scheduling strategy, wherein the queue information includes a queue priority and a queue admission rule, the job information includes a sub-job priority and a rule label, and the overall scheduling strategy is a preset rule for scheduling the plurality of job queues and corresponding sub-jobs; based on the queue information, the job information, and the overall scheduling strategy, determining the initial queuing order; and based on the initial queuing order, determining the queue state information.

[0012] Optionally, based on the queue information, the job information and the overall scheduling strategy, the initial queuing order is determined, including: determining a plurality of sub-jobs of the job queue; dividing the plurality of sub-jobs into a plurality of sets based on queue priorities and queue admission rules, wherein all sub-jobs in one set have the same queue priority range and the same admission rule; for each set, performing permutation and combination on all sub-jobs in the set to obtain a plurality of potential ordering results corresponding to the plurality of sets respectively; constructing a second scoring function taking the sub-task priority, the rule label and the overall scheduling strategy as the standard based on a preset weight, wherein the second scoring function is used to quantify the comprehensive matching degree of the potential ordering result and the target sub-task priority, the target rule label and the target overall scheduling strategy; based on the second scoring function, for each set, the plurality of potential ordering results are scored respectively to obtain a second scoring result corresponding to the plurality of potential ordering results respectively; based on the second scoring result, a target ordering corresponding to the plurality of sets is determined; and the plurality of sets are merged according to the target ordering to obtain the initial queuing order.

[0013] According to another aspect of the embodiments of the present application, a multi-queue job processing system is further provided, including: a central controller configured to execute any one of the multi-queue job processing methods described above; and a plurality of local controllers configured to sort sub-jobs in a local job queue based on a sorting instruction issued by the central controller.

[0014] According to still another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, including a stored program, wherein when the program is running, the non-volatile storage medium controls a device in which the non-volatile storage medium is located to execute any one of the multi-queue job processing methods described above.

[0015] According to yet another aspect of the embodiments of the present application, a computer device is further provided, including a processor configured to run a program, wherein when the program is running, the processor executes any one of the multi-queue job processing methods described above.

[0016] According to still another aspect of the embodiments of the present application, a computer program product is further provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the multi-queue job processing methods described above.

[0017] In the embodiment of the present application, a multi-queue job processing method is adopted, queue state information of each sub-job located in multiple job queues is acquired, wherein the multiple sub-jobs located in different job queues respectively belong to a complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the queue state information includes initial queuing order and resource information; based on the queue state information, target queuing orders of the sub-jobs in the multiple job queues are respectively determined; and based on the target queuing orders, a sorting instruction is issued to a controller corresponding to each of the multiple job queues, wherein the sorting instruction is used to sort the multiple sub-jobs according to the target queuing orders, so that the purpose of intelligently adjusting the arrangement orders of the sub-jobs in different queues is achieved, thereby realizing the technical effects of improving resource utilization efficiency and job execution speed, and further solving the technical problem of resource interlocking between multiple queues caused by the uncooperative queuing of sub-jobs in the multiple queues. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a computer terminal for implementing a multi-queue job processing method is shown;

[0020] Figure 2 A flowchart of a multi-queue job processing method according to an embodiment of the present application is shown;

[0021] Figure 3 A diagram of division of jobs in multiple platforms according to an optional embodiment of the present application is shown;

[0022] Figure 4 A diagram of adjustment of jobs in multiple platforms according to an optional embodiment of the present application is shown;

[0023] Figure 5 A flowchart of coordination of heterogeneous multi-queues according to an optional embodiment of the present application is shown;

[0024] Figure 6 A diagram of job division according to an optional embodiment of the present application is shown;

[0025] Figure 7 A basic architecture diagram of a cross-multi-queue coordination system according to an optional embodiment of the present application is shown;

[0026] Figure 8 A detailed architecture diagram of a cross-multi-queue coordination system according to an optional embodiment of the present application is shown;

[0027] Figure 9It is a whole architecture diagram of a cross multi-queue coordination system according to an optional embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiment of the present application will be clearly and completely described below in combination with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, not all. Based on the embodiment in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the protection scope of the present application.

[0029] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] According to an embodiment of the present application, a multi-queue job processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0031] The method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing a multi-queue job processing method is shown. As shown in Figure 1 The computer terminal 10 can include one or more processors (the processor can include but not limited to a microprocessor MCU or a programmable logic device FPGA processing device) (shown in 102a, 102b,..., 102n in the figure), a memory 104 for storing data. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1The illustrated structure is merely schematic and does not limit the structure of the electronic device described above. For example, the computer terminal 10 can further include more or fewer components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1. Figure 1 Figure 1 The illustrated structure is merely schematic and does not limit the structure of the electronic device described above. For example, the computer terminal 10 can further include more or fewer components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1.

[0032] It should be noted that the one or more processors and / or other data processing circuitry described above can be referred to herein generally as "data processing circuitry". The data processing circuitry can be embodied in whole or in part as software, hardware, firmware, or any combination thereof. Furthermore, the data processing circuitry can be a single independent processing module, or incorporated in whole or in part within any one of the other elements of the computer terminal 10. As referred to in embodiments of the present application, the data processing circuitry functions as a processor to control, for example, selection of the variable resistance terminal path connected to the interface.

[0033] The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the multi-queue job processing method of embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, i.e., implements the multi-queue job processing method of the application program described above. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include a memory disposed remotely with respect to the processor, which can be connected to the computer terminal 10 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The display can be, for example, a touch screen type liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10.

[0035] Figure 2 is a flowchart of the multi-queue job processing method according to embodiments of the present application, as shown in FIG. 2, the method includes the following steps: Figure 2

[0036] In step S201, queue state information of each sub-job located in a plurality of job queues is obtained, wherein the plurality of sub-jobs located in different job queues respectively belong to one complete job to be executed, the sub-job includes one or more tasks to be executed, and the queue state information includes initial queuing order and resource information.

[0037] ​​In this step, in the multi-queue synchronization queuing mechanism, a to-be-executed job, such as a large model training job, is subdivided into multiple sub-jobs, which are respectively entered into the job queue of different intelligent computing centers to wait for scheduling, and each job can be a set of one or more to-be-executed tasks. Specifically, taking a large model training job as an example, Figure 3 is a schematic diagram of multi-platform queue job division provided according to an optional embodiment of the present application, as shown in Figure 3 The cross-cluster scheduler splits the large model according to the demand of the large model training job and the resource state of each cluster / platform, and feeds back the cross-domain solution of the large model to the collaborative scheduling. After the collaborative scheduling feeds back the selected solution, the cross-domain deployment solution is distributed to the cluster / platform. After each cluster / platform receives the deployment demand, the demand is added to its job queue, and then the job scheduling is completed by the cluster / platform according to the scheduling strategy of the queue. The large model training job will be dispersed in the queue of each cluster / platform. Each queue independently schedules the job, and the time opportunity for the sub-jobs of the same training business to obtain scheduling and resource allocation is inconsistent, which may cause resource waste and affect the efficiency of large model training, or even cause job interlocking and starvation and resource idling. Therefore, a cross-domain multi-task queue coordinator needs to be constructed to coordinate the job scheduling among multiple job queues and realize the synchronous running of cross-domain jobs.

[0038] The cross-domain multi-queue coordinator is part of the load controller of the cross-domain large model training job, and its main function is to obtain the state of the large model training job dispersed in multiple cluster / platform queues, and to ensure that the sub-jobs belonging to the same training job can synchronously obtain the required resources. Figure 4 is a schematic diagram of multi-platform queue job adjustment provided according to an optional embodiment of the present application, as shown in Figure 4 The target of the cross-domain multi-queue coordinator is to adjust each sub-part of the large model training job dispersed in multiple queues to a suitable position, to ensure that the sub-jobs of each job can synchronously obtain resource allocation and avoid resource interlocking.

[0039] In order to realize the above-mentioned process of collaborative adjustment of multiple queues, queue state information needs to be obtained from these heterogeneous queues, including the initial queuing order of the sub-jobs and the resource information of each sub-job. The initial queuing order reflects the initial position of the sub-job in each heterogeneous queue, that is, the list of <job ID, queuing position> binary tuples. The resource information can include resource usage, resource availability and resource guarantee demand, etc. The resource demand is the resource amount applied by the sub-job to each heterogeneous queue, which indicates the required resource types and quantities such as GPU and CPU, and the resource availability is the current available resource amount of the queue. The resource guarantee demand can be determined according to the deadline of the job, and the job with deadline needs to be prioritized for resource allocation.

[0040] Step S202, based on the queue state information, respectively determine the target queuing order of each sub-job in the plurality of job queues.

[0041] In this step, when a plurality of sub-jobs collectively constitute a complete to-be-executed job, for example, a large model training job is distributed to the job queues of different intelligent computing centers, the task agent will continuously monitor and report the real-time state information of each queue. These information includes but not limited to the initial queuing position of the sub-job in the queue, the specific resource amount required by each sub-job (such as the number of GPUs, the size of memory, etc.) and the current resource available amount of each queue. After receiving these queue state information, a decision algorithm will be started, which will analyze the difference between the current queue state and the target state according to the preset global strategy target, and then calculate the target queuing order of each sub-job. This target queuing order aims to ensure that all sub-jobs can obtain resource allocation in the most optimized order, thereby improving the efficiency and resource utilization of the entire job execution.

[0042] Step S203, based on the target queuing order, issue ordering instructions to the controllers corresponding to the plurality of job queues respectively, wherein the ordering instructions are used to sort the plurality of sub-jobs according to the target queuing order.

[0043] In this step, after determining the target queuing order of each sub-job based on the queue state information, the next task is to translate this result into actual action, that is, by issuing ordering instructions to the controllers of each heterogeneous queue to guide them to adjust the queuing order of the sub-jobs in the queue. For example, taking large model training across multiple clusters as an example, Figure 5 is a flowchart of heterogeneous multi-queue coordination provided according to an optional embodiment of the present application, as shown in Figure 5 The flow of heterogeneous multi-queue coordination is divided into three parts: periodic reporting of job queuing state, task agent real-time acquires job queuing state on the platform where it is located through watch-list monitoring list or observer list, and reports to task scheduling; update the expected queuing state, the heterogeneous job queue coordinator in the task scheduling, judges whether to adjust the expected queuing state according to the job queuing status; adjust the queue according to the expected queuing state, the task agent listens to the change of the expected job queuing state, and adjusts the job queue of the platform where it is located according to the expected job queuing.

[0044] Through the above steps, the purpose of intelligently adjusting the arrangement order of each sub-job in different queues is achieved, thereby realizing the technical effect of improving resource utilization efficiency and job execution speed, and further solving the technical problem of resource interlocking between multiple queues caused by the uncoordinated queuing of sub-jobs in multiple queues.

[0045] As an optional embodiment, the queue state information of each sub-job located in the plurality of job queues is obtained, including: reading the queue state information uploaded by the local controller corresponding to each of the plurality of job queues from the central database.

[0046] Optionally, the queue state periodically uploaded by the local queue controller of each intelligent computing center can be read from the central database of the global management layer, such as ETCD. This state information covers the position of the sub-job in the queue, the details of the resource requirement, and the real-time resource availability of the queue, and is the key data for the coordinator to calculate the optimization strategy and determine the target queuing order of the sub-job. Through this information collection point of the central database, the coordinator can master the dynamic situation of each queue in the global, and then accurately schedule and adjust to ensure the efficient and synchronous execution of the sub-job in the multi-queue environment, and achieve the goal of improving resource utilization and job throughput.

[0047] As an optional embodiment, based on the queue state information, the target queuing order of each sub-job in the plurality of job queues is determined, including: obtaining an initial global order of a plurality of to-be-executed jobs; based on the queue state information, adjusting the initial global order according to a preset global strategy target to obtain a target global order of the plurality of to-be-executed jobs; and based on the target global order, adjusting an initial queuing order of each sub-job in the plurality of job queues to obtain the target queuing order.

[0048] Optionally, the initial global queue receives and stores a plurality of jobs to be executed according to certain rules, usually the time sequence of job submission, and obtains an initial global order. These jobs are then decomposed into smaller sub-jobs, each of which will enter the job queue of a different intelligent computing center that matches its resource requirements, forming their own initial queuing order. Real-time state information of all job queues is read from the central database, which includes not only the initial position of the sub-job in the queue, but also rich queue attributes such as priority, resource availability, and characteristics of the sub-job such as resource requirements, estimated running time, etc. Based on this information, the global job order can be re-evaluated according to the preset global strategy target, such as improving resource utilization, enhancing job throughput, or ensuring timely completion of jobs (deadline constraint), to calculate an optimized target global order. After the target global order is determined, this optimization order information can be issued to each queue. The controller of each queue checks and compares the difference between the current queuing order of the sub-job in the queue of the intelligent computing center it is located in and the global target queuing order, adjusts the position of the sub-job in the queue, and thus ensures that all sub-jobs can be scheduled and executed according to the target queuing order.

[0049] As an optional embodiment, based on the queue state information, the initial global order is adjusted according to a preset global strategy target to obtain a target global order of the plurality of to-be-executed jobs, including: performing permutation and combination on the plurality of to-be-executed jobs to obtain a plurality of original global orders; constructing a first scoring function with the global strategy target as the target, wherein the first scoring function is used to quantify the matching degree of the original global order and the global strategy target; based on the first scoring function, the plurality of original global orders are scored respectively to obtain a first scoring result corresponding to each of the plurality of original global orders; and based on the first scoring result, the initial global order is adjusted to obtain the target global order.

[0050] Optionally, the plurality of to-be-executed jobs can be preliminarily permuted and combined to generate a series of original global orders. These original global orders reflect a plurality of potential queuing manners of different jobs. Then, a first scoring function is constructed, which takes the global strategy target as the core and quantifies the matching degree of each original global order with the target, expressing the advantages and disadvantages of each original global order in numerical form. The first scoring function can be constructed for individual targets, such as resource utilization, job throughput or job deadline, and key performance indicators, or can be considered comprehensively, with weights set for each individual scoring function to obtain a multi-dimensional first scoring function. Subsequently, the first scoring function is applied to evaluate each original global order to generate a corresponding first scoring result, and the queue order configuration that best fits the global strategy target is found. Finally, based on the first scoring results of all original queue orders, the initial job queue order is adjusted, for example, the initial global order is adjusted to the original global order with the maximum scoring result, to achieve the optimal global strategy target.

[0051] Specifically, taking a large model training job as an example, Figure 6 is a schematic diagram of job division according to an optional embodiment of the present application, as Figure 6 shown, there are a total of 4 jobs, Job1- Job4. There are 3 heterogeneous queues, queue1- queue3. A global queue Global Queue is maintained in the management layer. Initially, jobs can be queued according to the order of job submission, and after scheduling by the management layer, the respective situations of the jobs are: Job1 is allocated to queue1; Job2 is split into Part1 and Part2 and allocated to queue1 and queue2; Job3 is split into Part1 and Part2 and allocated to queue2 and queue3; Job4 is allocated to queue4.

[0052] The job queue state adjustment is essentially based on the queue order of each sub-part of the jobs in each heterogeneous queue, the resource demand of each sub-part, and the available resource amount of each heterogeneous queue, to configure the queue order of the global queue (i.e., the expected job queue state) according to a certain strategy target. The task agent of each heterogeneous platform listens to the state of the global queue, and when the global queue state changes, it determines whether the queue order of the heterogeneous queue where it is located is consistent with the queue order of the global queue, and adjusts each heterogeneous queue according to the global queue. Therefore, determining the queue order of the global queue is the premise of job queue state adjustment, and configuring the global queue according to a certain strategy target is not limited, and different strategies produce different job queue order results, but the adjustment logic flow is consistent.

[0053] For example, the resource available amount of each heterogeneous queue and the demand amount of the job are as follows: the current available resource amount of the heterogeneous queues queue1, queue2, and queue3 is 6, 6, and 6 respectively; the resource demand amount of the job Job1 is 8; the resource demand amount of the job Job2 is 5 for Part1 and 5 for Part2; the resource demand amount of the job Job3 is 6 for Part1, 6 for Part2, and 6 for Part2; and the resource demand amount of the job Job4 is 4.

[0054] For example, if the goal is to improve the comprehensive resource utilization rate, the order of Job1, Job2, Job3, and Job4 needs to be adjusted to Job3, Job2, Job4, and Job1, so that the comprehensive resource utilization rate of each heterogeneous queue is the highest. If the goal is to improve the comprehensive job throughput, the order of Job1, Job2, Job3, and Job4 needs to be adjusted to Job4, Job2, Job3, and Job1, so that the comprehensive job throughput of each heterogeneous queue is the highest. If the goal is to guarantee the deadline of the job, if there is a job with a strict deadline, the resource of the job needs to be prioritized, for example, Job1 has a strict deadline, and the order of Job1, Job2, Job3, and Job4 needs to be adjusted to Job1, Job2, Job4, and Job3, so that Job4 can obtain the resource while Job1 is prioritized to obtain the resource, and Job2 and Job3 continue to queue due to the resource blocking of Job1.

[0055] Specifically, taking improving comprehensive resource utilization as an example, the initial global order is Job1, Job2, Job3, Job4, at this time, the jobs are all blocked. The comprehensive utilization is the percentage of resource usage to total resource, which can also be understood as the minimum idle resource. First, a candidate original global order set can be generated. For example, all existing queue orders can be generated by permutation and combination of Job1, Job2, Job3, Job4, or other strategies can be designed to generate the set. Then a corresponding scoring function can be constructed, and various original global orders are scored by the scoring function. In this optional embodiment, the score is scored according to the influence of the original global order on the resource utilization. Generally, the higher the resource utilization, that is, the more matching the global strategy target of improving the comprehensive resource utilization, the higher the score, and the specific scoring calculation rule can be flexibly designed. For example, the total resource utilization can be multiplied by 100 as the score. Assuming that the generated original global order is order1={Job1, Job2, Job3, Job4}, order2={Job3, Job2, Job4, Job1}. All jobs are blocked under order1, the resource utilization is 0%, that is, the score is 0. Under order2, Job3 can run, the resource utilization is 100%, that is, the score is 100. After Job3 runs, Job2 can run, and according to the scoring rule, the score is the highest. Finally, the original global order with the highest score is selected as the target global order, that is, order2 has the highest score.

[0056] Similarly, if the target is to improve the comprehensive job throughput, the difference lies in the rule of the scoring function, that is, a scoring function for the comprehensive job throughput is constructed, and the higher the throughput, the higher the score. If a scoring function is constructed for the job deadline, only whether the job with a deadline can guarantee limited resource allocation needs to be considered. If multiple factors are considered, multiple types of scoring functions can be constructed, and weights of each scoring function are set to obtain a multi-dimensional scoring function.

[0057] According to the embodiment of the application, a multi-queue job processing method is also provided, comprising: acquiring queue state information of a plurality of sub-jobs in a local job queue, wherein the sub-job comprises one or more to-be-executed tasks, and the queue state information comprises an initial queue order and resource information; sending the queue state information to a central controller; receiving an ordering instruction sent by the central controller, wherein the ordering instruction comprises a target queue order, and the target queue order is determined by the central controller based on the queue state information; and ordering the plurality of sub-jobs according to the target queue order based on the ordering instruction.

[0058] Optionally, the local controller first actively collects and masters the real-time status of the plurality of sub-jobs in the queue it is responsible for managing, which can include the initial queuing order of the sub-jobs in the queue (i.e. the original order when the sub-jobs are assigned to the queue), the specific amount of required resources (such as the number of GPUs, computing units, storage requirements, etc.), and the current available amount of resources of the queue (i.e. the amount of resources in the queue that can be immediately allocated to sub-jobs), etc., and then sends it to the central controller for analysis and processing. Then the local controller can receive the sorting instruction from the central controller of the global management layer, which contains the target queuing order calculated by the heterogeneous queue coordinator according to the global strategy target and all queue state information. Based on the received sorting instruction, the local controller will adjust the order of the sub-jobs in the queue it manages to ensure that each sub-job can be queued according to the target queuing order. This adjustment may involve resetting the priority of the sub-jobs, reserving resources, or applying specific scheduling algorithms, etc. to ensure that the queuing order of the sub-jobs is consistent with the global optimization strategy, thereby achieving effective management and optimization of the local queue and providing necessary resource support for the smooth execution of the sub-jobs.

[0059] As an optional embodiment, the queue state information of the plurality of sub-jobs in the job queue is obtained, including: obtaining queue information of the job queue, job information of each of the plurality of sub-jobs, and an overall scheduling strategy, wherein the queue information includes queue priority and queue admission rules, the job information includes sub-job priority and rule label, and the overall scheduling strategy is a preset rule for scheduling the plurality of job queues and corresponding sub-jobs; based on the queue information, the job information, and the overall scheduling strategy, an initial queuing order is determined; and based on the initial queuing order, the queue state information is determined.

[0060] Optionally, the local controller collects and analyzes the queue information of the local job queue, including: queue priority, the higher the numerical value, the higher the priority of the entire queue, the job is preferentially scheduled, equivalent to "cutting in line" before all low-priority queues; queue admission rules, if the job does not meet the ACL / QoS rules, it will be delayed or transferred to the secondary queue, resulting in additional queuing delay; queue weight, the scheduler weights according to the weight when allocating resources, the jobs in the queue with high weight have shorter average waiting time and relatively forward queuing order; queue current depth, the deeper the queue, the later the new job is queued; the scheduler may also enable depth limit, so that new jobs are rejected or transferred to other queues when the queue is full; queue maximum parallelism, if the queue parallelism is full, even if the job priority is high, it must continue to queue; the higher the upper limit of parallelism, the faster the job can leave the queue, indirectly shortening the queuing time; queue resource reservation, when the reserved resources are insufficient, new jobs will be queued in the sub-queue waiting for reserved resources, delaying the overall queuing order; queue time window, only within the window period, the job arriving outside the window needs to wait for the next window, and the queuing order is "frozen".

[0061] At the same time, it also acquires the job information of each sub-job, including: sub-job priority, the higher the numerical value, the job is in the same queue in front; most of the scheduler uses absolute priority queue or weighted priority queue to achieve; rule label, high QoS label corresponds to higher service level, equivalent to additional priority; low QoS job may be delayed even if the original priority is high; submission timestamp, when the priority is the same, the job submitted first is in front (FIFO); also used to calculate the waiting time to do aging promotion; resource request amount, the more the request, the more difficult to meet, may be "suspended" to the end of the queue until enough resources appear; also affect the candidate qualifications when backfill scheduling; estimated running time, short job priority (SF / Backfill) strategy, short job can pass long job in front; queue sorting may be in ascending order of walltime; user / group quota, when exceeding the quota, the job is delayed or put into the secondary queue; the quota is sufficient, then the job is queued according to the priority; dependency relationship, the job that the dependency is not satisfied remains "blocked" in the queue until the previous job is completed, and the actual queue order is adjusted dynamically; license / special resource, when the license is not available, the job is in the queue, but it is considered "unstartable", and the queue order is frozen until the license is available; job preemption flag, pre-emptible job will be "yielded" by the scheduler when resources are scarce, and re-queued; non-preemptible job remains in the original queue position; fair share, the scheduler calculates the fairshare factor according to the historical usage, the user job priority of the user with high factor decreases, and the queue order moves backward.

[0062] In addition, the local controller also follows the overall scheduling strategy, including: scheduling algorithm type, FIFO strictly orders by submission time, Backfill allows short jobs to be inserted in advance, Fair-share dynamically adjusts the virtual priority of each user job; preemption strategy, preemptive scheduling can directly promote high-priority jobs to running state, low-priority jobs are put back to the head or tail of the queue, changing the queuing order; backfill depth, the greater the depth, the more jobs that can be backfilled, and short jobs are more likely to be advanced; when the depth is 0, it degenerates to strict priority ordering; resource selection strategy, if the "best fit" or "worst fit" strategy is used, the degree of resource fragmentation varies, resulting in changes in the time point at which the job meets the conditions, thereby changing the queuing order; time slice rotation, round-robin scheduling switches between jobs, a short time slice makes the jobs in the queue appear to "advance equally"; a long time slice allows the job at the front of the queue to be completed first, and the subsequent jobs are shifted as a whole; multi-queue linkage strategy, cross-queue preemption or job migration strategy between queues, which "pulls" high-priority jobs from low-priority queues to high-priority queues, directly changing the queuing order; aging mechanism, the priority of jobs waiting for a long time is gradually increased, eventually exceeding newly submitted high-priority jobs, achieving dynamic adjustment of the queuing order; resource reservation strategy, the reservation strategy locks future resources for specific jobs, allowing these jobs to "insert in advance" before the reservation time point, and other jobs need to give way even if they arrive first.

[0063] Based on the collected queue information, job information and overall scheduling strategy, the local controller determines the initial queuing order of each sub-job in the local queue through algorithm calculation. This ordering process fully considers the priority and admission rules of the queue, as well as the priority and rule label of the sub-job, ensuring that the sub-job is reasonably queued in the local queue, both following the global scheduling strategy and reflecting the characteristics and resource status of the local queue. Subsequently, based on the determined initial queuing order, the local controller further determines the queue state information. This state information not only contains the position of the sub-job in the queue, but also records in detail the resource requirements of the sub-job, the resource availability of the queue, etc., providing detailed data basis for subsequent global scheduling optimization.

[0064] As an optional embodiment, based on the queue information, the job information and the overall scheduling strategy, the initial queuing order is determined, including: determining a plurality of sub-jobs of the job queue; dividing the plurality of sub-jobs into a plurality of sets based on the queue priority and the queue admission rule, wherein the queue priorities of all sub-jobs in one set are in the same priority range, and the admission rules of all sub-jobs in one set are the same; for each set, permutation and combination are performed on all sub-jobs in the set to obtain a plurality of potential ordering results corresponding to the plurality of sets respectively; based on a preset weight, a second scoring function is constructed taking the sub-task priority, the rule label and the overall scheduling strategy as the standard, wherein the second scoring function is used to quantify the influence of the potential ordering result on the target sub-task priority, the target rule label and the target overall scheduling strategy; based on the second scoring function, for each set, the plurality of potential ordering results are scored respectively to obtain a second scoring result corresponding to the plurality of potential ordering results respectively; based on the second scoring result, a target ordering corresponding to the plurality of sets is determined; the plurality of sets are merged according to the target ordering to obtain the initial queuing order.

[0065] Optionally, the local controller first identifies all sub-jobs to be managed, and then divides them into a plurality of optimization sets according to the queue priority and the admission rule. For example, sub-jobs with the same queue priority or within a range can be divided into a set, and sub-jobs with the same admission rule can also be divided into a set. Within each set, the sub-jobs can be sorted according to the sub-job priority, the rule label and the overall scheduling strategy to produce the optimal sub-sorting result, i.e. to obtain the target ordering. Finally, these target orderings are integrated to form an initial queuing order that adapts to the global strategy, ensuring efficient resource allocation and orderly job execution. This process accurately matches the characteristics of the queue and the job, supporting job coordination and optimization in a multi-intelligent algorithm center environment.

[0066] For example, all jobs in the waiting queue WaitQ can be sorted to get an OrderedQ that the scheduler matches resources against. The logic to determine the queuing order is as follows: first take the current waiting queue WaitQ, i.e. all sub-jobs, and cut WaitQ into several sub-queues (sets) SubQ1, SubQ2, …, SubQn according to queue priority and whether ACL / QoS is the same. Resort each SubQi based on the following criteria: calculate dynamic priority P*; handle dependencies, license blocking; apply scheduling policy (FIFO / Backfill / Fair-share…). Specifically, a weight can be set for each criterion according to the importance, and then a comprehensive second scoring function is constructed according to the weight, which quantitatively represents the matching degree of different sorting in each SubQi and the target level of each criterion, and all possible sorting is scored by the second scoring function to determine the optimal SubQi internal sorting result, i.e. the target sorting. Then merge all SubQi, which can include inter-queue preemption / migration and aging promotion, and finally output the final queuing order OrderedQ, i.e. the initial queuing order. The final queuing order is a list of <job ID, queuing position> tuples, which can mask the differences of the underlying heterogeneous queues and only need to update the necessary queuing order state to the management layer.

[0067] It should be noted that for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0068] From the above description of the embodiments, those skilled in the art can clearly understand that the multi-queue job processing method according to the above embodiments can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions to make a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0069] According to the embodiments of the present application, a multi-queue job processing system is also provided, which comprises: a central controller configured to execute any of the above multi-queue job processing methods; and a plurality of local controllers configured to sort sub-jobs in local job queues based on sorting instructions issued by the central controller.

[0070] Optionally, the system comprises the central controller and the plurality of local controllers. The central controller, as the core of the global strategy, executes the multi-queue job sorting method including determining all queues to be sorted, dividing sub-queues based on priority and rules, sorting sub-jobs, and merging results to generate a target queuing order. It collects the queue state and resource information of each local controller, and calculates the optimal job scheduling order according to the global optimization strategy, such as maximizing resource utilization or guaranteeing deadline, to form a unified sorting instruction. The local controllers are distributed in each intelligent computing center, responsible for receiving the instructions of the central controller, adjusting the queuing order of sub-jobs in the local queue, and ensuring that it is consistent with the global strategy target. This adjustment involves modifying the job priority, resource reservation strategy, etc., so that the local job queue can quickly respond to the scheduling needs of the central controller, and realize efficient resource utilization and orderly job execution.

[0071] As an optional embodiment, a multi-queue coordination system is also provided. Figure 7 As shown in FIG. 1, which is a basic architecture diagram of a multi-queue coordination system according to an optional embodiment of the present application, the working mechanism of the multi-queue coordination system is as follows: Figure 7 Heterogeneous queue state reporting: periodically updating the job queuing state of the heterogeneous queue to the management and control layer; updating the expected queue queuing state: the heterogeneous queue coordinator updates the expected job queuing state according to the current job queuing state; adjusting the queue queuing state: the task agent obtains the updated job queuing state, and calls the queue adjustment interface of the heterogeneous queue to adjust the queue queuing state.

[0072] Specifically, Figure 8 As shown in FIG. 2, which is a detailed architecture diagram of a multi-queue coordination system according to an optional embodiment of the present application, the working mechanism of the multi-queue coordination system is as follows: Figure 8As shown, each module is distributed in a global control layer (global total control platform) and a heterogeneous resource layer (heterogeneous resource platform), and each has a heterogeneous queue coordinator, a task agent, and a heterogeneous queue controller. The heterogeneous queue coordinator can obtain relevant information from the ETCD through the API server (API service interface), and then adjust the queuing order of the global job queue based on the obtained state of each heterogeneous queue (including the available resource amount of each heterogeneous queue and the queuing order and resource requirement of the job subpart in each heterogeneous queue). The task agent can obtain the queuing information from the heterogeneous queue controller to form the queuing order of the job in the current heterogeneous queue, and report the resource available amount of all heterogeneous queues and the resource requirement amount of each job subpart. In addition, the state of the global queue is monitored, and if it is inconsistent, the local queue queuing order is adjusted to be consistent with the global queue order by adjusting three types of index items. The heterogeneous queue controller can provide the task agent with the state information of the local queue. At the same time, it receives the queuing state adjustment request from the task agent and cooperates to complete the job queuing order adjustment of the local queue. Figure 9 is a whole architecture diagram of a cross-multi-queue coordination system according to an optional embodiment of the application, as Figure 9 shown, the global total control platform and the heterogeneous resource platform are in a one-to-many relationship.

[0073] The embodiment of the application can provide a computer device. Optionally, in the embodiment, the computer device can be located in at least one network device of a plurality of network devices of a computer network. The computer device comprises a memory and a processor.

[0074] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the multi-queue job processing method and device in the embodiment of the application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned multi-queue job processing method. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the computer terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0075] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: obtaining queue state information of each sub-job located in a plurality of job queues, wherein the plurality of sub-jobs located in different job queues respectively belong to a complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the queue state information includes initial queuing order and resource information; determining a target queuing order of each sub-job in the plurality of job queues based on the queue state information; and issuing a sorting instruction to a controller corresponding to each of the plurality of job queues based on the target queuing order, wherein the sorting instruction is used to sort the plurality of sub-jobs according to the target queuing order.

[0076] Optionally, the processor can further execute program codes of the following steps: obtaining queue state information of each sub-job located in a plurality of job queues includes: reading queue state information uploaded by a local controller corresponding to each of the plurality of job queues from a central database.

[0077] Optionally, the processor can further execute program codes of the following steps: determining a target queuing order of each sub-job in the plurality of job queues based on the queue state information includes: obtaining an initial global sorting of a plurality of to-be-executed jobs; adjusting the initial global sorting according to a preset global strategy target based on the queue state information to obtain a target global sorting of the plurality of to-be-executed jobs; and adjusting the initial queuing order of each sub-job in the plurality of job queues based on the target global sorting to obtain the target queuing order.

[0078] Optionally, the processor can further execute program codes of the following steps: adjusting the initial global sorting according to a preset global strategy target based on the queue state information to obtain a target global sorting of the plurality of to-be-executed jobs includes: performing permutation and combination on the plurality of to-be-executed jobs to obtain a plurality of original global sortings; constructing a first scoring function with the global strategy target as a target, wherein the first scoring function is used to quantify a matching degree between the original global sorting and the global strategy target; scoring the plurality of original global sortings respectively based on the first scoring function to obtain a first scoring result corresponding to each of the plurality of original global sortings; and adjusting the initial global sorting based on the first scoring result to obtain the target global sorting.

[0079] The embodiment of the present application provides a multi-queue job processing method. The queue state information of each sub-job in multiple job queues is acquired, wherein the multiple sub-jobs in different job queues belong to a complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the queue state information includes initial queuing order and resource information; the target queuing order of each sub-job in the multiple job queues is determined based on the queue state information; and the sorting instruction is issued to the controller corresponding to each of the multiple job queues based on the target queuing order, wherein the sorting instruction is used for sorting the multiple sub-jobs according to the target queuing order, so that the arrangement order of each sub-job in different queues is intelligently adjusted, the technical effect of improving resource utilization efficiency and job execution speed is realized, and the technical problem of resource interlocking between multiple queues caused by the uncooperative queuing of sub-jobs in the multiple queues is solved.

[0080] Those skilled in the art can understand that all or part of the steps in the above-mentioned various methods of the embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a nonvolatile storage medium, and the storage medium can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0081] The embodiment of the present application further provides a nonvolatile storage medium. Optionally, in the embodiment, the nonvolatile storage medium can be used to save the program code executed by the multi-queue job processing method provided by the above-mentioned embodiments.

[0082] Optionally, in the embodiment, the nonvolatile storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0083] Optionally, in the embodiment, the nonvolatile storage medium is configured to store program code for performing the following steps: acquiring the queue state information of each sub-job in multiple job queues, wherein the multiple sub-jobs in different job queues belong to a complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the queue state information includes initial queuing order and resource information; determining the target queuing order of each sub-job in the multiple job queues based on the queue state information; and issuing the sorting instruction to the controller corresponding to each of the multiple job queues based on the target queuing order, wherein the sorting instruction is used for sorting the multiple sub-jobs according to the target queuing order.

[0084] Optionally, in the embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining queue state information of each sub-job located in the plurality of job queues, including: reading, from the central database, the queue state information uploaded by the local controller corresponding to each of the plurality of job queues.

[0085] Optionally, in the embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: based on the queue state information, determining the target queuing order of each sub-job in the plurality of job queues, including: obtaining an initial global order of the plurality of to-be-executed jobs; based on the queue state information, adjusting the initial global order according to a preset global strategy target to obtain a target global order of the plurality of to-be-executed jobs; and based on the target global order, adjusting the initial queuing order of each sub-job in the plurality of job queues to obtain the target queuing order.

[0086] Optionally, in the embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: based on the queue state information, adjusting the initial global order according to a preset global strategy target to obtain a target global order of the plurality of to-be-executed jobs, including: performing permutation and combination on the plurality of to-be-executed jobs to obtain a plurality of original global orders; constructing a first scoring function with the global strategy target as the target, wherein the first scoring function is used to quantify the matching degree between the original global order and the global strategy target; based on the first scoring function, scoring the plurality of original global orders respectively to obtain a first scoring result corresponding to each of the plurality of original global orders; and based on the first scoring result, adjusting the initial global order to obtain the target global order.

[0087] The embodiment of the application also provides a computer program product, including a computer program, and optionally, in the embodiment, the computer program can be implemented by a processor to perform the following steps: obtaining queue state information of each sub-job located in the plurality of job queues, wherein the plurality of sub-jobs located in different job queues respectively belong to one complete to-be-executed job, the sub-job includes one or more to-be-executed tasks, and the queue state information includes an initial queuing order and resource information; based on the queue state information, determining a target queuing order of each sub-job in the plurality of job queues respectively; and based on the target queuing order, issuing an ordering instruction to the controller corresponding to each of the plurality of job queues, wherein the ordering instruction is used to order the plurality of sub-jobs according to the target queuing order.

[0088] The above-mentioned serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0089] In the above-mentioned embodiments of the application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0090] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0091] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0092] In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0093] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a non-volatile storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0094] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A multi-queue job processing method, characterized by, The method comprises: obtaining queue state information of each sub-job located in a plurality of job queues, wherein the plurality of sub-jobs located in different job queues respectively belong to a complete to-be-executed job, the sub-job comprises one or more to-be-executed tasks, and the queue state information comprises initial queuing order and resource information; determining a target queuing order of each sub-job in the plurality of job queues respectively based on the queue state information; issuing a sorting instruction to a controller corresponding to each of the plurality of job queues based on the target queuing order, wherein the sorting instruction is used to sort the plurality of sub-jobs according to the target queuing order.

2. The method of claim 1, wherein, The obtaining of the queue state information of each sub-job located in the plurality of job queues comprises: reading the queue state information uploaded by the local controller corresponding to each of the plurality of job queues from a central database.

3. The method of claim 1, wherein, The determining of the target queuing order of each sub-job in the plurality of job queues respectively based on the queue state information comprises: obtaining an initial global sorting of a plurality of to-be-executed jobs; adjusting the initial global sorting according to a preset global strategy target based on the queue state information to obtain a target global sorting of the plurality of to-be-executed jobs; adjusting the initial queuing order of each sub-job in the plurality of job queues based on the target global sorting to obtain the target queuing order.

4. The method of claim 3, wherein, The adjusting of the initial global sorting according to the preset global strategy target based on the queue state information to obtain the target global sorting of the plurality of to-be-executed jobs comprises: performing permutation and combination on the plurality of to-be-executed jobs to obtain a plurality of original global sortings; constructing a first scoring function with the global strategy target as a target, wherein the first scoring function is used to quantify a matching degree between the original global sorting and the global strategy target; scoring the plurality of original global sortings respectively based on the first scoring function to obtain a first scoring result corresponding to each of the plurality of original global sortings; adjusting the initial global sorting based on the first scoring result to obtain the target global sorting.

5. A multi-queue job processing method, characterized by, The method comprises: obtaining queue state information of a plurality of sub-jobs in a local job queue, wherein the sub-job comprises one or more to-be-executed tasks, and the queue state information comprises initial queuing order and resource information; sending the queue state information to a central controller; receiving a sorting instruction sent by the central controller, wherein the sorting instruction comprises a target queuing order, and the target queuing order is determined by the central controller based on the queue state information; sorting the plurality of sub-jobs according to the target queuing order based on the sorting instruction.

6. The method of claim 5, wherein, The obtaining of the queue state information of the plurality of sub-jobs in the local job queue comprises: obtain queue information of the job queue, job information of each of the plurality of sub-jobs, and an overall scheduling strategy, wherein the queue information comprises a queue priority and a queue admission rule, the job information comprises a sub-job priority and a rule label, and the overall scheduling strategy is a preset rule for scheduling the plurality of job queues and corresponding sub-jobs; determine the initial queuing order based on the queue information, the job information, and the overall scheduling strategy; determine the queue state information based on the initial queuing order.

7. The method of claim 6, wherein, The determining of the initial queuing order based on the queue information, the job information, and the overall scheduling strategy comprises: determining a plurality of sub-jobs of the job queue; dividing the plurality of sub-jobs into a plurality of sets based on the queue priority and the queue admission rule, wherein the queue priorities of all sub-jobs in one set are in the same priority range, and the admission rules of all sub-jobs in one set are the same; for each set, performing permutation and combination on all sub-jobs in the set to obtain a plurality of potential ordering results corresponding to the plurality of sets respectively; constructing a second scoring function taking the sub-task priority, the rule label, and the overall scheduling strategy as the standard based on a preset weight, wherein the second scoring function is used to quantify the comprehensive matching degree between the potential ordering results and target sub-task priority, target rule label, and target overall scheduling strategy; based on the second scoring function, scoring the plurality of potential ordering results respectively for each set to obtain a second scoring result corresponding to the plurality of potential ordering results respectively; determining a target ordering corresponding to each of the plurality of sets based on the second scoring result; merging the plurality of sets according to the target ordering to obtain the initial queuing order.

8. A multi-queue job processing system, characterized by comprise: a central controller configured to perform the multi-queue job processing method in any one of claims 1 to 4; and a plurality of local controllers configured to sort sub-tasks in a local job queue based on a sorting instruction issued by the central controller.

9. A non-volatile storage medium, comprising: The non-volatile storage medium comprises a stored program, wherein the program controls the device in which the non-volatile storage medium is located to perform the multi-queue job processing method in any one of claims 1 to 4 when the program is running.

10. A computer device, comprising: comprise: a memory and a processor, the memory stores a computer program; the processor is configured to execute the computer program stored in the memory, and the computer program causes the processor to perform the multi-queue job processing method in any one of claims 1 to 4 when running.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the multi-queue job processing method in any one of claims 1 to 3.

Images