A super-computer-based job scheduling optimization system and method

Abstract

The application provides a kind of job scheduling optimization system and method based on supercomputer, it is related to computer high-performance computing technical field, the job scheduling optimization system obtains and caches the number of waiting jobs of all current users in scheduling system by scheduling cache server;And through job cache server, the job submitted by user is received, and the job is added to the tail of the job cache queue of the user;The number of waiting jobs of the user is obtained from the scheduling cache server, if the number of waiting jobs is less than or equal to the preset number threshold, then the job of job cache queue is sorted and sequentially submitted to scheduling system, until the number of job cache of the user is greater than the number threshold, stop submitting job. In this way, in the case where the user scale of supercomputer cluster increases and the number of jobs increases, the access frequency of the scheduling system can be reduced, and its stability can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of high-performance computing technology, and in particular relates to a job scheduling optimization system and method based on supercomputing. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute the prior art that has been known to those skilled in the art.

[0003] With the development of high-performance computing technology, job scheduling for supercomputing clusters has become a crucial component of daily supercomputing operations. During the job scheduling process, users submit jobs to the scheduling system, which then allocates and schedules the resources of the supercomputing cluster.

[0004] To ensure stable operation of supercomputing cluster scheduling, a common approach is to limit the maximum number of jobs submitted by each user through the scheduling system to ensure that scheduling can run stably within a certain number of jobs. This approach is limited in that as the number of supercomputing cluster users increases, the number of jobs also increases, which in turn increases the number of accesses to the scheduling system, making the scheduling system itself increasingly unstable, thereby affecting the efficiency of supercomputing cluster resource allocation. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a supercomputing-based job scheduling optimization system and method, so as to reduce the number of visits to the scheduling system as much as possible and improve the stability of the scheduling system when the user scale of the supercomputing cluster increases and the number of jobs increases.

[0006] In order to achieve the above object, the present invention mainly includes the following aspects:

[0007] In a first aspect, an embodiment of the present invention provides a supercomputing-based job scheduling optimization system, comprising a scheduling system and a supercomputing cluster, wherein the scheduling system is used to allocate and schedule resources of the supercomputing cluster, and further comprising: a scheduling cache server and a job cache server;

[0008] The scheduling cache server is used to obtain and cache the number of waiting jobs of all users currently in the scheduling system;

[0009] The job cache server is used to receive jobs submitted by users and add the jobs to the end of the user's job cache queue; obtain the number of waiting jobs of the user from the scheduling cache server, and if the number of waiting jobs is less than or equal to a preset number threshold, sort the jobs in the job cache queue and submit them to the scheduling system in sequence until the number of job caches of the user exceeds the number threshold, and stop submitting jobs.

[0010] In one possible implementation, the job cache server is specifically used to calculate, for each job in the job cache queue, the product of the total number of processes used by the job and the job queue waiting time, determine the sum of the product and the initial weight as the weight of the job, and sort the jobs in the job cache queue according to the weight.

[0011] In a possible implementation, when the job cache server determines that the number of waiting jobs of the user is greater than a quantity threshold, the job cache server caches the received jobs.

[0012] In a possible implementation, upon receiving the job cache queue of the user, the scheduling system sends a notification of updating the number of waiting jobs of the user to the scheduling cache server.

[0013] In a possible implementation, when a job submitted by a user is completed, the scheduling system sends a notification to the scheduling cache server to update the number of user waiting jobs.

[0014] In a possible implementation, when the scheduling cache server obtains and caches the number of waiting jobs of the user, it sends a cache service notification to the job cache server.

[0015] In a second aspect, an embodiment of the present invention provides a method for optimizing job scheduling based on a supercomputer, comprising:

[0016] Get and cache the number of waiting jobs for all users currently in the scheduling system;

[0017] Receive the job submitted by the user and add the job to the end of the user's job cache queue; obtain the number of waiting jobs for the user. If the number of waiting jobs is less than or equal to a preset number threshold, sort the jobs in the job cache queue and submit them to the scheduling system in sequence until the number of waiting jobs for the user exceeds the number threshold, and then stop submitting jobs.

[0018] In a possible implementation, when it is determined that the number of waiting jobs of the user is greater than a quantity threshold, the received jobs are cached.

[0019] In a third aspect, an embodiment of the present invention provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the supercomputing-based job scheduling optimization method as described in the second aspect and any possible implementation of the second aspect are performed.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the supercomputing-based job scheduling optimization method as described in any possible implementation of the second aspect and the second aspect are executed.

[0021] One or more of the above technical solutions have the following beneficial effects:

[0022] The job scheduling optimization system provided by the present invention obtains and caches the number of waiting jobs for all users currently in the scheduling system through a scheduling cache server; receives jobs submitted by users through the job cache server and adds the jobs to the end of the user's job cache queue; obtains the number of waiting jobs for the user from the scheduling cache server, and if the number of waiting jobs is less than or equal to a preset number threshold, sorts the jobs in the job cache queue and submits them to the scheduling system in sequence until the number of job caches for the user exceeds the threshold, at which point job submission stops. In this way, as the user scale of the supercomputing cluster increases and the number of jobs increases, the number of accesses to the scheduling system is reduced, thereby improving the stability of the scheduling system.

[0023] In addition, the weight of the job in the job cache queue is determined based on the total number of processes used by the job, the job queue waiting time and the initial weight set by the user. The order of submission to the scheduling system is adjusted according to the weight of the job, which can meet the needs of users and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 Schematic diagram of the structure of the supercomputing-based job scheduling optimization system provided in the first embodiment of the present invention;

[0026] Figure 2 This is a flow chart of the supercomputing-based job scheduling optimization method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0029] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0030] Example 1

[0031] Supercomputing clusters are typically computing systems and environments composed of multiple supercomputing center processors or physical computers. The scheduling system is a distributed resource management and scheduling system. Using a specific algorithm, the scheduling system selects jobs from an external job queue, loads them into memory, creates processes for them, and allocates resources. To ensure stable operation of supercomputing cluster scheduling, the maximum number of jobs submitted by each user is currently limited. However, as the number of supercomputing cluster users and the number of jobs increases, the number of accesses to the scheduling system also increases, making the scheduling system increasingly unstable and impacting the efficiency of supercomputing cluster resource allocation.

[0032] Based on this, an embodiment of the present invention provides a supercomputing-based job scheduling optimization system. By caching user-submitted jobs, when the number of waiting jobs for a user is less than or equal to a preset threshold, the job cache queue is sorted and submitted sequentially to the scheduling system. This minimizes frequent access to the scheduling system and improves the stability of the scheduling system as the user scale of the supercomputing cluster increases and the number of jobs increases. Furthermore, the job cache queue can be sorted according to user needs, adjusting the order in which jobs are submitted to the scheduling system, thereby improving the user experience.

[0033] like Figure 1 As shown, this embodiment provides a job scheduling optimization system based on supercomputing, including a scheduling system and a supercomputing cluster. The scheduling system is used to allocate and schedule resources of the supercomputing cluster, and also includes: a scheduling cache server and a job cache server;

[0034] The scheduling cache server is used to obtain and cache the number of waiting jobs of all users currently in the scheduling system;

[0035] The job cache server is used to receive jobs submitted by users and add the jobs to the end of the user's job cache queue; obtain the number of waiting jobs of the user from the scheduling cache server, and if the number of waiting jobs is less than or equal to a preset number threshold, sort the jobs in the job cache queue and submit them to the scheduling system in sequence until the number of job caches of the user exceeds the number threshold, and stop submitting jobs.

[0036] On a supercomputing cluster, when a user submits a job using the scheduling system, the scheduling cache server is queried to determine whether the user has more than X pending jobs, where X is a custom value greater than 0. If the user has more than X pending jobs, the submitted job is added to the cache queue and does not interact with the scheduling system. If the user has fewer than or equal to X pending jobs, the scheduling system notifies the scheduling cache server to update the scheduling cache. After the scheduling cache server updates the cache, it notifies the job cache server to submit the job.

[0037] Here, when the number of waiting jobs is less than or equal to a preset threshold, the job cache server sorts the jobs in the job cache queue and submits them sequentially to the scheduling system. Furthermore, the job cache server is specifically configured to calculate, for each job in the job cache queue, the product of the total number of processes used by the job and the job's queue waiting time, determine the sum of this product and an initial weight as the job's weight, and sort the jobs in the job cache queue according to the weight.

[0038] In practice, the calculation rule is: Single Job Weight = Total Number of Processes Used by the Job * Number of Minutes the Job Has Been Queued + Initial Weight. The initial weight defaults to 1 to ensure that all submitted jobs are weighted at the same level. When submitting a job, the user specifies the total number of processes required and sets an initial weight for the job. This allows the order of jobs in the job cache queue to be adjusted based on actual user needs, improving the user experience.

[0039] When the job cache server determines that the number of waiting jobs of the user is greater than the number threshold, it caches the received job. At this time, the job does not interact with the scheduling system, which can effectively reduce the number of visits to the scheduling system and improve the stability of the scheduling system.

[0040] Furthermore, when the scheduling system receives the user's job cache queue, it sends a notification to the scheduling cache server to update the number of jobs waiting for the user; or, when the job submitted by the user is completed, the scheduling system sends a notification to the scheduling cache server to update the number of jobs waiting for the user.

[0041] When the scheduling cache server obtains and caches the number of waiting jobs of a user, it sends a cache service notification to the job cache server. This ensures that the scheduling cache server updates the scheduling information in a timely manner and notifies the job cache server to perform job cache services, thereby improving the allocation efficiency of supercomputing cluster resources.

[0042] Example 2

[0043] like Figure 2 As shown, the embodiment of the present invention also provides a method for optimizing job scheduling based on supercomputing, which specifically includes the following steps:

[0044] S201: Obtain and cache the number of waiting jobs for all users currently in the scheduling system;

[0045] S202: Receive a job submitted by a user and add the job to the end of the user's job cache queue; obtain the number of waiting jobs for the user, and if the number of waiting jobs is less than or equal to a preset threshold, sort the jobs in the job cache queue and submit them to the scheduling system in sequence until the number of waiting jobs for the user exceeds the threshold, and then stop submitting jobs.

[0046] As an optional implementation, when it is determined that the number of waiting jobs of the user is greater than a quantity threshold, the received jobs are cached.

[0047] The supercomputing-based job scheduling optimization method provided in this embodiment is obtained based on the aforementioned supercomputing-based job scheduling optimization system. The specific implementation method of the supercomputing-based job scheduling optimization method can be found in the embodiment section of the supercomputing-based job scheduling optimization system in the previous text, and will not be repeated here.

[0048] Example 3

[0049] An embodiment of the present invention further provides a computer device, which includes a processor, a memory, and a bus.

[0050] The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via a bus. When the machine-readable instructions are executed by the processor, the above-mentioned Figure 2 The specific implementation of the steps of the supercomputing-based job scheduling optimization method in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0051] Example 4

[0052] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the supercomputing-based job scheduling optimization method described in the above method embodiment are executed.

[0053] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0054] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A supercomputing-based job scheduling optimization system, comprising a scheduling system and a supercomputing cluster, wherein the scheduling system is used to allocate and schedule resources of the supercomputing cluster, and is characterized in that: Also included: a scheduling cache server and a job cache server; The scheduling cache server is used to obtain and cache the number of waiting jobs of all users currently in the scheduling system; The job cache server is used to receive jobs submitted by users and add the jobs to the end of the job cache queue of the user; obtain the number of waiting jobs of the user from the scheduling cache server, and if the number of waiting jobs is less than or equal to a preset number threshold, sort the jobs in the job cache queue and submit them to the scheduling system in sequence until the number of job caches of the user exceeds the number threshold, and then stop submitting jobs; when the job cache server determines that the number of waiting jobs of the user is greater than the number threshold, it caches the received jobs; The job cache server is specifically used to calculate, for each job in the job cache queue, the product of the total number of processes used by the job and the job queue waiting time, determine the sum of the product and the initial weight as the weight of the job, and sort the jobs in the job cache queue according to the weight.

2. The supercomputer-based job scheduling optimization system according to claim 1, characterized in that: When receiving the job cache queue of the user, the scheduling system sends a notification of updating the number of waiting jobs of the user to the scheduling cache server.

3. The supercomputer-based job scheduling optimization system according to claim 1, characterized in that: When a job submitted by a user is completed, the scheduling system sends a notification to the scheduling cache server to update the number of waiting jobs for the user.

4. The supercomputer-based job scheduling optimization system according to claim 1, characterized in that: When the scheduling cache server obtains and caches the number of waiting jobs of the user, it sends a cache service notification to the job cache server.

5. A method for optimizing job scheduling based on supercomputing, characterized in that: include: Get and cache the number of waiting jobs for all users currently in the scheduling system; Receive a job submitted by a user and add the job to the end of the user's job cache queue; obtain the number of waiting jobs for the user, and if the number of waiting jobs is less than or equal to a preset number threshold, sort the jobs in the job cache queue and submit them to the scheduling system in sequence until the number of the user's job cache exceeds the number threshold, then stop submitting jobs; the job cache server caches the received job when it determines that the number of waiting jobs for the user is greater than the number threshold; The job cache server is specifically used to calculate, for each job in the job cache queue, the product of the total number of processes used by the job and the job queue waiting time, determine the sum of the product and the initial weight as the weight of the job, and sort the jobs in the job cache queue according to the weight.

6. The method for optimizing job scheduling based on supercomputing according to claim 5, wherein: When it is determined that the number of waiting jobs of the user is greater than the number threshold, the received jobs are cached.

7. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate through the bus. When the machine-readable instructions are executed by the processor, the steps of the supercomputing-based job scheduling optimization method as described in any one of claims 5 to 6 are performed.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the supercomputing-based job scheduling optimization method according to any one of claims 5 to 6 are executed.

Images