Job task scheduling method and system based on computing power resource pool

Abstract

The embodiment of the invention discloses a job task scheduling method and system based on a computing power resource pool. The method comprises the steps of obtaining a target user identifier and a target scheduling mode through a computing power application portal subsystem, and determining a candidate computing power resource pool partition and a target computing power quota based on the target user identifier; if the target scheduling mode is detected to be a query scheduling mode through the computing power application portal subsystem, obtaining a target job task and a target job demand; performing partition screening on the candidate computing power resource pool partitions based on a target job demand through a computing power scheduling subsystem to determine a target computing power resource pool partition; and controlling the target computing power resource pool partition to execute the target job task based on the target computing power quota through the computing power base platform. Through the technical scheme of the embodiment of the invention, the job task can be quickly and accurately scheduled to the effective computing power resource pool, and the scheduling flexibility and the user experience are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a method and system for scheduling job tasks based on a computing resource pool. Background Art

[0002] With the development of technology, artificial intelligence (AI) and large models have developed extremely rapidly. Various AI applications represented by large models such as ChatGPT, Gemini, Wenxin Yiyan, and Tongyi Qianwen have been quickly launched and applied, resulting in a rapid increase in the demand for intelligent computing power in application scenarios including AI training and AI reasoning. The development of large models has put forward huge demands on computing power. With the continuous expansion of model parameter scale, the sharp increase in the amount of training data, and the increasing complexity of application scenarios, the demand for computing power has exploded. Training an advanced large model often requires thousands or even tens of thousands of high-performance chips to work together, consuming a lot of electricity and computing resources. It can be seen that computing power has become a key support for the development of artificial intelligence and large models, and its importance continues to be highlighted.

[0003] At present, computing power service providers usually use a single server as the minimum delivery and measurement unit, and provide computing power services through monthly and annual packages for the entire machine. However, the single-use fee of this computing power service provision method is high, which is difficult for individual users and small and micro enterprises to afford, resulting in a high threshold for computing power services. In addition, the super-intelligent computing core time products and services provided by institutions represented by the Super Intelligent Computing Center only access computing power resource pools managed by the institution itself. However, due to the small number of computing power resource pools managed by a single institution and the relatively limited computing power resources, users have few choices, the computing power service ecosystem is relatively closed, and the computing power resources located in different geographical locations are not integrated, the island effect is obvious, and the utilization rate of each computing power resource pool is uneven. Summary of the invention

[0004] The embodiments of the present invention provide a method and system for scheduling job tasks based on a computing power resource pool, so as to achieve fast and accurate scheduling of job tasks to an effective computing power resource pool, thereby improving scheduling flexibility and user experience.

[0005] In a first aspect, an embodiment of the present invention provides a method for scheduling a job task based on a computing resource pool, comprising:

[0006] Obtaining a target user identifier and a target scheduling mode of a target user through a computing power application portal subsystem, and determining a candidate computing power resource pool partition and a target computing power quota corresponding to the target user based on the target user identifier;

[0007] If the computing power application portal subsystem detects that the target scheduling mode is the query scheduling mode, the target job task and target job requirement of the target user are obtained;

[0008] The computing power scheduling subsystem performs partition screening on the candidate computing power resource pool partitions based on the acquired target job requirements, determines the target computing power resource pool partitions that meet the target job requirements, and sends the target computing power resource pool partitions to the computing power application portal subsystem;

[0009] The target computing power resource pool partition and the target computing power quota are sent to the computing power base platform through the computing power application portal subsystem, and the target job task is sent to the target computing power resource pool partition;

[0010] The target computing power resource pool partition is controlled by the computing power base platform based on the target computing power quota to execute the target job task.

[0011] Optionally, the method also includes: the target job requirements include: target computing power application, target computing power resource instance and expected execution strategy; the target scheduling mode includes: manual scheduling mode, query scheduling mode and transit agent scheduling mode; the query scheduling mode and the transit agent scheduling mode belong to automatic scheduling mode.

[0012] Optionally, the method also includes: if the computing power application portal subsystem detects that the target scheduling mode is a manual scheduling mode, then obtaining the target job task, target job demand and target computing power resource pool partition of the target user; sending the acquired target computing power resource pool partition and target computing power quota to the computing power base platform through the computing power scheduling subsystem; and sending the target job task to the target computing power resource pool partition through the computing power application portal subsystem.

[0013] Optionally, the method further includes: the expected execution strategy in the target job requirement is a null value.

[0014] Optionally, the method also includes: obtaining the target job tasks and target job requirements of the target user; screening the candidate computing power resource pool partitions corresponding to the target user based on the target computing power applications and target computing power resource instances in the target job requirements, determining and displaying the computing power resource pool partitions to be selected; and obtaining the target computing power resource pool partition selected by the target user from the computing power resource pool partitions to be selected.

[0015] Optionally, the method also includes: determining the target execution time of the target job task through the computing power base platform, and sending the target execution result corresponding to the target job task and the target execution time to the computing power application portal subsystem; determining the target execution cost corresponding to the target job task based on the target execution time and the target computing power resource instance in the target job requirement through the computing power application portal subsystem; if the computing power application portal subsystem detects that the target execution cost is obtained, the target execution result will be displayed.

[0016] Optionally, the method also includes: if the computing power application portal subsystem detects that the target scheduling mode is a transit agent scheduling mode, then obtaining the target job tasks and target job requirements of the target user; using the computing power scheduling subsystem to perform partition screening on the candidate computing power resource pool partitions based on the acquired target job requirements, determining the target computing power resource pool partition that meets the target job requirements, and sending the target computing power resource pool partition and the target computing power quota to the computing power base platform; using the computing power scheduling subsystem to store the acquired target job tasks in a public storage space, and creating a target file directory corresponding to the target user in the target computing power resource pool partition, and sending the target job tasks to the target file directory.

[0017] In a second aspect, an embodiment of the present invention further provides a job task scheduling system based on a computing power resource pool, the system comprising: a computing power application portal subsystem, a computing power scheduling subsystem, a computing power base platform, and a full computing power resource pool; the full computing power resource pool comprises: a candidate computing power resource pool partition and a target computing power resource pool partition;

[0018] The computing power application portal subsystem is used to obtain a target user identifier and a target scheduling mode of a target user, and determine a candidate computing power resource pool partition and a target computing power quota corresponding to the target user based on the target user identifier;

[0019] The computing power application portal subsystem is used to obtain the target job task and target job requirement of the target user if it is detected that the target scheduling mode is the query scheduling mode;

[0020] The computing power scheduling subsystem is used to screen the candidate computing power resource pool partitions based on the acquired target job requirements, determine the target computing power resource pool partitions that meet the target job requirements, and send the target computing power resource pool partitions to the computing power application portal subsystem;

[0021] The computing power application portal subsystem is used to send the target computing power resource pool partition and the target computing power quota to the computing power base platform, and send the target job task to the target computing power resource pool partition;

[0022] The computing power base platform is used to control the target computing power resource pool partition to execute the target job task based on the target computing power quota.

[0023] Optionally, the system also includes: the full computing power resource pool, which is used to provide underlying computing power resources; wherein the underlying computing power resources include: at least one of a central processing unit, a graphics processing unit, a memory space, a graphics card cache space and a network.

[0024] Optionally, the system also includes: each computing resource pool includes multiple clusters; each cluster is obtained by combining multiple computing hardware resources; each cluster includes multiple partitions; and the computing resources in each partition have the same configuration.

[0025] The technical solution of the embodiment of the present invention obtains the target user identifier and target scheduling mode of the target user through the computing power application portal subsystem, and determines the candidate computing power resource pool partition and target computing power quota corresponding to the target user based on the target user identifier; if the computing power application portal subsystem detects that the target scheduling mode is a query scheduling mode, the target job task and target job demand of the target user are obtained; the computing power scheduling subsystem performs partition screening on the candidate computing power resource pool partition based on the acquired target job demand, determines the target computing power resource pool partition that meets the target job demand, and sends the target computing power resource pool partition to the computing power application portal subsystem; sends the target computing power resource pool partition and the target computing power quota to the computing power base platform through the computing power application portal subsystem, and sends the target job task to the target computing power resource pool partition; controls the target computing power resource pool partition to execute the target job task based on the target computing power quota through the computing power base platform, thereby realizing the rapid and accurate scheduling of the job task to the effective computing power resource pool, thereby improving the scheduling flexibility and user experience.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 It is a flowchart of a method for scheduling a job task based on a computing resource pool provided in the first embodiment of the present invention;

[0029] Figure 2 This is an architectural diagram of a query scheduling mode involved in Embodiment 1 of the present invention;

[0030] Figure 3 It is a flowchart of a method for scheduling a job task based on a computing resource pool provided in Embodiment 2 of the present invention;

[0031] Figure 4 This is an architectural diagram of a manual scheduling mode involved in Embodiment 2 of the present invention;

[0032] Figure 5 It is a flowchart of a method for scheduling a job task based on a computing resource pool provided in Embodiment 3 of the present invention;

[0033] Figure 6 This is an architecture diagram of a transit agent scheduling mode involved in Embodiment 3 of the present invention;

[0034] Figure 7 It is a structural diagram of a job task scheduling system based on a computing resource pool provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment 1

[0038] Figure 1A flowchart of a method for scheduling a job task based on a computing resource pool is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a job task is effectively scheduled to a computing resource pool. The method can be executed by a job task scheduling system based on a computing resource pool. The job task scheduling system based on a computing resource pool can be implemented in the form of hardware and / or software. Figure 1 As shown, the method includes:

[0039] S110. Obtain the target user identifier and target scheduling mode of the target user through the computing power application portal subsystem, and determine the candidate computing power resource pool partition and target computing power quota corresponding to the target user based on the target user identifier.

[0040] Among them, the computing power application portal subsystem can be used for information acquisition and information display. The target user may refer to a user who logs in to the computing power application portal subsystem. The target user has a need to utilize computing power resources. For example, the target user may be an individual or an enterprise. The target user identifier may be used to uniquely identify the target user. For example, the target user identifier may be but is not limited to user account information. The target scheduling mode may refer to the mode of scheduling job tasks selected by the target user. For example, the target scheduling mode may include but is not limited to manual scheduling mode and automatic scheduling mode. The target computing power quota may refer to the total amount of computing power resources allowed to be used by the target user at the same time. The candidate computing power resource pool partition may refer to a partition in the computing power resource pool that the target user is allowed to use.

[0041] Specifically, the computing power application portal subsystem can obtain the target user ID corresponding to the target user when the target user logs in, and obtain the target scheduling mode corresponding to the target user when the target user selects the scheduling mode for this job. Based on the target user ID, the computing power application portal subsystem determines the candidate computing power resource pool partition and target computing power quota corresponding to the target user ID from the preset computing power resource pool partitions and preset computing power quotas pre-selected for each user.

[0042] Based on the above technical solution, each computing resource pool includes multiple clusters; each cluster is composed of a combination of multiple computing hardware resources; each cluster includes multiple partitions; and the computing resources in each partition have the same configuration.

[0043] Among them, computing power resource pools are geographically located in different locations. Computing power resource pools can come from different management agencies. Each computing power resource pool constitutes a full computing power resource pool. A computing power resource pool environment can contain multiple clusters, and a cluster can be composed of different computing hardware resources, such as NVIDIA A100 GPU and H800 GPU to form a cluster. A cluster can contain multiple partitions, and the configuration (hardware architecture) of the computing resources within a partition must be exactly the same, such as partition 1 is composed of A100 GPU computing resources, partition 2 is also composed of A100 GPU, and partition 3 is composed of H800 GPU resources. The total computing power resources of the cluster are the sum of the resources corresponding to all computing resources under all partitions under the cluster.

[0044] It should be noted that all computing nodes in each cluster share a set of storage resources to store the relevant file contents of jobs running on the cluster, including job input files (equivalent to the target job tasks) and job output files (equivalent to the target execution results). The job input files can include image files (which can contain AI models, algorithms, data sets, etc.).

[0045] Based on the above technical solutions, the underlying computing resources are provided through the full computing resource pool.

[0046] The underlying computing resources include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a memory space, a graphics card cache space, and a network.

[0047] Exemplarily, in an embodiment of the present invention, the computing power application portal subsystem can also be used to manage cluster partitions that users are authorized to use. System operators can flexibly configure the computing power resource pools-clusters-partitions allowed for each user in the background of the computing power application portal subsystem according to business needs. By default, users can use all computing power resource pools-clusters-partitions. An example of the cluster partition column that users are authorized to use is shown in Table 1 (taking the same resource pool containing multiple clusters and partitions as an example).

[0048] Table 1 List of cluster partitions that users are authorized to use

[0049] user Cluster 1-Partition 1 Cluster 1 - Partition 2 Cluster 2 - Partition 1 Cluster 2 - Partition 2 Cluster 3 - Partition 1 User 1 Available Available Available Unavailable Unavailable User 2 Available Available Available Unavailable Available User 3 Available Available Available Available Available

[0050] The candidate computing resource pool partitions corresponding to user 1 include: cluster 1-partition 1, cluster 1-partition 2, and cluster 2-partition 1. The candidate computing resource pool partitions corresponding to user 2 include: cluster 1-partition 1, cluster 1-partition 2, cluster 2-partition 1, and cluster 3-partition 1.

[0051] S120. If the computing power application portal subsystem detects that the target scheduling mode is the query scheduling mode, the target job task and target job requirement of the target user are obtained.

[0052] The query scheduling mode belongs to the automatic scheduling mode. The target job task may refer to the task file of the target user's current job. The target job requirement may refer to the job requirement that the target user proposes to meet for this job. For example, the target job requirement may include but is not limited to the job duration and the hardware resources required for the job.

[0053] Specifically, if the computing power application portal subsystem detects that the target scheduling mode is the query scheduling mode, it will display the to-be-filled information corresponding to the query scheduling mode to the user, and obtain the target job tasks and target job requirements corresponding to the target user when the target user fills in the information.

[0054] Based on the above technical solutions, the target job requirements include: target computing power application, target computing power resource instance and expected execution strategy; the target scheduling modes include: manual scheduling mode, query scheduling mode and transit agent scheduling mode; query scheduling mode and transit agent scheduling mode belong to automatic scheduling mode.

[0055] Among them, the target computing power application may refer to the application (Application, APP) required by the target user for this job. The target computing power resource instance may refer to the specifications and quantity of computing power hardware resources required by the target user for this job. The expected execution strategy may refer to the effect that the target user wants to achieve in this job scheduling. For example, the expected execution strategy may be, but is not limited to, the highest computing power pool resource utilization, the shortest job queuing time, and / or the shortest execution cost. In the automatic scheduling mode, it supports the scheduling of computing jobs to different intelligent computing resource pools based on different scheduling strategies (such as computing power pool resource utilization, job queuing time, execution cost and other factors), and can support the rapid expansion of computing power resource pools. In the automatic scheduling mode, the computing power scheduling subsystem is responsible for job scheduling. This mode is suitable for situations where multiple computing power resource pools have the same computing power application and the same computing power resource instance. For example, this mode is suitable for situations where computing power pool 1 has NVIDIA A100 model GPUs and computing power pool 2 also has A100 GPUs.

[0056] Exemplarily, the computing power scheduling subsystem queries the computing power applications deployed by each computing power resource pool through an API interface, and summarizes and saves the application lists of each computing power resource pool. Table 2 gives an example of a computing power resource pool application list.

[0057] Table 2 Example of computing resource pool application list

[0058]

[0059]

[0060] As shown in Table 2, the computing power scheduling subsystem aggregates and removes the computing power applications deployed by each computing power pool, and then provides them to the computing power application portal subsystem through the API interface for unified display. Among them, computing power pool 1 has computing power applications APP1 and APP2, and computing power pool 2 has computing power applications APP1 and APP3. Then only APP1, APP2, and APP3 are displayed on the computing power application display interface of the computing power application portal subsystem.

[0061] Exemplarily, a computing power resource instance refers to the specifications and quantity of computing power hardware resources required for user-submitted job execution, including basic computing components such as GPU, CPU, video memory, and memory. Each instance corresponds to a corresponding core time unit price (core time / hour), which is deduced by the operator based on cost and gross profit margin. Different instance unit prices can be combined according to different hardware resource configurations. It is managed through the instance unit price table in the background of the computing power application portal subsystem. Table 3 gives an example of a computing power resource instance.

[0062] Table 3 Examples of computing resource instances

[0063]

[0064] Among them, computing power core time usually refers to the amount of computing tasks that a computer system can complete in a unit of time. It is an important indicator for measuring computer performance. In the field of AI, computing power core time is usually measured in units of operations per second (FLOPS, Floating Point Operations Per Second), where P (PetaFLOPS) means quadrillion floating point operations per second.

[0065] S130. The computing power scheduling subsystem screens the candidate computing power resource pool partitions based on the acquired target job requirements, determines the target computing power resource pool partitions that meet the target job requirements, and sends the target computing power resource pool partitions to the computing power application portal subsystem.

[0066] Specifically, the user first selects the target computing application, target computing resource instance, and desired scheduling strategy on the computing power application portal subsystem. The computing power scheduling subsystem determines the target computing resource pool partition that meets the target job requirements from the candidate computing resource pool partitions corresponding to the target user based on the target computing application, target computing resource instance, and desired scheduling strategy, combined with the list of cluster partitions that the user is authorized to use, and returns it to the computing power application portal subsystem.

[0067] S140. Send the target computing power resource pool partition and the target computing power quota to the computing power base platform through the computing power application portal subsystem, and send the target job task to the target computing power resource pool partition.

[0068] in, Figure 2 A query scheduling architecture diagram is given. Figure 2 The computing power application portal subsystem obtains the target computing power resource pool partition from the computing power scheduling subsystem, creates a user file directory on the target computing power resource pool partition (if it has already been created, it will not be created again), and then uploads the target job task to the user file directory. Users submit job execution through the computing power application portal subsystem, that is, the target computing power resource pool partition and target computing power quota are sent to the computing power base platform, and the job starts. Compared with the manual scheduling mode, users do not need to know in advance or need to pay attention to which computing power resource pool-cluster-partition the submitted job is executed in.

[0069] S150. Control the target computing power resource pool partition to execute the target job task based on the target computing power quota through the computing power base platform.

[0070] Among them, the computing power base platform can be used for job management and scheduling within the computing power resource pool. For example, the computing power base platform can be used to start, queue, and stop jobs. Usually, a computing power base platform can manage a computing power resource pool (a computing power resource pool can contain multiple clusters, and a cluster can contain multiple partitions). The computing power applications on each computing power base platform are runnable on the cluster managed by the platform.

[0071] Exemplarily, each user will pre-configure a corresponding computing power quota template. Each template corresponds to a corresponding computing power quota resource. All job instances submitted by the user (including jobs in the submitted state, running state, queued state, and preempted state) share the computing power quota and cannot exceed the quota (that is, any item in GPU, CPU, and memory cannot exceed the quota). The computing power quota is used to limit the total amount of computing power resources that a single user can occupy at the same time. When a user's job changes from the above state to other states, the quota occupied by the job will be released. Table 4 gives an example of a computing power quota. Table 5 gives an example of a user computing power quota template.

[0072] Table 4 Example of computing power quota

[0073]

[0074] Table 5 Example of user computing power quota template

[0075]

[0076] Here is an example of a change in user computing power quota: User a's quota template is template 1 (including 32 GPUs, 64 CPUs, and 960 GB of memory). User a selects computing power application APP1 and submits a job. The computing power resource instance selected is instance 1, which uses 8 GPUs, 48 ​​CPUs, and 960 GB of memory resources. The job enters the running state. Then user a continues to submit the second job, and the instance selected is instance 2, which uses 4 GPUs, 24 CPUs, and 480 GB of memory resources. At this time, because the accumulated CPU and memory exceed the user's quota, the system will prompt the user that the quota is exceeded, and the second job cannot be submitted for execution. After a period of time, when the first job is completed, the system will release the 8 GPUs, 48 ​​CPUs, and 960 GB of memory quotas occupied by the first job. At this time, the user can resubmit the second job and run it normally.

[0077] The technical solution of the embodiment of the present invention obtains the target user identifier and target scheduling mode of the target user through the computing power application portal subsystem, and determines the candidate computing power resource pool partition and target computing power quota corresponding to the target user based on the target user identifier; if the computing power application portal subsystem detects that the target scheduling mode is the query scheduling mode, the target job task and target job demand of the target user are obtained; the computing power scheduling subsystem performs partition screening on the candidate computing power resource pool partition based on the acquired target job demand, determines the target computing power resource pool partition that meets the target job demand, and sends the target computing power resource pool partition to the computing power application portal subsystem; sends the target computing power resource pool partition and the target computing power quota to the computing power base platform through the computing power application portal subsystem, and sends the target job task to the target computing power resource pool partition; controls the target computing power resource pool partition to execute the target job task based on the target computing power quota through the computing power base platform, thereby realizing the rapid and accurate scheduling of the job task to the effective computing power resource pool, thereby improving the scheduling flexibility and user experience.

[0078] Based on the above technical solution, the method also includes: determining the target execution time of the target job task through the computing power base platform, and sending the target execution result and target execution time corresponding to the target job task to the computing power application portal subsystem; determining the target execution cost corresponding to the target job task based on the target execution time and the target computing power resource instance in the target job requirement through the computing power application portal subsystem; if the computing power application portal subsystem detects that the target execution cost is obtained, the target execution result will be displayed.

[0079] The target execution time may refer to the execution time of the target job task using computing resources. The target execution time does not include the waiting time in queue. The target execution result may refer to the result obtained by executing the target job task using computing resources. The target execution fee may refer to the fee that the target user needs to pay for executing the target job task.

[0080] Specifically, the computing power base platform determines the target execution time of the target operation task when executing and completing the target operation task, and sends the target execution result and target execution time corresponding to the target operation task to the computing power application portal subsystem. The computing power application portal subsystem determines the target execution fee corresponding to the target operation task based on the target execution time and the core time unit price in the target computing power resource instance in the target operation requirement, and displays the target execution fee to the target user. The computing power application portal subsystem detects the acquisition of the target execution fee when the target user pays the target execution fee, and displays the target execution result and allows downloading.

[0081] Exemplarily, the computing power application portal subsystem is specifically used for computing power core time product display, core time product subscription payment and service activation, computing power application display, job task submission, job metering and billing, and user self-service.

[0082] Among them, the computing power core time product may refer to a low-threshold computing power service product for individual customers (2C) and small B customers (2B), which is charged according to the amount of computing power core time consumed by the computing job tasks submitted by the user. The computing power core time product provides services to public users in the form of the Internet, realizes the effective integration of computing power resources located in different geographical locations, and facilitates users to call it conveniently. The display of computing power core time products can be understood as the computing power application portal subsystem in the embodiment of the present invention displaying the candidate computing power resource pool partitions, optional computing power applications, and optional computing power resource instances of the target user. The metering and billing of computing power core time products is the responsibility of the computing power portal subsystem, and the specific functions include recharging the user account core time balance, deducting the user's job core time, and generating and displaying the user's job consumption core time detailed list. The embodiment of the present invention supports the e-commerce of the entire process of computing power core time product ordering, payment, job management, billing and detailed list management to improve user experience.

[0083] Embodiment 2

[0084] Figure 3 This is a flowchart of a method for scheduling a job task based on a computing resource pool provided in the second embodiment of the present invention. Based on the above embodiment, this embodiment describes in detail the scheduling process of the manual scheduling mode. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 3 As shown, the method includes:

[0085] S310. Obtain the target user identifier and target scheduling mode of the target user through the computing power application portal subsystem, and determine the candidate computing power resource pool partition and target computing power quota corresponding to the target user based on the target user identifier.

[0086] S320. If the computing power application portal subsystem detects that the target scheduling mode is the manual scheduling mode, the target job task, target job requirement and target computing power resource pool partition of the target user are obtained.

[0087] Among them, the expected execution strategy in the target job requirement is a null value. In manual scheduling mode, the upload and download interactions of job files (equivalent to the target job tasks) are implemented through the computing power application portal subsystem and the computing power resource pool, without going through the computing power scheduling subsystem and the computing power base platform. Each user has a separate user job file storage space on the computing power resource pool, and different user job file storage spaces are isolated from each other to ensure user data security, including uploading input files for job execution or downloading output files for job execution.

[0088] Specifically, if the computing power application portal subsystem detects that the target scheduling mode is manual scheduling mode, it will display the to-be-filled information corresponding to the query scheduling mode to the user, and obtain the target job tasks, target job requirements and target computing power resource pool partitions corresponding to the target user when the target user fills in the information.

[0089] For example, Figure 4 A manual scheduling mode architecture diagram is given. Figure 4 Before submitting a job, the user needs to first select the computing power resource pool-cluster-partition where the job will run, and then select the computing power application and computing power resource instance on the cluster. The computing power application portal subsystem obtains the target computing power application, target computing power resource instance, and target computing power resource pool partition selected by the target user. When the target user uploads the job input file (equivalent to the target job task), the computing power application portal subsystem sends the job input file to the target computing power resource pool partition. The computing power scheduling subsystem sends the target computing power resource pool partition and target computing power quota provided by the computing power application portal subsystem to the computing power base platform and starts the job.

[0090] On the basis of the above technical solution, "obtaining the target job tasks, target job requirements and target computing power resource pool partitions of the target user" may include: obtaining the target job tasks and target job requirements of the target user; screening the candidate computing power resource pool partitions corresponding to the target user based on the target computing power application and target computing power resource instance in the target job requirements, determining and displaying the computing power resource pool partitions to be selected; obtaining the target computing power resource pool partition selected by the target user from the computing power resource pool partitions to be selected.

[0091] The computing power resource pool partition to be selected may refer to a computing power resource pool partition that meets the target user's job requirements for this operation.

[0092] Specifically, the target job task and target job requirement of the target user are obtained. The target computing power application and target computing power resource instance in the target job requirement are used to screen the candidate computing power resource pool partitions corresponding to the target user, and the candidate computing power resource pool partitions that have the target computing power application and meet the target computing power resource instance are used as the candidate computing power resource pool partitions, and the candidate computing power resource pool partitions are displayed to the target user. The target user selects a computing power resource pool partition from the candidate computing power resource pool partitions, so that the computing power application portal subsystem uses the computing power resource pool partition selected by the target user as the target computing power resource pool partition.

[0093] S330. Send the acquired target computing resource pool partition and target computing quota to the computing base platform through the computing power scheduling subsystem.

[0094] S340. Send the target job task to the target computing power resource pool partition through the computing power application portal subsystem.

[0095] S350. Control the target computing power resource pool partition to execute the target job task based on the target computing power quota through the computing power base platform.

[0096] The technical solution of the embodiment of the present invention is that if the computing power application portal subsystem detects that the target scheduling mode is the manual scheduling mode, the target job task, target job demand and target computing power resource pool partition of the target user are obtained; the acquired target computing power resource pool partition and target computing power quota are sent to the computing power base platform through the computing power scheduling subsystem; the target job task is sent to the target computing power resource pool partition through the computing power application portal subsystem, thereby supporting the manual scheduling mode, and scheduling the target job task of the target user to different computing power resource pools, thereby realizing flexible matching of user intentions, and further improving the flexibility of job task scheduling and user experience.

[0097] Embodiment 3

[0098] Figure 5 This is a flowchart of a method for scheduling a job task based on a computing resource pool provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment describes in detail the scheduling process of the transit agent scheduling mode. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0099] like Figure 5 As shown, the method includes:

[0100] S510. Obtain the target user identifier and target scheduling mode of the target user through the computing power application portal subsystem, and determine the candidate computing power resource pool partition and target computing power quota corresponding to the target user based on the target user identifier.

[0101] S520. If the computing power application portal subsystem detects that the target scheduling mode is the transit agent scheduling mode, the target job task and target job requirement of the target user are obtained.

[0102] Among them, in the transit agent scheduling mode, the computing power scheduling subsystem provides a public storage space for transit storage of all user job input files.

[0103] Specifically, Figure 6 A schematic diagram of the relay agent scheduling mode is given. Figure 6 ,The computing power application portal subsystem obtains the target computing power application, target computing power resource instance and expected execution strategy selected by the target user.

[0104] S530. The computing power scheduling subsystem performs partition screening on the candidate computing power resource pool partitions based on the acquired target job requirements, determines the target computing power resource pool partition that meets the target job requirements, and sends the target computing power resource pool partition and the target computing power quota to the computing power base platform.

[0105] Specifically, see Figure 6 The computing power scheduling subsystem determines the effective computing power resource pool-cluster-partition (equivalent to the target computing power resource pool partition) based on the target computing power application, target computing power resource instance, and expected execution strategy selected by the user, combined with the candidate computing power resource pool partition in the cluster partition list that the user is authorized to use.

[0106] S540. The computing power scheduling subsystem stores the acquired target job tasks in the public storage space, creates a target file directory corresponding to the target user in the target computing power resource pool partition, and sends the target job tasks to the target file directory.

[0107] Specifically, see Figure 6 The computing power scheduling subsystem creates a target file directory corresponding to the target user under the corresponding target computing power resource pool partition (if it has already been created, it will not be created again), and then the computing power scheduling subsystem automatically uploads the target user's job input file to the user's file directory.

[0108] It should be noted that in order to save the file storage space of the computing power scheduling subsystem, a cleanup policy can be set, such as deleting files 30 days after they are uploaded. In addition, after the job task is completed, the target user can directly download the job run output file (target execution result) from the target user's target file directory without transferring through the public storage space of the computing power scheduling subsystem.

[0109] S550. Control the target computing power resource pool partition to execute the target job task based on the target computing power quota through the computing power base platform.

[0110] The technical solution of the embodiment of the present invention is that if the computing power application portal subsystem detects that the target scheduling mode is the transit agent scheduling mode, the target job task and target job demand of the target user are obtained; the computing power scheduling subsystem performs partition screening on the candidate computing power resource pool partitions based on the obtained target job demand, determines the target computing power resource pool partition that meets the target job demand, and sends the target computing power resource pool partition and the target computing power quota to the computing power base platform; the computing power scheduling subsystem stores the obtained target job task in the public storage space, creates a target file directory corresponding to the target user in the target computing power resource pool partition, and sends the target job task to the target file directory, thereby ensuring that each user has an independent job file management space, which is isolated from each other, safe and controllable, thereby improving the security and confidentiality of user information and further improving the user experience.

[0111] The following is an embodiment of a job task scheduling system based on a computing power resource pool provided in an embodiment of the present invention. This system and the job task scheduling methods based on a computing power resource pool in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the job task scheduling system based on a computing power resource pool, please refer to the embodiment of the job task scheduling method based on a computing power resource pool.

[0112] Embodiment 4

[0113] Figure 7 This is a schematic diagram of the structure of a job task scheduling system based on a computing resource pool provided in the fourth embodiment of the present invention. Figure 7 As shown, the system includes: a computing power application portal subsystem 710, a computing power scheduling subsystem 720, a computing power base platform 730 and a full computing power resource pool 740; the full computing power resource pool 740 includes: a candidate computing power resource pool partition and a target computing power resource pool partition;

[0114] Among them, the computing power application portal subsystem 710 is used to obtain the target user identifier and target scheduling mode of the target user, and determine the candidate computing power resource pool partition and target computing power quota corresponding to the target user based on the target user identifier; the computing power application portal subsystem 710 is used to obtain the target job task and target job demand of the target user if it is detected that the target scheduling mode is the query scheduling mode; the computing power scheduling subsystem 720 is used to partition the candidate computing power resource pool partitions based on the acquired target job demand, determine the target computing power resource pool partition that meets the target job demand, and send the target computing power resource pool partition to the computing power application portal subsystem 710; the computing power application portal subsystem 710 is used to send the target computing power resource pool partition and the target computing power quota to the computing power base platform 730, and send the target job task to the target computing power resource pool partition; the computing power base platform 730 is used to control the target computing power resource pool partition to execute the target job task based on the target computing power quota.

[0115] The technical solution of the embodiment of the present invention obtains the target user identifier and target scheduling mode of the target user through the computing power application portal subsystem 710, and determines the candidate computing power resource pool partition and target computing power quota corresponding to the target user based on the target user identifier; if the computing power application portal subsystem 710 detects that the target scheduling mode is the query scheduling mode, the target job task and target job demand of the target user are obtained; the computing power scheduling subsystem 720 performs partition screening on the candidate computing power resource pool partition based on the acquired target job demand, determines the target computing power resource pool partition that meets the target job demand, and sends the target computing power resource pool partition to the computing power application portal subsystem 710; sends the target computing power resource pool partition and the target computing power quota to the computing power base platform 730 through the computing power application portal subsystem 710, and sends the target job task to the target computing power resource pool partition; the computing power base platform 730 controls the target computing power resource pool partition to execute the target job task based on the target computing power quota, thereby realizing the rapid and accurate scheduling of the job task to the effective computing power resource pool, thereby improving the scheduling flexibility and user experience.

[0116] Based on the above technical solutions, the target job requirements include: target computing power application, target computing power resource instance and expected execution strategy; the target scheduling modes include: manual scheduling mode, query scheduling mode and transit agent scheduling mode; query scheduling mode and transit agent scheduling mode belong to automatic scheduling mode.

[0117] On the basis of the above technical solution, the computing power application portal subsystem 710 is further used to obtain the target job task, target job requirement and target computing power resource pool partition of the target user if it is detected that the target scheduling mode is the manual scheduling mode;

[0118] The computing power scheduling subsystem 720 is also used to send the acquired target computing power resource pool partition and target computing power quota to the computing power base platform 730;

[0119] The computing power application portal subsystem 710 is also used to send the target job task to the target computing power resource pool partition.

[0120] Based on the above technical solution, the expected execution strategy in the target job requirement is a null value.

[0121] On the basis of the above technical solution, the computing power application portal subsystem 710 is also used to obtain the target operation tasks and target operation requirements of the target user;

[0122] The computing power application portal subsystem 710 is further used to screen the candidate computing power resource pool partitions corresponding to the target user based on the target computing power application and the target computing power resource instance in the target job requirement, determine the computing power resource pool partitions to be selected and display them;

[0123] The computing power application portal subsystem 710 is also used to obtain the target computing power resource pool partition selected by the target user from the computing power resource pool partitions to be selected.

[0124] On the basis of the above technical solution, the computing power base platform 730 is also used to determine the target execution time of the target operation task, and send the target execution result and target execution time corresponding to the target operation task to the computing power application portal subsystem 710;

[0125] The computing power application portal subsystem 710 is further used to determine the target execution cost corresponding to the target job task based on the target execution time and the target computing power resource instance in the target job requirement;

[0126] The computing power application portal subsystem 710 is also used to display the target execution result if it is detected that the target execution fee is obtained.

[0127] On the basis of the above technical solution, the computing power application portal subsystem 710 is further used to obtain the target job task and target job requirement of the target user if it is detected that the target scheduling mode is the transit agent scheduling mode;

[0128] The computing power scheduling subsystem 720 is further used to screen the candidate computing power resource pool partitions based on the acquired target job requirements, determine the target computing power resource pool partitions that meet the target job requirements, and send the target computing power resource pool partitions and the target computing power quota to the computing power base platform 730;

[0129] The computing power scheduling subsystem 720 is also used to store the acquired target job tasks in the public storage space, create a target file directory corresponding to the target user in the target computing power resource pool partition, and send the target job tasks to the target file directory.

[0130] On the basis of the above technical solution, the full computing power resource pool 740 is used to provide the underlying computing power resources;

[0131] The underlying computing resources include at least one of a central processing unit, a graphics processing unit, a memory space, a graphics card cache space, and a network.

[0132] Based on the above technical solution, each computing resource pool includes multiple clusters; each cluster is composed of a combination of multiple computing hardware resources; each cluster includes multiple partitions; and the computing resources in each partition have the same configuration.

[0133] The job task scheduling system based on computing power resource pool provided in an embodiment of the present invention can execute the job task scheduling method based on computing power resource pool provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the job task scheduling method based on computing power resource pool.

[0134] It is worth noting that in the above-mentioned embodiment of job task scheduling based on computing power resource pool, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for scheduling job tasks based on a computing resource pool, characterized in that: include: Obtaining a target user identifier and a target scheduling mode of a target user through a computing power application portal subsystem, and determining a candidate computing power resource pool partition and a target computing power quota corresponding to the target user based on the target user identifier; If the computing power application portal subsystem detects that the target scheduling mode is the query scheduling mode, the target job task and target job requirement of the target user are obtained; The computing power scheduling subsystem performs partition screening on the candidate computing power resource pool partitions based on the acquired target job requirements, determines the target computing power resource pool partitions that meet the target job requirements, and sends the target computing power resource pool partitions to the computing power application portal subsystem; The target computing power resource pool partition and the target computing power quota are sent to the computing power base platform through the computing power application portal subsystem, and the target job task is sent to the target computing power resource pool partition; The target computing power resource pool partition is controlled by the computing power base platform based on the target computing power quota to execute the target job task.

2. The method according to claim 1, characterized in that The target job requirements include: target computing power application, target computing power resource instance and expected execution strategy; the target scheduling mode includes: manual scheduling mode, query scheduling mode and transit agent scheduling mode; the query scheduling mode and the transit agent scheduling mode belong to automatic scheduling mode.

3. The method according to claim 1, characterized in that The method further includes: If the computing power application portal subsystem detects that the target scheduling mode is the manual scheduling mode, the target job task, target job requirement and target computing power resource pool partition of the target user are obtained; The target computing resource pool partition and target computing quota obtained are sent to the computing base platform through the computing power scheduling subsystem; The target job task is sent to the target computing power resource pool partition through the computing power application portal subsystem.

4. The method according to claim 3, characterized in that: The expected execution strategy in the target job requirement is a null value.

5. The method according to claim 3, characterized in that: The obtaining of the target job task, target job requirement and target computing resource pool partition of the target user includes: Obtain the target job tasks and target job requirements of the target user; Based on the target computing power application and the target computing power resource instance in the target job requirement, the candidate computing power resource pool partitions corresponding to the target user are screened, and the candidate computing power resource pool partitions are determined and displayed; Obtain the target computing power resource pool partition selected by the target user from the computing power resource pool partitions to be selected.

6. The method according to claim 1, characterized in that The method further includes: Determine the target execution time of the target job task through the computing power base platform, and send the target execution result corresponding to the target job task and the target execution time to the computing power application portal subsystem; Determine, by the computing power application portal subsystem, a target execution cost corresponding to the target job task based on the target execution time and the target computing power resource instance in the target job requirement; If the computing power application portal subsystem detects that the target execution fee is obtained, the target execution result will be displayed.

7. The method according to claim 1, characterized in that The method further includes: If the computing power application portal subsystem detects that the target scheduling mode is a transit agent scheduling mode, the target job task and target job requirement of the target user are obtained; The computing power scheduling subsystem performs partition screening on the candidate computing power resource pool partitions based on the acquired target job requirements, determines the target computing power resource pool partition that meets the target job requirements, and sends the target computing power resource pool partition and the target computing power quota to the computing power base platform; The acquired target job tasks are stored in the public storage space through the computing power scheduling subsystem, and a target file directory corresponding to the target user is created in the target computing power resource pool partition, and the target job tasks are sent to the target file directory.

8. A job task scheduling system based on computing resource pool, characterized in that: The operation task scheduling system based on the computing power resource pool includes: a computing power application portal subsystem, a computing power scheduling subsystem, a computing power base platform and a full computing power resource pool; the full computing power resource pool includes: a candidate computing power resource pool partition and a target computing power resource pool partition; The computing power application portal subsystem is used to obtain a target user identifier and a target scheduling mode of a target user, and determine a candidate computing power resource pool partition and a target computing power quota corresponding to the target user based on the target user identifier; The computing power application portal subsystem is used to obtain the target job task and target job requirement of the target user if it is detected that the target scheduling mode is the query scheduling mode; The computing power scheduling subsystem is used to screen the candidate computing power resource pool partitions based on the acquired target job requirements, determine the target computing power resource pool partitions that meet the target job requirements, and send the target computing power resource pool partitions to the computing power application portal subsystem; The computing power application portal subsystem is used to send the target computing power resource pool partition and the target computing power quota to the computing power base platform, and send the target job task to the target computing power resource pool partition; The computing power base platform is used to control the target computing power resource pool partition to execute the target job task based on the target computing power quota.

9. The system according to claim 8, characterized in that The full computing power resource pool is used to provide underlying computing power resources; The underlying computing resources include at least one of a central processing unit, a graphics processing unit, a memory space, a graphics card cache space, and a network.

10. The system according to claim 8, characterized in that Each computing resource pool includes multiple clusters; each cluster is composed of a variety of computing hardware resources; each cluster includes multiple partitions; the computing resources in each partition have the same configuration.

Images