Job scheduling method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a job scheduling method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring to-be-processed job information of a target user through a job scheduling server of a local cluster, wherein the to-be-processed job information comprises to-be-processed jobs, user information of the target user and a target job queue where the to-be-processed jobs are located; and the job scheduling server determines whether to schedule the to-be-processed job to the cloud cluster according to the to-be-processed job information, and if yes, the to-be-processed job is scheduled to the cloud cluster. Whether the to-be-processed job of the target user is scheduled to the cloud cluster or not can be automatically decided, the target user does not need to check the job condition of the local cluster before submitting the job, and the working efficiency of the user can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a job scheduling method, device, electronic device, storage medium and program product. Background Art

[0002] High-performance computing clusters (i.e. local clusters) have become the basic infrastructure of enterprises. After deploying a high-performance computing cluster, the computing power resource scale of the cluster is fixed. In actual applications, different resource pools (i.e. queues) can be divided according to different departments / sections. Usually, the resources of high-performance clusters can be managed through the job scheduling system. Specifically, it can be managed according to queues, that is, the user's job is submitted to the corresponding queue for queuing according to the department / section to which the user belongs.

[0003] Due to the different busyness of different departments / sections, the job queuing conditions of different queues are different, and it is easy for some queues to have jobs queued for a long time. In order to meet the job needs, you can connect to the cloud cluster, so that users can make independent decisions on whether to submit jobs to the cloud cluster for operation based on the job queuing conditions of the local cluster, which causes inconvenience to users and affects work efficiency. Summary of the invention

[0004] The present application provides a job scheduling method, device, electronic device, storage medium and program product to solve the problem in the related art that users independently decide whether to submit jobs to a cloud cluster for execution, causing inconvenience to users and affecting work efficiency.

[0005] In a first aspect, the present application provides a job scheduling method, which is applied to a job scheduling server of a local cluster, and the method comprises:

[0006] Acquire the pending job information of the target user, wherein the pending job information includes the pending job, the user information of the target user, and the target job queue where the pending job is located;

[0007] Determine whether to schedule the pending job to a cloud cluster according to the pending job information;

[0008] If yes, the pending job is dispatched to the cloud cluster.

[0009] In this embodiment, the job scheduling server of the local cluster can obtain the pending job information of the target user, which includes the pending job, the user information of the target user, and the target job queue where the pending job is located. The job scheduling server can determine whether to schedule the pending job to the cloud cluster based on the pending job information, and if so, schedule the pending job to the cloud cluster. This application can automatically decide whether to schedule the pending job of the target user to the cloud cluster. The target user does not need to check the job status of the local cluster before submitting the job, which can improve the user's work efficiency.

[0010] Optionally, the method as described above, obtaining the target user's pending job information, includes:

[0011] Obtaining a job request of a target user, wherein the job request includes the job to be processed and user information of the target user;

[0012] Allocating the pending job to a target job queue in the local cluster;

[0013] The job information to be processed is determined according to the job to be processed, the user information and the target job queue.

[0014] In this embodiment, after obtaining the target user's job request, a target job queue can be allocated to the target user's pending job, so that the pending job information can be determined, so that it can be determined whether to automatically allocate the pending job to the cloud cluster for operation based on the pending job information.

[0015] Optionally, the method as described above, determining whether to schedule the pending job to a cloud cluster according to the pending job information, includes:

[0016] Determining whether the pending job meets the cloud job conditions according to the pending job information;

[0017] If so, determine to schedule the pending job to the cloud cluster;

[0018] The cloud operation conditions include one or more of the following:

[0019] The number of queued jobs in the target job queue is greater than a first threshold;

[0020] The target job queue is a cloud job queue;

[0021] The number of occupied cores in the target job queue is greater than a second threshold;

[0022] The target user is a designated user in the cloud.

[0023] In this embodiment, it is possible to determine whether the pending job meets the cloud job conditions based on the pending job information. If so, it is possible to determine whether the pending job will be scheduled to the cloud cluster. By judging the cloud job conditions, job scheduling is automatically implemented to improve user experience and work efficiency.

[0024] Optionally, in the method described above, if the job information to be processed also includes node information of the target computing node specified by the target user;

[0025] The cloud operation conditions also include:

[0026] The target computing node is a computing node that is abnormally offline.

[0027] In this embodiment, it is possible to determine whether the cloud operation conditions are met based on the node information of the target computing node specified by the target user. If so, it is possible to determine that the pending job will be scheduled to the cloud cluster to avoid the situation where the target user's pending job is queued for a long time and cannot be run due to a failure of the target computing node.

[0028] Optionally, the method described above, determining whether the cloud operation condition is met according to the node information of the target computing node, includes:

[0029] Obtaining a faulty node configuration file, wherein the faulty node configuration file includes node information of multiple faulty computing nodes;

[0030] Determining, according to the node information of the target computing node, whether the target computing node belongs to the multiple faulty computing nodes;

[0031] If so, it is determined that the pending operation meets the cloud operation conditions.

[0032] In this embodiment, it is possible to determine whether the target computing node is a faulty node based on the node information of the recorded faulty computing node. If so, it can be determined that the pending job meets the cloud job conditions, and the target user's pending job can be scheduled to the cloud cluster for operation, thereby avoiding the situation where the target user's pending job is queued for a long time and cannot be run due to the failure of the target computing node.

[0033] Optionally, the method as described above further comprises:

[0034] If a faulty computing node is detected in the local cluster, the faulty computing node is taken offline;

[0035] Obtaining a faulty node configuration file, wherein the faulty node configuration file includes node information of multiple faulty computing nodes;

[0036] The faulty node configuration file is updated according to the node information of the faulty computing node to obtain an updated faulty node configuration file, and the updated faulty node configuration file is used to determine whether the job meets the cloud job condition.

[0037] In this embodiment, the computing nodes of the local cluster can be monitored, and when a fault occurs, it can be recorded and the faulty computing node can be taken offline, so that when the target user specifies the faulty computing node, the target user's pending jobs can be scheduled to the cloud cluster for operation.

[0038] In a second aspect, the present application provides a job scheduling device, which is applied to a job scheduling server of a local cluster, and the device includes:

[0039] An acquisition module, used to acquire the pending job information of the target user, wherein the pending job information includes the pending job, the user information of the target user, and the target job queue where the pending job is located;

[0040] A processing module, used for determining whether to schedule the pending job to a cloud cluster according to the pending job information;

[0041] The scheduling module is used to schedule the pending job to the cloud cluster if it is determined that the pending job is to be scheduled to the cloud cluster.

[0042] Optionally, in the above-mentioned device, the acquisition module is specifically used for:

[0043] Obtaining a job request of a target user, wherein the job request includes the job to be processed and user information of the target user;

[0044] Allocating the pending job to a target job queue in the local cluster;

[0045] The job information to be processed is determined according to the job to be processed, the user information and the target job queue.

[0046] Optionally, the apparatus and the processing module as described above are specifically used for:

[0047] Determining whether the pending job meets the cloud job conditions according to the pending job information;

[0048] If so, determine to schedule the pending job to the cloud cluster;

[0049] The cloud operation conditions include one or more of the following:

[0050] The number of queued jobs in the target job queue is greater than a first threshold;

[0051] The target job queue is a cloud job queue;

[0052] The number of occupied cores in the target job queue is greater than a second threshold;

[0053] The target user is a designated user in the cloud.

[0054] Optionally, in the above-mentioned device, if the to-be-processed job information also includes node information of the target computing node specified by the target user;

[0055] The cloud operation conditions also include:

[0056] The target computing node is a computing node that is abnormally offline.

[0057] Optionally, the apparatus and the processing module as described above are specifically used for:

[0058] Obtaining a faulty node configuration file, wherein the faulty node configuration file includes node information of multiple faulty computing nodes;

[0059] Determining, according to the node information of the target computing node, whether the target computing node belongs to the multiple faulty computing nodes;

[0060] If so, it is determined that the pending operation meets the cloud operation conditions.

[0061] Optionally, the device as described above further comprises a fault processing module, wherein the fault processing module is used to:

[0062] If a faulty computing node is detected in the local cluster, the faulty computing node is taken offline;

[0063] Obtaining a faulty node configuration file, wherein the faulty node configuration file includes node information of multiple faulty computing nodes;

[0064] The faulty node configuration file is updated according to the node information of the faulty computing node to obtain an updated faulty node configuration file, and the updated faulty node configuration file is used to determine whether the job meets the cloud job condition.

[0065] The protection device for the computing center provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar, which will not be described in detail here.

[0066] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0067] The memory stores computer-executable instructions;

[0068] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0069] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, they are used to implement the method described in the first aspect.

[0070] The computer-readable storage medium provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar, which will not be repeated here.

[0071] In a fifth aspect, the present application provides a computer program product, including a computer program, which is used to implement the method described in the first aspect when executed by a computer.

[0072] The computer program product provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar and will not be described in detail here.

[0073] The job scheduling method, device, electronic device, storage medium and program product provided by the present application obtain the pending job information of the target user through the job scheduling server of the local cluster, and the pending job information includes the pending job, the user information of the target user and the target job queue where the pending job is located. The job scheduling server determines whether to schedule the pending job to the cloud cluster based on the pending job information, and if so, schedules the pending job to the cloud cluster. The present application can automatically decide whether to schedule the pending job of the target user to the cloud cluster. The target user does not need to check the job status of the local cluster before submitting the job, which can improve the user's work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0075] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0076] Figure 2 A schematic diagram of a job scheduling scenario provided in an embodiment of the present application;

[0077] Figure 3 A flowchart of a job scheduling method provided in an embodiment of the present application;

[0078] Figure 4 A flowchart of another job scheduling method provided in an embodiment of the present application;

[0079] Figure 5 A flowchart of another method for scheduling a job provided in an embodiment of the present application;

[0080] Figure 6 A flowchart of another method for scheduling a job provided in an embodiment of the present application;

[0081] Figure 7 A schematic diagram of the structure of a job scheduling device provided in an embodiment of the present application;

[0082] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0083] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0084] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0085] After the deployment of the high-performance computing cluster (i.e., local cluster), different resource pools (i.e., queues) can be divided according to different departments / sections in actual applications. However, the computing power requirements of different departments / sections are constantly changing, and the computing power requirements fluctuate with the urgency of the job tasks. When there are many jobs, the jobs of the high-performance computing cluster will have a large number of job queues, which will seriously affect the work progress.

[0086] To solve this problem, the pending jobs can be dispatched to the cloud cluster for operation to alleviate the problem of job queuing in the local cluster. Specifically, users are required to query the job queuing status of the local cluster before submitting the job. Based on the job queuing status of the local cluster, they can make their own decisions on whether to submit the job to the cloud cluster for operation, which causes inconvenience to users and affects work efficiency.

[0087] Therefore, the present application provides a job scheduling method, in which the job scheduling system of the local cluster can automatically decide whether to schedule the target user's pending job to the cloud cluster based on the target user's pending job information. The target user does not need to check the job status of the local cluster before submitting the job, which can improve the user's work efficiency.

[0088] Combine the following Figure 1 and Figure 2 , the application scenarios to which the embodiments of the present application are applicable are described.

[0089] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, the local cluster 10 includes multiple computing nodes (such as Figure 1 101a, 101b, 101c are collectively referred to as 101) and the job scheduling server 102.

[0090] The job scheduling server 102 can schedule the pending job to the cloud cluster 20 through the job scheduling interface of the cloud cluster 20, and receive the job result after the cloud cluster 20 executes the pending job through the job scheduling interface. Exemplarily, the job scheduling interface can be implemented through an Internet protocol (IP) address and a network service port for file transfer between the local cluster 10 and the cloud cluster 20. The job scheduling server 102 can implement job scheduling based on the IP address and the network service port.

[0091] In a possible implementation, the IP address and network service port of the license authorization server may be configured in the job scheduling server 102. The IP address and network service port of the license authorization server may be used for the job scheduling server 102 to effectively manage and use software licenses, thereby ensuring the operation of jobs.

[0092] In a possible implementation, the job scheduling server 102 can query the job queuing status of the cloud cluster 20 through the access interface of the cloud cluster 20. This can ensure whether the pending job is scheduled to the cloud cluster 20 to ensure that the user's pending job is executed.

[0093] Figure 2 A schematic diagram of a job scheduling scenario provided in an embodiment of the present application, such as Figure 2 As shown, the user can submit a job to the job scheduling module 1021 of the job scheduling server 102, and the job scheduling module 1021 determines to submit the pending job to the queue of the local cluster 10 (for example Figure 2The job to be processed may be queued in queue A, queue B, queue C or queue D as shown in the example, or the job to be processed may be scheduled to the cloud cluster 20 through the cloud connection module 1022 of the job scheduling server 102. Specifically, the cloud connection module 1022 may schedule the job to be processed to the cloud cluster 20 through the cloud application programming interface (API) of the cloud cluster 20. Exemplarily, the cloud API may include an access interface and a job scheduling interface.

[0094] It can be understood that the user can submit a job to the job scheduling module 1021 of the job scheduling server 102 through the terminal device where the user is located.

[0095] like Figure 2 As shown, the cloud cluster 20 also includes cloud computing nodes and a cloud job scheduling server. The cloud job scheduling server is used to perform resource management and job scheduling on the cloud computing nodes.

[0096] It can be understood that in the embodiment of the present application, the computing node in the local cluster 10 or the computing node in the cloud cluster 20 can be a physical server or a physical or virtual server composed of multiple servers, and the present application does not impose any restrictions on this.

[0097] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0098] Figure 3 The flowchart of a method for scheduling a job provided in an embodiment of the present application is shown in FIG. 1 . The method can be executed by a job scheduling device, which can be implemented by a computer program; it can also be implemented by a medium storing a relevant computer program, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with a relevant computer program, such as a chip or an electronic device. The electronic device can be, for example, Figure 1 The job scheduling server 102 of the local cluster 10 shown in FIG. Figure 3 As shown, the method may include the following steps.

[0099] S301. Obtain information about pending jobs of a target user, where the information about pending jobs includes the pending jobs, user information of the target user, and a target job queue where the pending jobs are located.

[0100] The job scheduling server may obtain the pending job information of the target user, where the pending job information includes the pending job, user information of the target user, and the target job queue where the pending job is located.

[0101] The jobs to be processed may be files such as job scripts.

[0102] The user information of the target user may include, for example, the user ID, the user's name, the user's department / section, etc.

[0103] The target job queue may be a queue allocated to the target user by the job scheduling server after receiving the job request extracted by the target user.

[0104] In a possible implementation, the job scheduling server may obtain a job request from a target user, the job request including the job to be processed and the user information of the target user, and then schedule the job to be processed to a target job queue in a local cluster, and determine the job information to be processed according to the job to be processed, the user information and the target job queue.

[0105] The job scheduling server schedules pending jobs to the target job queue in the local cluster in the following ways, for example:

[0106] Method 1: The department / section to which the target user belongs may be determined based on the user information of the target user, and the pending job may be allocated to the target job queue corresponding to the department / section to which the target user belongs.

[0107] Method 2: Randomly assign pending jobs to any queue in the local cluster as the target job queue.

[0108] Method 3: Allocate pending jobs to the cloud job queue, that is, use the cloud job queue as the target job queue. Jobs in the cloud job queue are scheduled to the cloud cluster first.

[0109] After allocating the target job queue for the target user's pending job, the job scheduling server can determine the pending job information, so that it can determine whether to automatically allocate the pending job to the cloud cluster for operation based on the pending job information.

[0110] S302: Determine whether to allocate the pending job to a cloud cluster according to the pending job information.

[0111] The job scheduling server can determine whether to allocate the pending jobs to the cloud cluster based on the pending job information.

[0112] If yes, execute S303, if no, execute S304.

[0113] Exemplarily, the job scheduling server may determine whether to allocate the pending job to the cloud cluster based on the pending job by, for example, determining whether to allocate the pending job to the cloud cluster based on user information of the target user or the target job queue.

[0114] S303: Schedule the pending jobs to the cloud cluster.

[0115] If it is determined to schedule the pending job to the cloud cluster, the job scheduling server may schedule the pending job to the cloud cluster through the job scheduling interface of the cloud cluster.

[0116] Exemplarily, the job scheduling server may submit the pending job to a designated directory of the cloud cluster through the job scheduling interface of the cloud cluster, and the cloud job scheduling server may schedule the pending job to run on a corresponding node.

[0117] S304: Schedule the pending jobs to the local cluster.

[0118] If it is determined not to schedule the pending jobs to the cloud cluster, the job scheduling server may schedule the pending jobs to run on the computing nodes of the local cluster.

[0119] For example, if the target job queue where the job to be processed is located is queued, the job needs to be queued in the target job queue to wait for being dispatched to a corresponding computing node for execution.

[0120] In this embodiment, the job scheduling server of the local cluster can obtain the pending job information of the target user, which includes the pending job, the user information of the target user, and the target job queue where the pending job is located. The job scheduling server can determine whether to schedule the pending job to the cloud cluster based on the pending job information, and if so, schedule the pending job to the cloud cluster. This application can automatically decide whether to schedule the pending job of the target user to the cloud cluster. The target user does not need to check the job status of the local cluster before submitting the job, which can improve the user's work efficiency.

[0121] Next, how the job scheduling server determines whether to schedule the pending job to the cloud cluster based on the pending job is described.

[0122] Figure 4 A flowchart of another method for scheduling a job provided in an embodiment of the present application is provided. The method can be executed by a job scheduling device, which can be implemented by a computer program; it can also be implemented by a medium storing a relevant computer program, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with a relevant computer program, such as a chip or an electronic device. The electronic device can be, for example, Figure 1 The job scheduling server 102 of the local cluster 10 shown in FIG. Figure 4 As shown, the method may include the following steps.

[0123] S401, obtaining information of pending jobs.

[0124] S402: Determine whether the pending job meets the cloud job conditions based on the pending job information.

[0125] If yes, execute S402.

[0126] The cloud operation conditions include one or more of the following:

[0127] The number of queued jobs in the target job queue is greater than a first threshold.

[0128] The target job queue is the cloud job queue.

[0129] The number of occupied cores in the target job queue is greater than a second threshold.

[0130] The target user is the specified user in the cloud.

[0131] If the number of queued jobs in the target job queue is greater than the first threshold, the pending jobs are dispatched to the cloud cluster for processing, which can solve the problem that the pending jobs currently occupy the local cluster for a long time and cannot be completed quickly.

[0132] If the number of occupied cores in the target job queue is greater than the second threshold, the pending job will be dispatched to the cloud cluster for operation, which can solve the problem of idle local cluster resources or large resource fragmentation caused by the pending jobs queuing in the local cluster. In other words, if the number of occupied cores corresponding to the jobs in the target job queue is large, if the number of occupied cores required by the pending jobs is also large, if the pending jobs continue to queue in the target job queue, the pending jobs may queue for a long time without getting resources, resulting in a waste of some of the remaining resources of the target job queue (these remaining resources are not enough to run the jobs).

[0133] The number of cores refers to the number of central processing unit (CPU) cores.

[0134] If the target job queue is a cloud job queue, the pending job is dispatched to the cloud cluster for operation, and the pending job of the target user can be submitted to the cloud cluster first. For example, if the application software corresponding to the pending job requires specific hardware resources (such as large memory resources), and the computing power cost of the local cluster is high and there is a long queue, by allocating the pending job to the cloud job queue, the cloud cluster with relatively low computing power cost can be used to improve the job processing speed.

[0135] In a possible implementation, the job scheduling server can sense the job queue status of the cloud cluster through the access interface of the cloud cluster. If there is a job queue status in the cloud cluster or the number of jobs in the queue is greater than a certain value, the job scheduling server can schedule the jobs in the cloud job queue to the local cluster for execution. Exemplarily, the job scheduling server can assign the jobs in the cloud job queue to the job queue with a smaller number of jobs in the queue, so as to wait for scheduling to the corresponding computing node for execution.

[0136] If the user information indicates that the target user is a designated user on the cloud, for example, the department / section where the user is located in the user information indicates that the target user is a designated user on the cloud, it means that all users in the department / section where the target user is located are designated users on the cloud. In this way, the pending jobs of the target user can be scheduled to the cloud cluster for execution. This allows the jobs of certain departments / sections to be executed through the specific resources of the cloud cluster when the resources of the local cluster cannot meet the demand, thereby meeting the task requirements of the department / section.

[0137] In a possible implementation, the job submitted by the designated user on the cloud may not use the resources of the local cluster. That is, for the job submitted by the designated user on the cloud, the job scheduling server directly submits the job to the cloud cluster.

[0138] S403: Determine to dispatch the pending job to the cloud cluster.

[0139] If the target user's pending job meets the cloud job conditions, it can be determined that the processing job will be scheduled to the cloud cluster.

[0140] In a possible implementation, if the number of queued jobs in the target job queue is greater than a first threshold, all jobs in the target job queue after the first threshold can be dispatched to the cloud cluster for operation. For example, if the number of queued jobs in the target job queue is 10 and the first threshold is 5, then the jobs corresponding to the sixth to tenth jobs in the queue can be submitted to the cloud cluster, which can reduce the queuing time of these jobs.

[0141] In this embodiment, it is possible to determine whether the pending job meets the cloud job conditions based on the pending job information. If so, it is possible to determine whether the pending job will be scheduled to the cloud cluster. By judging the cloud job conditions, job scheduling is automatically implemented to improve user experience and work efficiency.

[0142] In a possible implementation, if the job information to be processed also includes node information of the target computing node specified by the target user, for example, the target user can specify the target computing node in the job request when initiating the job request. Then the cloud job condition may also include: the target computing node is an abnormally offline computing node.

[0143] That is to say, if the target computing node specified by the target user is an abnormally offline computing node, since the target computing node has been offline due to a fault (i.e., unavailable), if the target user's pending job is added to the queue where the target computing node is located for queuing, the target user's pending job will be queued for a long time. When the target computing node specified by the target user is an abnormally offline computing node, the job scheduling server can submit the pending job to the cloud cluster, which can avoid the situation where the target user's pending job is queued for a long time and cannot be run due to the target computing node being an abnormally offline computing node.

[0144] Next, it is described how the job scheduling server determines whether the cloud job conditions are met based on the node information of the target computing node.

[0145] Figure 5 A flowchart of another method for scheduling a job provided in an embodiment of the present application is provided. The method can be executed by a job scheduling device, which can be implemented by a computer program; it can also be implemented by a medium storing a relevant computer program, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with a relevant computer program, such as a chip or an electronic device. The electronic device can be, for example, Figure 1 The job scheduling server 102 of the local cluster 10 shown in FIG. Figure 5 As shown, the method may include the following steps.

[0146] S501: Obtain a faulty node configuration file, where the faulty node configuration file includes node information of multiple faulty computing nodes.

[0147] The job scheduling server can obtain a faulty node configuration file, which includes node information of multiple faulty computing nodes. The faulty node configuration file is obtained by the job scheduling server monitoring the computing nodes of the local cluster, so that when it is determined that a computing node has a fault, the node information of the computing node is written into a pre-configured configuration file.

[0148] In a possible implementation, the job scheduling server may update the configuration file of the faulty node in the following manner:

[0149] If a faulty computing node is detected in the local cluster, the faulty computing node is taken offline. The faulty node configuration file is obtained, which includes node information of multiple faulty computing nodes. Then, the faulty node configuration file is updated according to the node information of the faulty computing node to obtain an updated faulty node configuration file, which is used to determine whether the job meets the cloud job conditions.

[0150] It can be understood that the job here refers to the job submitted by the user.

[0151] Exemplarily, the job scheduling server may write the node information of the faulty computing node into the faulty node configuration file, thereby updating the faulty node configuration file.

[0152] Exemplarily, the node information of a computing node (including a target computing node or a faulty computing node) may include a node identifier, a node name, network information (eg, an IP address, a physical address, etc.), and the like.

[0153] By monitoring the computing nodes of the local cluster, recording when a fault occurs and taking the faulty computing node offline, when the target user specifies the faulty computing node, the target user's pending jobs can be scheduled to the cloud cluster for execution.

[0154] S502: Determine, according to the node information of the target computing node, whether the target computing node belongs to multiple faulty computing nodes.

[0155] If yes, it is determined that the pending job meets the cloud job conditions, that is, S503 is executed. If no, it can be determined that the pending job does not meet the cloud job conditions, that is, S504 is executed. If the pending job does not meet other cloud job conditions, the job scheduling server can schedule the pending job to run in the local cluster.

[0156] S503: Determine whether the job to be processed meets the cloud job conditions.

[0157] S504: Determine that the pending job does not meet the cloud job conditions.

[0158] In this embodiment, it is possible to determine whether the target computing node is a faulty node based on the node information of the recorded faulty computing node. If so, it can be determined that the pending job meets the cloud job conditions, and the target user's pending job can be scheduled to the cloud cluster for operation, thereby avoiding the situation where the target user's pending job is queued for a long time and cannot be run due to the failure of the target computing node.

[0159] Next, another job scheduling method provided in an embodiment of the present application is described.

[0160] Figure 6 A flowchart of another method for scheduling a job provided in an embodiment of the present application is provided. The method can be executed by a job scheduling device, which can be implemented by a computer program; it can also be implemented by a medium storing a relevant computer program, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with a relevant computer program, such as a chip or an electronic device. The electronic device can be, for example, Figure 1 The job scheduling server 102 of the local cluster 10 shown in FIG. Figure 6 As shown, the method may include the following steps.

[0161] S601: The job scheduling server obtains a job request from a target user, where the job request includes the job to be processed and user information of the target user.

[0162] S602: The job scheduling server allocates the pending job to a target job queue in the local cluster.

[0163] S603: The job scheduling server determines information of the job to be processed according to the job to be processed, user information and the target job queue.

[0164] S604: The job scheduling server determines whether the pending job meets the cloud job conditions based on the pending job information.

[0165] If yes, then S605 is executed; if no, then it can be determined to schedule the to-be-processed job to the local cluster.

[0166] S605: The job scheduling server sends the pending jobs to the cloud cluster through the job scheduling interface of the cloud cluster, so that the pending jobs are submitted to the designated directory of the cloud cluster.

[0167] S606. The cloud job scheduling server schedules the pending job to the cloud computing node for execution and obtains the execution result.

[0168] S607: The cloud job scheduling server sends the operation result to the job scheduling server through the job scheduling interface of the cloud cluster, so that the operation result is sent to the designated directory of the local cluster.

[0169] In one possible implementation, the job scheduling server can sense whether the calculation of pending jobs in the cloud is completed through the access interface of the cloud cluster. When it is determined that the calculation is completed, the operation results can be downloaded to the specified directory of the local cluster through the job scheduling interface.

[0170] S608: When the job scheduling server detects that there is a faulty computing node in the local cluster, the faulty computing node is taken offline.

[0171] S609: Obtain a faulty node configuration file, where the faulty node configuration file includes node information of multiple faulty computing nodes.

[0172] S610: Update the faulty node configuration file according to the node information of the faulty computing node to obtain an updated faulty node configuration file.

[0173] The updated fault node configuration file is used to determine whether the pending job meets the cloud job conditions.

[0174] It can be understood that the present application does not limit the execution order of S608-S610 relative to S601-S607.

[0175] For the specific implementation and beneficial effects of the above steps, please refer to the above embodiments, which will not be described again here.

[0176] Figure 7 The structure diagram of a job scheduling device provided in an embodiment of the present application is shown in FIG. 1 , which can be applied to a job scheduling server of a local cluster. Figure 7 As shown, the device 70 includes: an acquisition module 701, a processing module 702 and a scheduling module 703.

[0177] The acquisition module 701 is used to acquire the to-be-processed job information of the target user, wherein the to-be-processed job information includes the to-be-processed job, the user information of the target user, and the target job queue where the to-be-processed job is located.

[0178] The processing module 702 is used to determine whether to schedule the job to be processed to the cloud cluster according to the information of the job to be processed.

[0179] The scheduling module 703 is used to schedule the pending job to the cloud cluster if it is determined that the pending job is to be scheduled to the cloud cluster.

[0180] In a possible implementation, the acquisition module 701 is specifically configured to:

[0181] Get the target user's job request, which includes the job to be processed and the target user's user information.

[0182] Dispatches pending jobs to the target job queue in the local cluster.

[0183] According to the pending jobs, user information and the target job queue, information about the pending jobs is determined.

[0184] In a possible implementation, the processing module 702 is specifically configured to:

[0185] Determine whether the pending job meets the cloud job conditions based on the pending job information.

[0186] If so, it is determined that the pending job is scheduled to the cloud cluster.

[0187] The cloud operation conditions include one or more of the following:

[0188] The number of queued jobs in the target job queue is greater than a first threshold.

[0189] The target job queue is the cloud job queue.

[0190] The number of occupied cores in the target job queue is greater than a second threshold.

[0191] The target user is the specified user in the cloud.

[0192] In a possible implementation manner, if the job information to be processed also includes node information of a target computing node specified by a target user.

[0193] Cloud operation conditions also include:

[0194] The target computing node is a computing node that has abnormally gone offline.

[0195] In a possible implementation, the processing module 702 is specifically configured to:

[0196] A faulty node configuration file is obtained, where the faulty node configuration file includes node information of multiple faulty computing nodes.

[0197] According to the node information of the target computing node, it is determined whether the target computing node belongs to the computing nodes with multiple faults.

[0198] If so, it is determined that the pending job meets the cloud job conditions.

[0199] In a possible implementation, the device further includes a fault processing module 704, where the fault processing module 704 is configured to:

[0200] If a faulty computing node is detected in the local cluster, the faulty computing node will be taken offline.

[0201] A faulty node configuration file is obtained, where the faulty node configuration file includes node information of multiple faulty computing nodes.

[0202] According to the node information of the faulty computing node, the faulty node configuration file is updated to obtain an updated faulty node configuration file, and the updated faulty node configuration file is used to determine whether the job meets the cloud job condition.

[0203] The job scheduling device provided in the embodiment of the present application can execute the technical solution corresponding to the job scheduling server of the local cluster in the above method embodiment, and its beneficial effects are similar and will not be repeated here.

[0204] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device 80 may include: at least one processor 801 and a memory 802 .

[0205] The memory 802 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer-executable instructions.

[0206] The memory 802 may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0207] The processor 801 is used to execute the computer-executable instructions stored in the memory 802 to implement the method described in the above method embodiment. The processor 801 may be a CPU, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0208] Optionally, the electronic device 80 may further include: a communication interface 803. In a specific implementation, if the communication interface 803, the memory 802 and the processor 801 are implemented independently, the communication interface 803, the memory 802 and the processor 801 may be interconnected through a bus and communicate with each other. The bus may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0209] Optionally, in a specific implementation, if the communication interface 803, the memory 802 and the processor 801 are integrated on a chip, the communication interface 803, the memory 802 and the processor 801 can communicate through an internal interface.

[0210] The electronic device 80 may be a server, such as a job scheduling server of a local cluster.

[0211] The electronic device of this embodiment can be used to execute the technical solution of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0212] An embodiment of the present application provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a disk or an optical disk, and other media that can store computer-executable instructions. Specifically, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0213] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0214] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0215] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0216] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0217] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0218] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processing technology (DSP) chip, and an application-specific integrated circuit (ASIC), etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0219] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard disks, magnetic disks or optical disks.

[0220] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0221] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0222] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A job scheduling method, characterized in that: The method is applied to a job scheduling server of a local cluster, and comprises: Acquire the pending job information of the target user, wherein the pending job information includes the pending job, the user information of the target user, and the target job queue where the pending job is located; Determine whether to schedule the pending job to a cloud cluster according to the pending job information; If yes, the pending job is dispatched to the cloud cluster.

2. The method according to claim 1, characterized in that Get the target user's pending job information, including: Obtaining a job request of a target user, wherein the job request includes the job to be processed and user information of the target user; Allocating the pending job to a target job queue in the local cluster; The job information to be processed is determined according to the job to be processed, the user information and the target job queue.

3. The method according to claim 1 or 2, characterized in that: Determining whether to schedule the pending job to a cloud cluster according to the pending job information includes: Determining whether the pending job meets the cloud job conditions according to the pending job information; If so, determine to schedule the pending job to the cloud cluster; The cloud operation conditions include one or more of the following: The number of queued jobs in the target job queue is greater than a first threshold; The target job queue is a cloud job queue; The number of occupied cores in the target job queue is greater than a second threshold; The target user is a designated user in the cloud.

4. The method according to claim 3, characterized in that If the to-be-processed job information also includes node information of the target computing node specified by the target user; The cloud operation conditions also include: The target computing node is a computing node that is abnormally offline.

5. The method according to claim 4, characterized in that Based on the node information of the target computing node, determine whether it meets the cloud operation conditions, including: Obtaining a faulty node configuration file, wherein the faulty node configuration file includes node information of multiple faulty computing nodes; Determining, according to the node information of the target computing node, whether the target computing node belongs to the multiple faulty computing nodes; If so, it is determined that the pending operation meets the cloud operation conditions.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: If a faulty computing node is detected in the local cluster, the faulty computing node is taken offline; Obtaining a faulty node configuration file, wherein the faulty node configuration file includes node information of multiple faulty computing nodes; The faulty node configuration file is updated according to the node information of the faulty computing node to obtain an updated faulty node configuration file, and the updated faulty node configuration file is used to determine whether the job meets the cloud job condition.

7. A job scheduling device, characterized in that: A job scheduling server applied to a local cluster, the device comprising: An acquisition module, used to acquire the pending job information of the target user, wherein the pending job information includes the pending job, the user information of the target user, and the target job queue where the pending job is located; A processing module, used for determining whether to schedule the pending job to a cloud cluster according to the pending job information; The scheduling module is used to schedule the pending job to the cloud cluster if it is determined that the pending job is to be scheduled to the cloud cluster.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images