Cluster job scheduling method

Abstract

The present invention discloses a cluster job scheduling method, comprising: obtaining the total number of queued jobs and the number of remaining submittable jobs corresponding to each computing node in a platform database, and determining the total number of submittable jobs; if the total number of queued jobs is less than or equal to the total number of submittable jobs, taking each queued job as a job to be assigned, and determining the node assignment job corresponding to each computing node according to the total number of queued jobs, the number of remaining submittable jobs, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned; otherwise, determining the job to be assigned according to the number of remaining submittable jobs, and determining the node assignment job corresponding to each computing node according to the number of remaining submittable jobs, the current estimated duration, and the estimated duration of each job; submitting the corresponding node assignment job to the computing node for each computing node; and updating the running node and job status corresponding to each node assignment job in the platform database, thereby achieving effective load balancing within the cluster.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a cluster job scheduling method. Background Art

[0002] Currently, job scheduling and resource management issues have become hot topics. Job scheduling allocates computing resources of cluster nodes to jobs. In this process, inappropriate job scheduling strategies and unreasonable cluster size will affect the load balancing status of the cluster.

[0003] However, the lack of a job scheduling strategy targeting load balancing will have multiple impacts on system performance, stability, and user experience.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a cluster job scheduling method, which realizes effective load balancing within the cluster.

[0006] An embodiment of the present invention provides a cluster job scheduling method, the method comprising:

[0007] Obtain the total number of queued jobs in the platform database and the number of remaining submittable jobs corresponding to each computing node in the platform database, and determine the total number of submittable jobs based on the number of remaining submittable jobs corresponding to each computing node;

[0008] In response to the total number of queued jobs being less than or equal to the total number of submittable jobs, each queued job is treated as a job to be assigned, and a node assignment job corresponding to each computing node is determined based on the total number of queued jobs, the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned;

[0009] In response to the total number of queued jobs being greater than the total number of submittable jobs, determining the jobs to be assigned based on the number of remaining submittable jobs corresponding to each computing node, and determining the node assignment jobs corresponding to each computing node based on the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned;

[0010] For each computing node, submit the node allocation job corresponding to the computing node to the computing node;

[0011] Update the running nodes and job status corresponding to each node allocation job in the platform database.

[0012] The embodiments of the present invention have the following technical effects:

[0013] By obtaining the total number of queued jobs in the platform database and the number of remaining submittable jobs corresponding to each computing node in the platform database, the total number of submittable jobs is determined based on the remaining submittable jobs corresponding to each computing node. If the total number of queued jobs is less than or equal to the total number of submittable jobs, each queued job is used as a job to be assigned. The node assignment job corresponding to each computing node is determined based on the total number of queued jobs, the remaining number of submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned. Otherwise, the job to be assigned is determined based on the remaining number of submittable jobs corresponding to each computing node. The node assignment job corresponding to each computing node is determined based on the remaining number of submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned, so as to provide an effective load balancing strategy and improve the effect of job assignment. For each computing node, the node assignment job corresponding to the computing node is submitted to the computing node, and the running node and job status corresponding to each node assignment job in the platform database are updated, thereby achieving effective load balancing within the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 This is a flow chart of a cluster job scheduling method provided by an embodiment of the present invention;

[0016] Figure 2 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0018] The cluster job scheduling method provided by the embodiment of the present invention is mainly applicable to the situation of load balancing of computing nodes in combination with the characteristics of various systems. The cluster job scheduling method provided by the embodiment of the present invention can be executed by an electronic device.

[0019] Example 1

[0020] Figure 1 This is a flow chart of a cluster job scheduling method provided by an embodiment of the present invention. Figure 1 , the cluster job scheduling method specifically includes:

[0021] S110 , obtaining the total number of queued jobs in the platform database and the number of remaining submittable jobs corresponding to each computing node in the platform database, and determining the total number of submittable jobs according to the number of remaining submittable jobs corresponding to each computing node.

[0022] The platform database is used to record relevant information about each submitted job, such as user ID, job ID, job name, script path, job status, running node, and estimated job duration. It also records the jobs currently being executed on each compute node and their corresponding current estimated duration. A compute node is a node in the cluster that performs job computations and processing tasks. The number of remaining submittable jobs is the number of jobs that a compute node can currently accept and execute. The total number of submittable jobs is the number of jobs that can still be submitted for processing within the cluster and is the sum of the number of remaining submittable jobs corresponding to each compute node.

[0023] Specifically, the number of jobs in the platform database with a queued status is obtained as the total number of queued jobs. The remaining number of submittable jobs corresponding to each compute node in the platform database is also obtained. For each compute node, the difference between the number of valid compute cores corresponding to the compute node and the number of jobs occupied by the node is used as the remaining number of submittable jobs for the compute node. The sum of the remaining number of submittable jobs corresponding to each compute node is then used to determine the total number of submittable jobs.

[0024] Based on the above example, when a user submits a job to the cluster, information extraction and preliminary processing of the submitted job is required and recorded in the platform database. Specifically, the following steps can be performed:

[0025] Submit the queued job through the target browsing interface, obtain the corresponding identification information of the queued job, set the job status of the queued job to queued, and set the running node of the queued job to empty;

[0026] Determine the target historical job corresponding to the queued job based on the software types and software versions corresponding to the queued job;

[0027] For each target historical job, determine the job similarity between the queued job and the target historical job based on the queued job and the target historical job;

[0028] Determine the estimated duration of the queued job based on the historical calculation duration of each target historical job and the job similarity between the queued job and each target historical job;

[0029] Update the platform database based on the identification information, job status, running node, and estimated job duration corresponding to the queued jobs.

[0030] The target browsing interface is a browser interface that users can view and operate. It can be used to control the cluster, submit jobs, and view job status. Queued jobs are jobs submitted by users. These jobs are in the queued state upon submission and are subsequently assigned to a corresponding compute node for processing, becoming node-assigned jobs. Identification information includes user ID, job ID, and job name. The job status is the current status of the job, which can be queued, running, successful, or failed. The running node is the compute node assigned to the job, which can be the node ID or the IP (Internet Protocol) address of the compute node. The software type is the type of software required by the queued job, and the software version is the version number of the software type corresponding to the queued job. The target historical job is a historical job that uses the same software type and version as the queued job. Job similarity is a comprehensive evaluation of the similarity between the target historical job and the queued job in terms of the various files included. The historical computation time is the time it took for the historical job to complete and successfully process.

[0031] Specifically, users can submit queued jobs through the target browsing interface. Upon receiving a queued job, the user ID submitting the queued job, the job ID, and the job name of the queued job are obtained and used as the corresponding identification information for the queued job. Furthermore, the job status of the queued job is set to queued, and the running node of the queued job is set to null. The queued job is analyzed to determine its corresponding software types and software versions. The portion of the cluster's historical jobs that share the same software types and software versions is used as the target historical jobs corresponding to the queued job. For each target historical job, the job similarity between the queued job and the target historical job is determined based on the queued job and the target historical job. The various files of the queued job and the various files of each target historical job are analyzed to calculate the job similarity between the queued job and each target historical job. A weight is determined for each target historical job based on the proportion of each job similarity to the total job similarity. Based on the weight of each target historical job, the historical computation duration of each target historical job is weighted and summed to obtain the estimated job duration for the queued job. Update the identification information, job status, running nodes, and estimated job duration corresponding to the queued jobs submitted by users to the platform database to facilitate subsequent allocation of computing nodes.

[0032] Based on the above example, you can determine the job similarity between the queued job and the target historical job using the following method:

[0033] determining a first similarity based on the queued job script file of the queued job and the historical job script file of the target historical job;

[0034] determining a second similarity based on the queued job model file of the queued job and the historical job model file of the target historical job;

[0035] The job similarity between the queued job and the target historical job is determined based on the first similarity and the second similarity.

[0036] Among them, queued jobs include queued job script files and queued job model files, and target historical jobs include historical job script files and historical job model files. Script files are text files containing executable code and are used to define operational logic or task processes. Model files are files that store static data structures that describe object properties (such as geometry, material, and hierarchical relationships) and do not contain executable logic. The first similarity is the similarity between the queued job script file and the historical job script file. The second similarity is the similarity between the queued job model file and the historical job model file.

[0037] Specifically, a similarity calculation is performed between the queued job script file of the queued job and the historical job script file of the target historical job, which can be any existing similarity calculation method, to determine a first similarity between the script files. A similarity calculation is performed between the queued job model file of the queued job and the historical job model file of the target historical job, which can be any existing similarity calculation method, to determine a second similarity between the model files. The first similarity and the second similarity are weighted and summed according to preset script file weights and model file weights to determine the job similarity between the queued job and the target historical job.

[0038] Based on the above example, the first similarity can be determined according to the queued job script file of the queued job and the historical job script file of the target historical job in the following manner:

[0039] Determining a first number of repeated character strings according to a queued job script file of the queued job and a historical job script file of the target historical job;

[0040] The ratio of the first number of repeated character strings to the number of character strings in the historical job script file is used as a first ratio, and the ratio of the first number of repeated character strings to the number of character strings in the queued job script file is used as a second ratio;

[0041] A first similarity is determined according to the first ratio and the second ratio.

[0042] The first number of repeated strings is the number of identical strings between the queued job script file and the target historical job and the historical job script file. The first ratio is the ratio of the first number of repeated strings to the number of strings in the historical job script file. The second ratio is the ratio of the first number of repeated strings to the number of strings in the queued job script file.

[0043] Specifically, the queued job script file of the queued job and the historical job script file of the target historical job are compared, and the number of repeated strings is used as a first repeated string count. The ratio of the first repeated string count to the number of strings in the historical job script file is used as a first ratio, and the ratio of the first repeated string count to the number of strings in the queued job script file is used as a second ratio. Furthermore, the first ratio and the second ratio can be combined to obtain a first similarity. This can be achieved by multiplying the first ratio and the second ratio as a first process value, the sum of the first ratio and the second ratio as a second process value, and then multiplying the quotient of the first process value and the second process value by 2 to obtain the first similarity.

[0044] Based on the above example, the second similarity can be determined according to the queued job model file of the queued job and the historical job model file of the target historical job in the following manner:

[0045] In response to the queued job model file being a binary model file, the queued job model file of the queued job is divided into blocks according to a preset block size to obtain a plurality of first blocks, and the historical job model file of the target historical job is divided into blocks to obtain a plurality of second blocks; a number of repeated blocks is determined based on each of the first blocks and each of the second blocks; a ratio of the number of repeated blocks to the total number of blocks in the first blocks is used as a third ratio, and a ratio of the number of repeated blocks to the total number of blocks in the second blocks is used as a fourth ratio;

[0046] In response to the queued job model file being a non-binary model file, determining a second number of repeated character strings based on the queued job model file of the queued job and the historical job model file of the target historical job; using a ratio of the second number of repeated character strings to the number of character strings in the historical job model file as a third ratio, and using a ratio of the second number of repeated character strings to the number of character strings in the queued job model file as a fourth ratio;

[0047] A second similarity is determined according to the third ratio and the fourth ratio.

[0048] Among them, the binary model file is a model file obtained by encoding binary characters, and the non-binary model file is a model file obtained by encoding characters other than binary characters. The preset block size is a pre-set block size for block processing, for example, 1k. The first block is each block obtained by block processing the queued job model file according to the preset block size. The second block is each block obtained by block processing the historical job model file according to the preset block size. The number of repeated blocks is the number of identical blocks between the set of first blocks and the set of second blocks. The second number of repeated character strings is the number of identical character strings between the queued job model file and the target historical job and the historical job model file. The third ratio is the ratio of the number of repeated blocks to the total number of blocks in the first block, or the ratio of the second number of repeated character strings to the number of character strings in the historical job model file. The fourth ratio is the ratio of the number of repeated blocks to the total number of blocks in the second block, or the ratio of the second number of repeated character strings to the number of character strings in the queued job model file.

[0049] Specifically, the file type of the queued job model file is determined. If the queued job model file is a binary model file, the calculation cannot be performed based on repeated character strings. The queued job model file of the queued job must be segmented according to a preset block size to obtain multiple first segments. Furthermore, the historical job model file of the target historical job must be segmented to obtain multiple second segments. The repeated segments between each first segment and each second segment are analyzed to obtain a number of repeated segments. The ratio of the number of repeated segments to the total number of segments in the first segment is used as the third ratio, and the ratio of the number of repeated segments to the total number of segments in the second segment is used as the fourth ratio. If the queued job model file is a non-binary model file, similar to the processing of the job script file, the queued job model file of the queued job and the historical job model file of the target historical job are compared, and the number of repeated character strings is used as the second number of repeated character strings. The ratio of the second number of repeated character strings to the number of strings in the historical job model file is used as the third ratio, and the ratio of the second number of repeated character strings to the number of strings in the queued job model file is used as the fourth ratio. Furthermore, for model files of different file types, a third ratio and a fourth ratio can be obtained, and the second similarity can be obtained by combining the two ratios. The product of the third ratio and the fourth ratio can be used as the third process value, the sum of the third ratio and the fourth ratio can be used as the fourth process value, and the quotient of the third process value and the fourth process value can be multiplied by 2 as the second similarity.

[0050] S120: In response to the total number of queued jobs being less than or equal to the total number of submittable jobs, each queued job is treated as a job to be assigned, and a node assignment job corresponding to each computing node is determined based on the total number of queued jobs, the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned.

[0051] The pending jobs are queued jobs that can be assigned to a compute node in the current cycle. The current estimated duration is the maximum difference between the estimated duration of each job corresponding to the compute node and the corresponding elapsed duration. This is the maximum remaining estimated duration required for the compute node to complete the currently assigned jobs. The node-assigned jobs are queued jobs assigned to each compute node.

[0052] Specifically, if the total number of queued jobs is less than or equal to the total number of submittable jobs, it indicates that all queued jobs can be assigned to a compute node within the current cycle. Therefore, each queued job can be considered a pending job. For each compute node, the difference between the estimated job duration and the elapsed runtime of each node-occupied job on that compute node is calculated. The maximum of these differences is then used as the current estimated job duration for that compute node. Assuming that after each compute node is assigned a queued job, a node assignment job is added to each compute node. The sum of the number of node-occupied jobs and the number of node-assigned jobs corresponding to each compute node is used as the number of node jobs for each compute node. The current estimated job duration can be updated based on the job estimated durations of each pending job and the current estimated job duration for each compute node. With the goal of minimizing the variance of both the number of node jobs corresponding to each compute node and the variance of the current estimated job duration for each compute node, the compute node to which each pending job should be assigned is determined. Based on this, the pending job corresponding to each compute node can be determined, and the job can be assigned to the corresponding compute node.

[0053] Based on the above example, the node allocation job corresponding to each compute node can be determined in the following way based on the total number of queued jobs, the number of remaining submittable jobs corresponding to each compute node, the current estimated duration corresponding to each compute node, and the estimated duration of each job to be assigned:

[0054] With the goal of minimizing the mean square error of the number of node jobs on each computing node, determine at least one first submission plan based on the total number of queued jobs, the number of remaining submittable jobs corresponding to each computing node, and each job to be assigned;

[0055] For each first submission plan, determine the mean square error of the node computation time corresponding to the first submission plan based on the first submission plan, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned;

[0056] The first target duration mean square deviation is determined according to the node calculation duration mean square deviation corresponding to each first submission scheme, and the node allocation job corresponding to each computing node is determined according to the first submission scheme corresponding to the first target duration mean square deviation.

[0057] The node job count is the total number of jobs in the compute node after all pending jobs have been assigned. The first submission scheme is the allocation scheme determined by minimizing the mean squared deviation of the node job counts for each compute node. The node computation duration mean squared deviation corresponding to the first submission scheme is the mean squared deviation of the current estimated durations of each compute node, updated after allocation according to the first submission scheme. The first target duration mean squared deviation is the minimum value among the node computation duration mean squared deviations corresponding to the first submission schemes.

[0058] Specifically, the number of node-occupied jobs corresponding to each computing node can be obtained, that is, the number of jobs being run. With the minimum mean square deviation of the number of node jobs of each computing node as the goal, and the number of remaining submittable jobs corresponding to each computing node as the limit, the number of assignable jobs corresponding to each computing node is determined, and the number of assignable jobs is less than or equal to the number of remaining submittable jobs. The jobs to be assigned are combined and assigned according to the number of assignable jobs corresponding to each computing node, and a variety of combination methods can be obtained, that is, a variety of first submission schemes. For each first submission scheme, according to the allocation method in the first submission scheme, combined with the current estimated duration corresponding to each computing node and the estimated duration of each job to be assigned, the job status of each computing node after allocation according to the first submission scheme can be determined, and the current estimated duration corresponding to each computing node can be updated. Then, the mean square deviation is calculated using each current estimated duration to obtain the mean square deviation of the node computing duration corresponding to the first submission scheme. The minimum value of the mean square error of the node calculation duration corresponding to each first submission plan is used as the first target mean square error, and the first submission plan corresponding to the first target mean square error is determined. According to the first submission plan corresponding to the first target mean square error, each to-be-assigned job is allocated, that is, the node allocation job corresponding to each computing node can be determined.

[0059] S130. In response to the total number of queued jobs being greater than the total number of submittable jobs, the jobs to be assigned are determined based on the remaining number of submittable jobs corresponding to each computing node, and the node assignment jobs corresponding to each computing node are determined based on the remaining number of submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned.

[0060] Specifically, if the total number of queued jobs is greater than the total number of submittable jobs, it means that all currently queued jobs cannot be assigned to a compute node within the current cycle. Only the total number of submittable jobs can be assigned. Therefore, the queued jobs can be sorted by upload time, and the jobs with the highest total number of submittable jobs can be selected as the jobs to be assigned. For each compute node, the difference between the estimated job duration and the elapsed runtime of each job occupied on that compute node is calculated. The maximum of these differences is then used as the current estimated job duration for that compute node. Assuming that each compute node is fully occupied after all queued jobs are assigned, the number of node jobs for each compute node is fixed. The current estimated job duration can be updated by combining the estimated job durations of each job to be assigned and the current estimated job duration for each compute node. With the goal of minimizing the variance of the current estimated job durations for each compute node, the compute node to which each job to be assigned is determined. Based on this, the jobs to be assigned to each compute node can be determined, and the jobs can be assigned to the corresponding compute nodes.

[0061] Based on the above example, the following method can be used to determine the jobs to be assigned based on the number of remaining submittable jobs corresponding to each computing node, and to determine the node assignment job corresponding to each computing node based on the number of remaining submittable jobs corresponding to each computing node, the current estimated duration of each computing node, and the estimated duration of each job to be assigned:

[0062] According to the queueing time of each queued job, determine the number of jobs to be assigned from the total number of jobs that can be submitted;

[0063] Determine at least one second submission plan based on the number of remaining submittable jobs corresponding to each computing node and each job to be assigned;

[0064] For each second submission plan, determine the mean square error of the node computation duration corresponding to the second submission plan based on the second submission plan, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned;

[0065] The second target mean square error of the duration is determined according to the node calculation duration corresponding to each second submission scheme, and the node allocation job corresponding to each computing node is determined according to the second submission scheme corresponding to the second target mean square error.

[0066] The second submission scheme is a combination of multiple allocation schemes determined based on the number of remaining submittable jobs on each computing node. The mean squared deviation of the node computation duration corresponding to the second submission scheme is the mean squared deviation of the current estimated durations of each computing node updated after allocation according to the second submission scheme. The second target duration mean squared deviation is the minimum value among the mean squared deviations of the node computation durations corresponding to each second submission scheme.

[0067] Specifically, since the total number of queued jobs is greater than the total number of submittable jobs, the computing nodes should all be fully loaded after allocation. Therefore, the remaining number of submittable jobs corresponding to each computing node can be directly used as the number of allocable jobs. By combining and allocating each to-be-allocated job according to the number of allocable jobs corresponding to each computing node, a variety of combinations can be obtained, i.e., a variety of second submission schemes. For each second submission scheme, according to the allocation method in the second submission scheme, combined with the current estimated duration corresponding to each computing node and the estimated duration of each to-be-allocated job, the job status of each computing node after allocation according to the second submission scheme can be determined, and the current estimated duration corresponding to each computing node can be updated. Then, the mean square error of each current estimated duration is calculated to obtain the mean square error of the node calculation duration corresponding to the second submission scheme. The minimum value of the mean square error of the node calculation duration corresponding to each second submission scheme is used as the second target duration mean square error, and the second submission scheme corresponding to the second target duration mean square error is determined. According to the second submission scheme corresponding to the second target duration mean square error, each to-be-allocated job is allocated, i.e., the node allocation job corresponding to each computing node can be determined.

[0068] S140 . For each computing node, submit the node allocation job corresponding to the computing node to the computing node.

[0069] Specifically, after determining the node allocation job corresponding to each computing node, each node allocation job is submitted to the corresponding computing node, so that each computing node can run the newly allocated node allocation job.

[0070] Based on the above example, you can submit the node allocation job corresponding to each compute node to the compute node in the following way:

[0071] For each computing node, determine the system type corresponding to the computing node;

[0072] In response to the system type being a Windows system, submitting a node allocation job corresponding to the computing node to the computing node through a restricted external interface provided by a service layer configured on the computing node;

[0073] In response to the system type being the Linux system, a node allocation job corresponding to the computing node is submitted to the computing node through an open external interface of the computing node.

[0074] System types include Windows and Linux. The service layer uses web interface encapsulation technology to implement Windows system calls, including reading job files and calling Windows scripts. It also provides external services. Restricted external interfaces are external interfaces provided by the service layer, limiting the sources from which they can be called. For example, in this example, they can only be called by systems that implement the sample method. Open external interfaces are commonly exposed externally, such as port 22.

[0075] Specifically, for each compute node, we first determine whether the corresponding system type is Windows or Linux. If the system type is Windows, we configure a service layer on the Windows system. This means that node-assignment jobs corresponding to the compute node can be submitted to the compute node through the restricted external interface provided by the service layer configured on the compute node to ensure system security. If the system type is Linux, due to its inherent security, there is no need to configure a service layer. Node-assignment jobs corresponding to the compute node can be submitted directly to the compute node through the open external interface of the compute node.

[0076] For example, for non-Windows systems, such as Linux, remote command execution via Python to query system load and other information can be performed directly through port 22 (an open interface) using the Paramiko library (a Python-based SSHv2 protocol library) and command-line access. For Windows systems, using Python to make direct system calls is more cumbersome and requires installing components such as OpenSSH, a free and open-source implementation of the SSH protocol (Secure Shell). Furthermore, exposing the commonly used port 22 (an open interface) and using sensitive parameters such as usernames and passwords to log in before executing command-line operations is relatively risky. Therefore, a service layer for Windows systems is developed, implemented using the Django framework (an advanced Python web application framework), exposing an HTTP interface (restricted external access), and publishing the interface through uncommon ports such as 8886. By filtering abnormal commands through the service layer corresponding to the Windows system, security can be improved to a certain extent. After the service layer is installed on each computing node of the Windows system, the system corresponding to this example method can directly call the HTTP interface provided by the service layer to operate the computing node, query the node load status, submit node allocation jobs, etc.

[0077] S150: Update the running nodes and job status corresponding to each node allocation job in the platform database.

[0078] Specifically, after allocating jobs to each node, the computing nodes corresponding to the jobs allocated to each node are updated as running nodes in the platform database, and the job status of the jobs allocated to these nodes is updated from queued to running.

[0079] It is understandable that after each job is completed, the field of the corresponding running node can be changed to the completion node, and the job status can be updated to processing success, processing failure, etc. according to the running result.

[0080] Based on the above example, you can also periodically view the usage of each node and the status of each submitted job on the target browsing interface. Specifically, you can:

[0081] Based on a preset period, determine each submitted job corresponding to each computing node from the platform database and obtain the job status of each submitted job;

[0082] The node address of each computing node, each submitted job corresponding to the computing node, and the job status of each submitted job are displayed on the target browsing interface.

[0083] The preset period may be a period at which the user wishes to view the status of jobs in the cluster, and may be set according to the user's needs. Submitted jobs are jobs running on the computing nodes.

[0084] Specifically, based on a preset period, the platform database can retrieve the submitted jobs corresponding to each compute node and determine the job status of each submitted job. Data in the platform database can be updated based on the job status of each compute node, either through active requests or passive monitoring. Updates can also be made using interfaces specific to different system types to retrieve relevant information. Furthermore, the node address of each compute node, the submitted jobs corresponding to the compute node, and the job status of each submitted job are displayed on the target browsing interface for the user to review the job status.

[0085] Based on the above example, you can also adjust the cluster size and node load on the target browsing interface. Specifically, you can:

[0086] In response to the user identity being the target identity, obtaining a management operation;

[0087] In response to the management operation being a cluster scale control operation, performing an operation of adding or reducing computing nodes according to the cluster scale control operation;

[0088] In response to the management operation being a node load control operation, a target node and a load quantity are determined according to the node load control operation, and the load quantity is set as a maximum number of jobs for the target node.

[0089] The user identity is the user's login identity after logging in with their account and password. User permissions are determined based on the user identity. The target identity is the administrator identity, which corresponds to a higher level of permissions. Management operations are operations performed by users on the cluster, including cluster scale control operations and node load control operations. The target node is the compute node whose load needs to be adjusted. The load number is the number of nodes configured for the load control operation. The maximum number of jobs is the number of jobs that the compute node can process in parallel.

[0090] Specifically, after the user logs in, the user identity can be obtained, and when the user identity is the target identity, the user's management operations can be received. If the management operation is a cluster scale control operation, the cluster scale control operation is obtained to determine whether it is an operation to increase or decrease nodes. When performing an operation to increase computing nodes, the corresponding IP address can be configured through the interface, and the node addition can be completed. Alternatively, when performing an operation to decrease computing nodes, the computing node that needs to be deleted is determined and the computing node is deleted from the cluster. If the management operation is a node load control operation, the node load control operation is parsed, and the computing node that needs to adjust the load can be determined as the target node, and the load quantity corresponding to the target node is obtained, and the load quantity is set as the maximum number of jobs for the target node.

[0091] Based on the above example, we can design various hierarchical structures based on the B / S architecture (Browser / Server Architecture) to connect the B / S user layer with the cluster's compute nodes. These hierarchical structures can include: platform layer, job status update layer, job scheduling layer, and compute node service layer.

[0092] The platform layer leverages Vue (a JavaScript framework for building user interfaces), Django, and MySQL (a relational database management system) technologies, using the REST Framework (a third-party extension library based on Django) interface to implement business logic such as user job submission, job list retrieval, job details, cluster scale control, and node load control. The job status update layer utilizes polling scripts, interface calls, and MySQL to call the Windows compute node service layer for timely job status updates. The job scheduling layer utilizes polling scripts, interface calls, web interface encapsulation, and Windows script calls, using the methods described in this example (S110-S150) to ensure consistent job counts and durations across all compute nodes, enabling automatic submission of queued jobs and load balancing across all compute nodes. The Windows compute node service layer (service layer) utilizes web interface encapsulation to implement Windows system calls, including job file reading and Windows script invocation. It also provides external services for the job status update layer and job scheduling layer.

[0093] The present invention has the following technical effects: by obtaining the total number of queued jobs in the platform database and the remaining number of submittable jobs corresponding to each computing node in the platform database, the total number of submittable jobs is determined according to the remaining number of submittable jobs corresponding to each computing node; if the total number of queued jobs is less than or equal to the total number of submittable jobs, each queued job is used as a job to be assigned; according to the total number of queued jobs, the remaining number of submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned, the node assignment job corresponding to each computing node is determined; otherwise, according to the remaining number of submittable jobs corresponding to each computing node, the job to be assigned is determined; according to the remaining number of submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned, the node assignment job corresponding to each computing node is determined, so as to provide an effective load balancing strategy and improve the effect of job assignment; for each computing node, the node assignment job corresponding to the computing node is submitted to the computing node; the running node and job status corresponding to each node assignment job in the platform database are updated, thereby achieving effective load balancing within the cluster.

[0094] Example 2

[0095] Figure 2 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 2 As shown, the electronic device 200 includes one or more processors 201 and a memory 202 .

[0096] The processor 201 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 200 to perform desired functions.

[0097] Memory 202 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 201 may execute the program instructions to implement the cluster job scheduling method of any embodiment of the present invention described above and / or other desired functions. Various contents such as initial external parameters and thresholds may also be stored in the computer-readable storage medium.

[0098] In one example, the electronic device 200 may further include an input device 203 and an output device 204, which are interconnected via a bus system and / or other connection mechanisms (not shown). The input device 203 may include, for example, a keyboard, a mouse, etc. The output device 204 may output various information to the outside, including warning information, braking force, etc. The output device 204 may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0099] Of course, to simplify, Figure 2 Only some of the components related to the present invention in the electronic device 200 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 200 may further include any other appropriate components according to specific application scenarios.

[0100] Example 3

[0101] In addition to the above methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the cluster job scheduling method provided by any embodiment of the present invention.

[0102] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the cluster job scheduling method provided by any embodiment of the present invention.

[0104] The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0105] It should be noted that the terms used in the present invention are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "an", "a kind of" and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method or device comprising a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method or device comprising the elements.

[0106] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A cluster job scheduling method, characterized in that: include: Obtaining the total number of queued jobs in the platform database and the number of remaining submittable jobs corresponding to each computing node in the platform database, and determining the total number of submittable jobs based on the number of remaining submittable jobs corresponding to each computing node; In response to the total number of queued jobs being less than or equal to the total number of submittable jobs, each queued job is treated as a job to be assigned, and a node assignment job corresponding to each computing node is determined based on the total number of queued jobs, the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned; In response to the total number of queued jobs being greater than the total number of submittable jobs, determining the jobs to be assigned based on the number of remaining submittable jobs corresponding to each computing node, and determining the node assignment jobs corresponding to each computing node based on the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned; For each computing node, submitting the node allocation job corresponding to the computing node to the computing node; Update the running nodes and job status corresponding to each node allocation job in the platform database.

2. The method according to claim 1, characterized in that The determining of the node allocation job corresponding to each computing node according to the total number of queued jobs, the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be allocated includes: With the goal of minimizing the mean square error of the number of node jobs of each computing node, determining at least one first submission scheme according to the total number of queued jobs, the number of remaining submittable jobs corresponding to each computing node, and each job to be assigned; For each first submission plan, determine the mean square error of the node computing duration corresponding to the first submission plan based on the first submission plan, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned; The first target mean square error of duration is determined according to the node calculation duration mean square error corresponding to each first submission scheme, and the node allocation job corresponding to each computing node is determined according to the first submission scheme corresponding to the first target mean square error.

3. The method according to claim 1, characterized in that The method of determining the jobs to be assigned based on the number of remaining submittable jobs corresponding to each computing node, and determining the node assignment jobs corresponding to each computing node based on the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned, includes: According to the queueing time of each queued job, determine the number of jobs to be assigned from the total number of jobs that can be submitted; Determine at least one second submission plan based on the number of remaining submittable jobs corresponding to each computing node and each job to be assigned; For each second submission plan, determine the mean square error of the node computation duration corresponding to the second submission plan based on the second submission plan, the current estimated duration corresponding to each computing node, and the estimated duration of each to-be-assigned job; The second target mean square error of the duration is determined according to the node calculation duration corresponding to each second submission scheme, and the node allocation job corresponding to each computing node is determined according to the second submission scheme corresponding to the second target mean square error.

4. The method according to claim 1, wherein Also includes: Submit a queued job through the target browsing interface, obtain identification information corresponding to the queued job, set the job status of the queued job to queued, and set the running node of the queued job to empty; Determining a target historical job corresponding to the queued job according to each software type and corresponding software version corresponding to the queued job; For each target historical job, determining the job similarity between the queued job and the target historical job according to the queued job and the target historical job; Determine an estimated duration of the queued job based on the historical calculation duration of each target historical job and the job similarity between the queued job and each target historical job; The platform database is updated according to the identification information, job status, running node, and estimated job duration corresponding to the queued job.

5. The method according to claim 4, characterized in that The determining, based on the queued job and the target historical job, the job similarity between the queued job and the target historical job includes: determining a first similarity based on the queued job script file of the queued job and the historical job script file of the target historical job; determining a second similarity based on the queued job model file of the queued job and the historical job model file of the target historical job; The job similarity between the queued job and the target historical job is determined according to the first similarity and the second similarity.

6. The method according to claim 5, characterized in that The determining of the first similarity based on the queued job script file of the queued job and the historical job script file of the target historical job includes: determining a first number of repeated character strings according to the queued job script file of the queued job and the historical job script file of the target historical job; using the ratio of the first number of repeated character strings to the number of character strings in the historical job script file as a first ratio, and using the ratio of the first number of repeated character strings to the number of character strings in the queued job script file as a second ratio; A first similarity is determined according to the first ratio and the second ratio.

7. The method according to claim 5, characterized in that The determining of the second similarity based on the queued job model file of the queued job and the historical job model file of the target historical job includes: In response to the queued job model file being a binary model file, the queued job model file of the queued job is divided into blocks according to a preset block size to obtain a plurality of first blocks, and the historical job model file of the target historical job is divided into blocks to obtain a plurality of second blocks; a number of repeated blocks is determined based on each of the first blocks and each of the second blocks; a ratio of the number of repeated blocks to the total number of blocks in the first blocks is used as a third ratio, and a ratio of the number of repeated blocks to the total number of blocks in the second blocks is used as a fourth ratio; In response to the queued job model file being a non-binary model file, determining a second number of repeated character strings based on the queued job model file of the queued job and the historical job model file of the target historical job; using a ratio of the second number of repeated character strings to the number of character strings in the historical job model file as a third ratio, and using a ratio of the second number of repeated character strings to the number of character strings in the queued job model file as a fourth ratio; A second similarity is determined according to the third ratio and the fourth ratio.

8. The method according to claim 1, characterized in that Submitting, for each computing node, a node allocation job corresponding to the computing node to the computing node, includes: For each computing node, determining a system type corresponding to the computing node; In response to the system type being a Windows system, submitting a node allocation job corresponding to the computing node to the computing node through a restricted external interface provided by a service layer configured on the computing node; In response to the system type being a Linux system, a node allocation job corresponding to the computing node is submitted to the computing node through an open external interface of the computing node.

9. The method according to claim 1, characterized in that Also includes: Based on a preset period, determining each submitted job corresponding to each computing node from the platform database, and obtaining the job status of each submitted job; The node address of each computing node, each submitted job corresponding to the computing node, and the job status of each submitted job are displayed on the target browsing interface.

10. The method according to claim 1, characterized in that Also includes: In response to the user identity being the target identity, obtaining a management operation; In response to the management operation being a cluster size control operation, performing an operation of adding or reducing computing nodes according to the cluster size control operation; In response to the management operation being a node load control operation, a target node and a load quantity are determined according to the node load control operation, and the load quantity is set as a maximum number of jobs for the target node.

Images