Cluster job scheduling method

Abstract

The invention discloses a cluster job scheduling method, which comprises the following steps of: obtaining the total queuing job number in a platform database and the residual submitted job number corresponding to each computing node, and determining the total submitted job number; if the total number of queuing jobs is smaller than or equal to the total number of submitted jobs, taking each queuing job as a job to be allocated, and determining a node allocation job corresponding to each computing node according to the total number of queuing jobs, each remaining number of submitted jobs, the current estimated duration corresponding to each computing node and the job estimated duration of each job to be allocated, otherwise, determining the node allocation job corresponding to each computing node. Determining jobs to be distributed according to the number of the remaining submitted jobs, and determining node distribution jobs corresponding to the computing nodes according to the number of the remaining submitted jobs, the current estimated durations and the estimated durations of the jobs; for each computing node, submitting the corresponding node allocation job to the computing node; and updating the running nodes and the job states corresponding to the node allocation jobs in the platform database, thereby realizing effective load balancing in the cluster.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a cluster job scheduling method. Background Art

[0002] Currently, job scheduling and resource management issues have become hot topics. Job scheduling allocates computing resources of cluster nodes to jobs. In this process, inappropriate job scheduling strategies and unreasonable cluster size will affect the load balancing status of the cluster.

[0003] However, the lack of a job scheduling strategy targeting load balancing will have multiple impacts on system performance, stability, and user experience.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a cluster job scheduling method, which realizes effective load balancing within the cluster.

[0006] An embodiment of the present invention provides a cluster job scheduling method, the method comprising:

[0007] Obtain the total number of queued jobs in the platform database and the number of remaining submittable jobs corresponding to each computing node in the platform database, and determine the total number of submittable jobs based on the number of remaining submittable jobs corresponding to each computing node;

[0008] In response to the total number of queued jobs being less than or equal to the total number of submittable jobs, each queued job is treated as a job to be assigned, and a node assignment job corresponding to each computing node is determined based on the total number of queued jobs, the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned;

[0009] In response to the total number of queued jobs being greater than the total number of submittable jobs, determining the jobs to be assigned based on the number of remaining submittable jobs corresponding to each computing node, and determining the node assignment jobs corresponding to each computing node based on the number of remaining submittable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned;

[0010] For each computing node, submit the node allocation job corresponding to the computing node to the computing node;

[0011] Update the running nodes and job status corresponding to each node allocation job in the platform database.

[0012] The embodiments of the present invention have the following technical effects:

[0013] By obtaining the total number of queued jobs in the platform database and the remaining number of jobs that can be submitted corresponding to each computing node in the platform database, determining the total number of jobs that can be submitted according to the remaining number of jobs that can be submitted corresponding to each computing node, if the total number of queued jobs is less than or equal to the total number of jobs that can be submitted, then taking each queued job as a job to be assigned, and determining the node-assigned jobs corresponding to each computing node according to the total number of queued jobs, the remaining number of jobs that can be submitted corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be assigned; otherwise, determining the jobs to be assigned according to the remaining number of jobs that can be submitted corresponding to each computing node, and determining the node-assigned jobs corresponding to each computing node according to the remaining number of jobs that can be submitted corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be assigned, so as to provide an effective load balancing strategy, improve the effect of job assignment, and for each computing node, submitting the node-assigned jobs corresponding to the computing node to the computing node, and updating the running nodes and job statuses corresponding to each node-assigned job in the platform database, thus achieving effective load balancing within the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0015] Figure 1 is a flowchart of a cluster job scheduling method provided by an embodiment of the present invention;

[0016] Figure 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0018] The cluster job scheduling method provided by the embodiment of the present invention is mainly applicable to the situation of load balancing of computing nodes in combination with the characteristics of various systems. The cluster job scheduling method provided by the embodiment of the present invention can be executed by an electronic device.

[0019] Embodiment 1

[0020] Figure 1 It is a flowchart of a cluster job scheduling method provided by an embodiment of the present invention. Refer to Figure 1 , the cluster job scheduling method specifically includes:

[0021] S110. Obtain the total number of queued jobs in the platform database and the remaining number of jobs that can be submitted corresponding to each computing node in the platform database, and determine the total number of jobs that can be submitted according to the remaining number of jobs that can be submitted corresponding to each computing node.

[0022] Among them, the platform database is a database for recording relevant information of each submitted job, such as user identification, job identification, job name, script path, job status, running node, and estimated job duration, etc., and is also used to record the jobs being executed in each computing node and the corresponding current estimated duration, etc. A computing node is a node in the cluster used to execute job calculation and processing tasks. The remaining number of jobs that can be submitted is the number of jobs that a computing node can currently receive and execute. The total number of jobs that can be submitted is the number of jobs that can still be submitted and run in the cluster, which is the sum of the remaining number of jobs that can be submitted corresponding to each computing node.

[0023] Specifically, obtaining the number of jobs with the queued status in the platform database as the total number of queued jobs, and obtaining the remaining number of jobs that can be submitted corresponding to each computing node in the platform database. It can be that, for each computing node, the difference between the number of effective computing cores corresponding to the computing node and the number of jobs occupied by the node is used as the remaining number of jobs that can be submitted for the computing node. The sum of the remaining number of jobs that can be submitted corresponding to each computing node is determined as the total number of jobs that can be submitted.

[0024] Based on the above example, when a user submits a job to the cluster, it is necessary to extract and preliminarily process the information of the submitted job and record it in the platform database. Specifically, it can be:

[0025] Submit a queued job through the target browsing interface, obtain each identification information corresponding to the queued job, set the job status of the queued job to queued, and set the running node of the queued job to empty;

[0026] Determine the target historical job corresponding to the queued job according to each software type and corresponding software version of the queued job;

[0027] For each target historical job, determine the job similarity between the queued job and the target historical job according to the queued job and the target historical job;

[0028] Determine the estimated job duration of the queued job according to the historical calculation duration of each target historical job and the job similarity between the queued job and each target historical job;

[0029] Update the platform database according to the respective identification information, job status, running node, and estimated job duration corresponding to the queued job.

[0030] Among them, the target browsing interface is a browser interface that users can view and operate, and can be used to control the cluster, submit jobs, view job status, etc. A queued job is a job submitted by a user, which is in a queued state when submitted, and will become a node-assigned job after being assigned corresponding computing nodes for processing. The identification information includes user identification, job identification, job name, etc. The job status is the current status of the job, which can be queued, running, processed successfully, processed failed, etc. The running node is the computing node assigned for job processing, which can be the node identification of the computing node, and can also include the IP (Internet Protocol) address of the computing node, etc. The software type is the type of software required for the queued job, and the software version is the version number of the software type corresponding to the queued job. The target historical job is a historical job that uses the same software type and the same software version as the queued job. The job similarity is a comprehensive evaluation value of the similarity of various files contained in the target historical job and the queued job. The historical computing duration is the duration used when the historical job is completed and processed successfully.

[0031] Specifically, users can submit queued jobs through the target browsing interface. When receiving a queued job, the user identification of the submitted queued job, the job identification, and the job name of this queued job can be obtained, and these information are used as the respective identification information corresponding to this queued job. And, set the job status of the queued job to queued, and set the running node of the queued job to empty. Analyzing the queued job can determine its corresponding software types and corresponding software versions, and take the part of the historical jobs processed by the cluster that have the same software type and corresponding software version as the target historical job corresponding to the queued job. For each target historical job, determine the job similarity between the queued job and the target historical job according to the queued job and the target historical job. Analyze the various files of the queued job and the various files of each target historical job, and calculate the job similarity between the queued job and each target historical job. According to the proportion of each job similarity in the total job similarity, the weight corresponding to each target historical job can be determined. According to the weight corresponding to each target historical job, perform a weighted sum of the historical computing durations of each target historical job to obtain the estimated job duration of the queued job. Update the respective identification information, job status, running node, and estimated job duration corresponding to the queued job submitted by the user to the platform database for subsequent assignment of computing nodes.

[0032] Based on the above example, the job similarity between the queued job and the target historical job can be determined in the following way according to the queued job and the target historical job:

[0033] Determine a first similarity based on the queuing job script file of the queuing job and the historical job script file of the target historical job;

[0034] Determine a second similarity based on the queuing job model file of the queuing job and the historical job model file of the target historical job;

[0035] Determine the job similarity between the queuing job and the target historical job based on the first similarity and the second similarity.

[0036] Among them, the queuing job includes a queuing job script file and a queuing job model file, and the target historical job includes a historical job script file and a historical job model file. The script file is a text file of executable code, which is used to define the operation logic or task flow. The model file is a file that stores static data structures, describes object attributes (such as geometric shape, material, hierarchical relationship), and does not contain executable logic. The first similarity is the similarity between the queuing job script file and the historical job script file. The second similarity is the similarity between the queuing job model file and the historical job model file.

[0037] Specifically, calculate the similarity between the queuing job script file of the queuing job and the historical job script file of the target historical job. It can be any existing similarity calculation method to determine the first similarity on the script file. Calculate the similarity between the queuing job model file of the queuing job and the historical job model file of the target historical job. It can be any existing similarity calculation method to determine the second similarity on the model file. According to the preset weights of the script file and the model file, perform a weighted sum on the first similarity and the second similarity to determine the job similarity between the queuing job and the target historical job.

[0038] Based on the above example, the following method can be used to determine the first similarity according to the queuing job script file of the queuing job and the historical job script file of the target historical job:

[0039] Determine the number of first repeated strings according to the queuing job script file of the queuing job and the historical job script file of the target historical job;

[0040] Take the ratio of the number of first repeated strings to the number of strings in the historical job script file as the first ratio, and take the ratio of the number of first repeated strings to the number of strings in the queuing job script file as the second ratio;

[0041] Determine the first similarity according to the first ratio and the second ratio.

[0042] Among them, the first number of repeated strings is the number of identical strings between the queued job script file and the target historical job and the historical job script file. The first ratio is the ratio of the first number of repeated strings to the number of strings in the historical job script file. The second ratio is the ratio of the first number of repeated strings to the number of strings in the queued job script file.

[0043] Specifically, by comparing the queued job script file of the queued job and the historical job script file of the target historical job, the number of repeated strings is taken as the first number of repeated strings. The ratio of the first number of repeated strings to the number of strings in the historical job script file is taken as the first ratio, and the ratio of the first number of repeated strings to the number of strings in the queued job script file is taken as the second ratio. Furthermore, the first similarity can be obtained by integrating the first ratio and the second ratio. It can be that the product of the first ratio and the second ratio is taken as the first process value, the sum value of the first ratio and the second ratio is taken as the second process value, and the quotient of the first process value and the second process value multiplied by 2 is taken as the first similarity.

[0044] Based on the above example, the second similarity can be determined according to the queued job model file of the queued job and the historical job model file of the target historical job in the following way:

[0045] In response to the queued job model file being a binary model file, the queued job model file of the queued job is block-processed according to a preset block size to obtain a plurality of first blocks, and the historical job model file of the target historical job is block-processed to obtain a plurality of second blocks; the number of repeated blocks is determined according to each first block and each second block; the ratio of the number of repeated blocks to the total number of blocks of the first blocks is taken as the third ratio, and the ratio of the number of repeated blocks to the total number of blocks of the second blocks is taken as the fourth ratio;

[0046] In response to the queued job model file not being a binary model file, the second number of repeated strings is determined according to the queued job model file of the queued job and the historical job model file of the target historical job; the ratio of the second number of repeated strings to the number of strings in the historical job model file is taken as the third ratio, and the ratio of the second number of repeated strings to the number of strings in the queued job model file is taken as the fourth ratio;

[0047] The second similarity is determined according to the third ratio and the fourth ratio.

[0048] Among them, the binary model file is a model file encoded with binary character encoding, and the non-binary model file is a model file encoded with characters other than binary characters. The preset block size is a pre-set block size for block processing, such as 1k. The first block is each block obtained by block-processing the queued job model file according to the preset block size. The second block is each block obtained by block-processing the historical job model file according to the preset block size. The repeated block number is the number of identical blocks between the set of the first blocks and the set of the second blocks. The second repeated string number is the number of identical strings between the queued job model file and the target historical job and the historical job model file. The third ratio is the ratio of the repeated block number to the total number of blocks of the first blocks, or the ratio of the second repeated string number to the number of strings of the historical job model file. The fourth ratio is the ratio of the repeated block number to the total number of blocks of the second blocks, or the ratio of the second repeated string number to the number of strings of the queued job model file.

[0049] Specifically, determine the file type of the queued job model file. If the queued job model file is a binary model file, it cannot be calculated in the way of repeated strings. It is necessary to perform block processing on the queued job model file of the queued job according to the preset block size to obtain multiple first blocks. Moreover, perform block processing on the historical job model file of the target historical job to obtain multiple second blocks. Then, analyze the repeated blocks between each first block and each second block to obtain the repeated block number. Take the ratio of the repeated block number to the total number of blocks of the first blocks as the third ratio, and take the ratio of the repeated block number to the total number of blocks of the second blocks as the fourth ratio. If the queued job model file is a non-binary model file, it is similar to the way of processing the job script file. Compare the queued job model file of the queued job and the historical job model file of the target historical job, and take the number of repeated strings as the second repeated string number. Take the ratio of the second repeated string number to the number of strings of the historical job model file as the third ratio, and take the ratio of the second repeated string number to the number of strings of the queued job model file as the fourth ratio. Then, for model files of different file types, the third ratio and the fourth ratio can be obtained. By synthesizing the two ratios, the second similarity can be obtained. It can be to take the product of the third ratio and the fourth ratio as the third process value, take the sum value of the third ratio and the fourth ratio as the fourth process value, and take the quotient of the third process value and the fourth process value multiplied by 2 as the second similarity.

[0050] S120. In response to the total number of queued jobs being less than or equal to the total number of jobs that can be submitted, each queued job is used as a job to be assigned. According to the total number of queued jobs, the remaining number of jobs that can be submitted corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be assigned, determine the node assignment jobs corresponding to each computing node.

[0051] Among them, the jobs to be assigned are the queued jobs for which the computing nodes can be assigned in the current cycle. The current estimated duration is the maximum value of the differences between the estimated durations of the jobs corresponding to each computing node and the corresponding running durations, that is, the longest remaining estimated duration required for the computing node to complete the currently assigned jobs. The node-assigned jobs are the queued jobs assigned to each computing node.

[0052] Specifically, if the total number of queued jobs is less than or equal to the total number of jobs that can be submitted, it means that each queued job in the current queuing state can be assigned a computing node in the current cycle. Therefore, each queued job can be used as a job to be assigned. For each computing node, calculate the difference between the estimated duration of the job corresponding to each node-occupied job on the computing node and the running duration, and then take the maximum value of these differences as the current estimated duration corresponding to the computing node. Assume that after the queued jobs are assigned to each computing node, node-assigned jobs are added to each computing node, and the sum of the number of node-occupied jobs and the number of node-assigned jobs corresponding to each computing node is used as the number of node jobs of each computing node. Moreover, by combining the estimated durations of the jobs to be assigned and the current estimated durations corresponding to each computing node, the current estimated duration can be updated. With the goal of minimizing the variance of the number of node jobs corresponding to each computing node and the variance of the current estimated duration corresponding to each computing node, determine which computing node each job to be assigned should be assigned to. Accordingly, the jobs to be assigned corresponding to each computing node can be determined, which are the node-assigned jobs for the corresponding computing node.

[0053] Based on the above example, the node-assigned jobs corresponding to each computing node can be determined in the following manner according to the total number of queued jobs, the remaining number of jobs that can be submitted corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned:

[0054] With the goal of minimizing the mean square deviation of the number of node jobs of each computing node, determine at least one first submission plan according to the total number of queued jobs, the remaining number of jobs that can be submitted corresponding to each computing node, and each job to be assigned;

[0055] For each first submission plan, determine the mean square deviation of the node computing duration corresponding to the first submission plan according to the first submission plan, the current estimated duration corresponding to each computing node, and the estimated duration of each job to be assigned;

[0056] According to the mean square deviation of the node computing duration corresponding to each first submission plan, determine the first target mean square deviation, and according to the first submission plan corresponding to the first target mean square deviation, determine the node-assigned jobs corresponding to each computing node.

[0057] Among them, the number of node jobs is the total number of jobs in the computing nodes after the jobs to be allocated are allocated. The first submission plan is a variety of allocation plans determined according to the minimum mean square deviation of the number of node jobs of each computing node. The mean square deviation of the node computing duration corresponding to the first submission plan is the mean square deviation of the current estimated duration of each computing node updated after allocation according to the first submission plan. The first target duration mean square deviation is the minimum value among the mean square deviations of the node computing durations corresponding to each first submission plan.

[0058] Specifically, the number of jobs occupying nodes corresponding to each computing node can be obtained, that is, the number of jobs that are running. Taking the minimum mean square deviation of the number of node jobs of each computing node as the goal and the remaining submitable jobs corresponding to each computing node as the limit, the allocable number of jobs corresponding to each computing node is determined, and the allocable number of jobs is less than or equal to the remaining submitable jobs. The jobs to be allocated are combined and allocated according to the allocable number of jobs corresponding to each computing node, and multiple combination methods can be obtained, that is, multiple first submission plans. For each first submission plan, according to the allocation method in the first submission plan, combined with the current estimated duration of each computing node and the job estimated duration of each job to be allocated, the job situation of each computing node after allocation according to the first submission plan can be determined, that is, the current estimated duration corresponding to each computing node can be updated. Furthermore, the mean square deviation is calculated using each current estimated duration to obtain the mean square deviation of the node computing duration corresponding to this first submission plan. The minimum value among the mean square deviations of the node computing durations corresponding to each first submission plan is used as the first target duration mean square deviation, and the first submission plan corresponding to the first target duration mean square deviation is determined. According to the first submission plan corresponding to the first target duration mean square deviation, the jobs to be allocated are allocated, that is, the node allocation jobs corresponding to each computing node can be determined.

[0059] S130. In response to the total number of queued jobs being greater than the total number of submitable jobs, the jobs to be allocated are determined according to the remaining submitable jobs corresponding to each computing node, and the node allocation jobs corresponding to each computing node are determined according to the remaining submitable jobs corresponding to each computing node, the current estimated duration of each computing node, and the job estimated duration of each job to be allocated.

[0060] Specifically, if the total number of queued jobs is greater than the total number of jobs that can be submitted, it means that not all queued jobs in the current queuing state can be assigned computing nodes within the current cycle. Only the queued jobs with the total number of jobs that can be submitted can be assigned. Therefore, the queued jobs can be sorted according to the upload time, and the queued jobs with the total number of jobs that can be submitted can be selected as the jobs to be assigned. For each computing node, calculate the difference between the estimated job duration and the running duration corresponding to each node-occupied job on the computing node. Furthermore, take the maximum value among these differences as the current estimated duration corresponding to the computing node. Assuming that after each computing node is assigned queued jobs, each computing node should be in a fully occupied state. Therefore, the number of node jobs for each computing node is determined. Combining the estimated job duration of each job to be assigned and the current estimated duration corresponding to each computing node, the current estimated duration can be updated. With the goal of minimizing the variance of the current estimated duration corresponding to each computing node, determine which computing node each job to be assigned should be assigned to. Accordingly, the jobs to be assigned corresponding to each computing node can be determined, and node assignment jobs for the corresponding computing nodes can be obtained.

[0061] Based on the above example, the jobs to be assigned can be determined according to the remaining number of jobs that can be submitted corresponding to each computing node in the following way. According to the remaining number of jobs that can be submitted corresponding to each computing node, the current estimated duration corresponding to each computing node, and the estimated job duration of each job to be assigned, determine the node assignment jobs corresponding to each computing node:

[0062] Determine the jobs to be assigned with the total number of jobs that can be submitted from each queued job according to the queuing time of each queued job;

[0063] According to the remaining number of jobs that can be submitted corresponding to each computing node and each job to be assigned, determine at least one second submission plan;

[0064] For each second submission plan, according to the second submission plan, the current estimated duration corresponding to each computing node, and the estimated job duration of each job to be assigned, determine the mean square deviation of the node computing durations corresponding to the second submission plan;

[0065] According to the mean square deviation of the node computing durations corresponding to each second submission plan, determine the second target mean square deviation of the duration, and according to the second submission plan corresponding to the second target mean square deviation of the duration, determine the node assignment jobs corresponding to each computing node.

[0066] Among them, the second submission plan is a variety of assignment plans determined according to the remaining number of jobs that can be submitted for each computing node. The mean square deviation of the node computing durations corresponding to the second submission plan is the mean square deviation of the current estimated durations of each computing node updated according to the second submission plan. The second target mean square deviation of the duration is the minimum value among the mean square deviations of the node computing durations corresponding to each second submission plan.

[0067] Specifically, since the total number of queued jobs is greater than the total number of jobs that can be submitted, after allocation, each computing node should be in a full-load state. Therefore, the remaining number of jobs that can be submitted corresponding to each computing node can be directly used as the number of jobs to be allocated. Combining each job to be allocated according to the number of jobs to be allocated corresponding to each computing node, multiple combination methods can be obtained, that is, multiple second submission plans. For each second submission plan, according to the allocation method in the second submission plan, combined with the current estimated duration corresponding to each computing node and the job estimated duration of each job to be allocated, the job situation of each computing node after allocation according to the second submission plan can be determined, and then the current estimated duration corresponding to each computing node can be updated. Furthermore, the mean square error is calculated using each current estimated duration to obtain the mean square error of the node computing duration corresponding to this second submission plan. Taking the minimum value among the mean square errors of the node computing durations corresponding to each second submission plan as the second target mean square error, and determining the second submission plan corresponding to the second target mean square error, and allocating each job to be allocated according to the second submission plan corresponding to the second target mean square error, that is, the node allocation jobs corresponding to each computing node can be determined.

[0068] S140. For each computing node, submit the node allocation job corresponding to the computing node to the computing node.

[0069] Specifically, after determining the node allocation jobs corresponding to each computing node, submit each node allocation job to the corresponding computing node so that each computing node can run each newly allocated node allocation job.

[0070] Based on the above example, the following method can be used to submit the node allocation job corresponding to each computing node to the computing node for each computing node:

[0071] For each computing node, determine the system type corresponding to the computing node;

[0072] In response to the system type being the windows system, submit the node allocation job corresponding to the computing node to the computing node through the restricted external interface provided by the service layer configured on the computing node;

[0073] In response to the system type being the linux system, submit the node allocation job corresponding to the computing node to the computing node through the open external interface of the computing node.

[0074] Among them, the system types include the Windows system and the Linux system. The service layer uses web interface encapsulation technology to implement Windows system calls, including job file reading, Windows script calls, etc., and can also implement the layer structure for external services. The restricted external interface is the external interface provided by the service layer, and the call source of this interface can be restricted. For example, in this example, it is only called by the system that implements the method of this example. The open external interface is a commonly used external interface, such as port 22, etc.

[0075] Specifically, for each computing node, first determine whether the system type corresponding to the computing node is the Windows system or the Linux system. If the system type is the Windows system, configure the service layer on the Windows system. That is, the node allocation job corresponding to the computing node can be submitted to the computing node through the restricted external interface provided by the service layer configured on this computing node to ensure system security. If the system type is the Linux system, due to its own security, there is no need to configure the service layer, and the node allocation job corresponding to the computing node can be directly submitted to the computing node through the open external interface of this computing node.

[0076] Exemplarily, for non-Windows systems, such as the Linux system, to query the system load situation, etc. by remotely executing commands through Python, it can be directly extracted through port 22 (open external interface) using the paramiko library (a library implementing the SSHv2 protocol based on Python) through the command line method. For the Windows system, directly making system calls using Python is relatively cumbersome and requires installing components such as OpenSSH for the Windows system to achieve. OpenSSH is a free and open-source implementation of the SSH protocol (Secure Shell). Moreover, if using the commonly exposed port 22 (open external interface) and logging in with sensitive parameters such as account passwords and then performing command line operations is relatively dangerous. Therefore, develop the service layer corresponding to the Windows system. The service layer corresponding to the Windows system is implemented through the django framework (an advanced Python web application framework), exposes an http interface (restricted external interface) externally, and publishes the interface through ports such as 8886 that are not commonly used. By filtering and controlling abnormal commands through the service layer corresponding to the Windows system, the security can be improved to a certain extent. After installing the service layer on each computing node of the Windows system, the system corresponding to the method of this example can directly call the http interface provided by the service layer to operate on the computing node, query the node load status, submit node allocation jobs, etc.

[0077] S150. Update the running nodes and job status corresponding to the jobs assigned to each node in the platform database.

[0078] Specifically, after the jobs assigned to each node are allocated, use the computing nodes corresponding to the jobs assigned to each node as the running nodes and update them to the platform database. Moreover, update the job status of these jobs assigned to nodes from the queuing status to in progress.

[0079] It can be understood that after each job is completed, the field of the corresponding running node can be changed to a completed node, and moreover, update the job status to successfully processed, failed to process, etc. according to the running result.

[0080] Based on the above example, it is also possible to periodically view the usage of each node and the status of each submitted job on the target browsing interface. Specifically, it can be:

[0081] Based on a preset period, determine each submitted job corresponding to each computing node from the platform database and obtain the job status of each submitted job.

[0082] Display the node address of each computing node, each submitted job corresponding to the computing node, and the job status of each submitted job on the target browsing interface.

[0083] Among them, the preset period can be the period when the user hopes to view the job situation in the cluster and can be set according to the user's needs. A submitted job is each job running on a computing node.

[0084] Specifically, based on the preset period, it is possible to obtain each submitted job corresponding to each computing node from the platform database and determine the job status of each submitted job. The data in the platform database can be updated according to the job situation in each computing node, which can be an active request for update or a passive monitoring update. When updating, corresponding interfaces can also be used according to different system types to obtain relevant information. Furthermore, display the node address of each computing node, each submitted job corresponding to the computing node, and the job status of each submitted job on the target browsing interface for the user to view the job situation.

[0085] Based on the above example, it is also possible to adjust the cluster scale and node load on the target browsing interface. Specifically, it can be:

[0086] In response to the user identity being the target identity, obtain the management operation.

[0087] In response to the management operation being a cluster scale control operation, perform an operation to increase or decrease the computing nodes according to the cluster scale control operation.

[0088] In response to a management operation being a node load control operation, according to the node load control operation, determine the target node and the load quantity, and set the load quantity as the maximum number of jobs of the target node.

[0089] Among them, the user identity is the identity after the user logs in using the account password, and the user permissions can be determined according to the user identity. The target identity is the administrator identity, that is, the corresponding high-level permissions. The management operation is the operation for the user to manage the cluster, which can include the cluster scale control operation and the node load control operation. The target node is the computing node that needs to adjust the load. The load quantity is the quantity configured by the node load control operation. The maximum number of jobs is the amount of jobs that the computing node can process in parallel.

[0090] Specifically, after the user logs in, the user identity can be obtained. When the user identity is the target identity, the user's management operation can be received. If the management operation is the cluster scale control operation, obtain the cluster scale control operation, determine whether it is an operation to increase or decrease nodes. When performing the operation to increase computing nodes, the corresponding IP address, etc. can be configured in the form of an interface, wait for the node addition to complete, or when performing the operation to decrease computing nodes, determine the computing nodes to be deleted and delete the computing nodes from the cluster. If the management operation is the node load control operation, parse the node load control operation, determine the computing node that needs to adjust the load as the target node, and obtain the load quantity corresponding to the target node, and set the load quantity as the maximum number of jobs of the target node.

[0091] Based on the above example, multiple hierarchical structures can be designed based on the B / S architecture (Browser / Server Architecture), to connect the user layer of the B / S architecture with the computing nodes of the cluster. The multiple hierarchical structures can include: the platform layer, the job status update layer, the job scheduling layer, and the computing node service layer.

[0092] Among them, the platform layer can use Vue (a JavaScript framework for building user interfaces), Django technology, MySQL (a relational database management system) technology, and use the REST Framework (a third-party extension library based on Django) interface to implement business logics such as user job submission, job list acquisition, job detail acquisition, cluster scale control, and node load control. The job status update layer can use polling script technology, interface call technology, and MySQL technology to achieve timely update of job status by calling the Windows computing node service layer. The job scheduling layer can use polling script technology, interface call technology, web interface encapsulation technology, and Windows script call technology, and use the method of S110 - S150 in this example to make the number of jobs on each computing node basically the same and the duration basically the same, and achieve automatic submission of queued jobs and load balancing of each computing node. The Windows computing node service layer (service layer) uses web interface encapsulation technology to implement Windows system calls, including job file reading, Windows script call, etc., and at the same time implements external services for use by the job status update layer and the job scheduling layer.

[0093] The present invention has the following technical effects: By obtaining the total number of queued jobs in the platform database and the remaining submitable jobs corresponding to each computing node in the platform database, determining the total submitable jobs according to the remaining submitable jobs corresponding to each computing node. If the total number of queued jobs is less than or equal to the total submitable jobs, then take each queued job as a job to be allocated, and determine the node-allocated jobs corresponding to each computing node according to the total number of queued jobs, the remaining submitable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated. Otherwise, determine the jobs to be allocated according to the remaining submitable jobs corresponding to each computing node, and determine the node-allocated jobs corresponding to each computing node according to the remaining submitable jobs corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated, so as to provide an effective load balancing strategy and improve the effect of job allocation. For each computing node, submit the node-allocated jobs corresponding to the computing node to the computing node, and update the running node and job status corresponding to each node-allocated job in the platform database, thus achieving effective load balancing within the cluster.

[0094] Embodiment 2

[0095] Figure 2 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As Figure 2 shown, the electronic device 200 includes one or more processors 201 and a memory 202.

[0096] The processor 201 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 200 to perform desired functions.

[0097] The memory 202 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 201 may run the program instructions to implement the cluster job scheduling method of any embodiment of the present invention described above and / or other desired functions. Various contents such as initial extrinsic parameters, thresholds, etc. may also be stored in the computer-readable storage media.

[0098] In one example, the electronic device 200 may further include: an input device 203 and an output device 204, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). The input device 203 may include, for example, a keyboard, a mouse, etc. The output device 204 may output various information to the outside, including warning prompt information, braking force, etc. The output device 204 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0099] Of course, for simplicity, Figure 2 only some of the components related to the present invention in the electronic device 200 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 200 may further include any other appropriate components.

[0100] Embodiment III

[0101] In addition to the above methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to execute the steps of the cluster job scheduling method provided by any embodiment of the present invention.

[0102] The computer program product can be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present invention. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The programming code can be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] In addition, an embodiment of the present invention can also be a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are run by a processor, the processor is caused to execute the steps of the cluster job scheduling method provided by any embodiment of the present invention.

[0104] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0105] It should be noted that the terms used in the present invention are only for describing specific embodiments and do not limit the scope of the present application. As shown in the specification of the present invention, unless the context clearly indicates an exception, words such as "a", "an", "one", and / or "the" are not specifically singular and may also include the plural. The term "comprising", "including", or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, or device including the element.

[0106] It should also be noted that the orientation or positional relationship indicated by terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention. Unless otherwise clearly specified and defined, terms such as "installed", "connected", "coupled" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A cluster job scheduling method, characterized in that, Including: Obtain the total number of queued jobs in the platform database and the remaining submitable job numbers corresponding to each computing node in the platform database, and determine the total submitable job number according to the remaining submitable job numbers corresponding to each computing node; In response to the total number of queued jobs being less than or equal to the total submitable job number, regard each queued job as a job to be allocated, and determine the node-allocated jobs corresponding to each computing node according to the total number of queued jobs, the remaining submitable job numbers corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated; In response to the total number of queued jobs being greater than the total submitable job number, determine the jobs to be allocated according to the remaining submitable job numbers corresponding to each computing node, and determine the node-allocated jobs corresponding to each computing node according to the remaining submitable job numbers corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated; For each computing node, submit the node-allocated jobs corresponding to the computing node to the computing node; Update the running nodes and job statuses corresponding to the node-allocated jobs in the platform database.

2. The method according to claim 1, characterized in that The determining the node-allocated jobs corresponding to each computing node according to the total number of queued jobs, the remaining submitable job numbers corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated includes: Aiming at minimizing the mean square deviation of the number of node jobs of each computing node, determine at least one first submission plan according to the total number of queued jobs, the remaining submitable job numbers corresponding to each computing node, and each job to be allocated; For each first submission plan, determine the mean square deviation of the node computing duration corresponding to the first submission plan according to the first submission plan, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated; Determine the first target mean square deviation of the duration according to the mean square deviation of the node computing duration corresponding to each first submission plan, and determine the node-allocated jobs corresponding to each computing node according to the first submission plan corresponding to the first target mean square deviation of the duration.

3. The method according to claim 1, wherein The determining the jobs to be allocated according to the remaining submitable job numbers corresponding to each computing node, and determining the node-allocated jobs corresponding to each computing node according to the remaining submitable job numbers corresponding to each computing node, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated includes: Determine the jobs to be allocated with the total submitable job number from each queued job according to the queuing time of each queued job; Determine at least one second submission plan according to the remaining submitable job numbers corresponding to each computing node and each job to be allocated; For each second submission plan, determine the mean square deviation of the node computing duration corresponding to the second submission plan according to the second submission plan, the current estimated duration corresponding to each computing node, and the job estimated duration of each job to be allocated; Determine the second target mean square deviation of the duration according to the mean square deviation of the node computing duration corresponding to each second submission plan, and determine the node-allocated jobs corresponding to each computing node according to the second submission plan corresponding to the second target mean square deviation of the duration.

4. The method according to claim 1, wherein It further includes: Submit a queued job through the target browsing interface, obtain each identification information corresponding to the queued job, set the job status of the queued job to queued, and set the running node of the queued job to empty; Determine the target historical job corresponding to the queued job according to each software type and corresponding software version of the queued job; For each target historical job, determine the job similarity between the queued job and the target historical job according to the queued job and the target historical job; Determine the estimated job duration of the queued job according to the historical calculation duration of each target historical job and the job similarity between the queued job and each target historical job; Update the platform database according to each identification information, job status, running node, and estimated job duration corresponding to the queued job.

5. The method according to claim 4, characterized in that The determining the job similarity between the queued job and the target historical job according to the queued job and the target historical job includes: Determine the first similarity according to the queued job script file of the queued job and the historical job script file of the target historical job; Determine the second similarity according to the queued job model file of the queued job and the historical job model file of the target historical job; Determine the job similarity between the queued job and the target historical job according to the first similarity and the second similarity.

6. The method according to claim 5, characterized in that, The determining the first similarity according to the queued job script file of the queued job and the historical job script file of the target historical job includes: Determine the number of first repeated strings according to the queued job script file of the queued job and the historical job script file of the target historical job; Take the ratio of the number of first repeated strings to the number of strings in the historical job script file as the first ratio, and take the ratio of the number of first repeated strings to the number of strings in the queued job script file as the second ratio; Determine the first similarity according to the first ratio and the second ratio.

7. The method according to claim 5, characterized in that, The determining the second similarity according to the queued job model file of the queued job and the historical job model file of the target historical job includes: In response to the queued job model file being a binary model file, perform block processing on the queued job model file of the queued job according to a preset block size to obtain a plurality of first blocks, perform block processing on the historical job model file of the target historical job to obtain a plurality of second blocks; determine the number of repeated blocks according to each first block and each second block; take the ratio of the number of repeated blocks to the total number of blocks of the first blocks as the third ratio, and take the ratio of the number of repeated blocks to the total number of blocks of the second blocks as the fourth ratio; In response to the queuing job model file being a non-binary model file, determine the second number of repeated strings according to the queuing job model file of the queuing job and the historical job model file of the target historical job; use the ratio of the second number of repeated strings to the number of strings of the historical job model file as the third ratio, and use the ratio of the second number of repeated strings to the number of strings of the queuing job model file as the fourth ratio; Determine the second similarity according to the third ratio and the fourth ratio.

8. The method according to claim 1, wherein For each computing node, submitting the node allocation job corresponding to the computing node to the computing node includes: For each computing node, determine the system type corresponding to the computing node; In response to the system type being the windows system, submit the node allocation job corresponding to the computing node to the computing node through the restricted external interface provided by the service layer configured on the computing node; In response to the system type being the linux system, submit the node allocation job corresponding to the computing node to the computing node through the open external interface of the computing node.

9. The method according to claim 1, wherein It further includes: Based on a preset period, determine the submitted jobs corresponding to each computing node from the platform database, and obtain the job status of each submitted job; Display the node address of each computing node, the submitted jobs corresponding to the computing node, and the job status of each submitted job on the target browsing interface.

10. The method according to claim 1, characterized in that, It further includes: In response to the user identity being the target identity, obtain the management operation; In response to the management operation being a cluster scale control operation, perform an increase operation or a decrease operation on the computing nodes according to the cluster scale control operation; In response to the management operation being a node load control operation, determine the target node and the load quantity according to the node load control operation, and set the load quantity as the maximum number of jobs of the target node.

Images