Job scheduling method and device, computer device, readable storage medium and program product

Abstract

The application relates to a job scheduling method and device, computer equipment, a readable storage medium and a program product. A historical similar job set corresponding to a to-be-scheduled job is determined, a demand resource prediction value corresponding to the historical similar job set is predicted based on the historical similar job set, a target job type is determined according to the demand resource prediction value, a target node corresponding to the target job type is determined based on the demand resource prediction value, the target job type and a job type currently processed by each node, and the job is processed by the target node. Compared with the traditional job scheduling based on the user-applied resources, the application utilizes the historical similar job set similar to the job to obtain the demand resource prediction value corresponding to the job, so that the demand resources allocated for the job are more accurate, the job is allocated to the target node meeting the job type requirement by using the demand resource prediction value for processing, and the technical effect of improving the accuracy of job scheduling is achieved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a job scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the launch of the supercomputing internet platform, domestically produced computing power has become more accessible and easier to use, leading to a rapid increase in the platform's user base. The amount of data from jobs requiring scheduling and execution on the platform is also constantly growing. Jobs require corresponding resources for processing during scheduling and execution. Currently, job scheduling is typically based on the resources requested by users. However, the resources requested by users do not accurately match the actual resource needs of the jobs, thus reducing the accuracy of resource allocation during job scheduling.

[0003] Therefore, current job scheduling methods suffer from low scheduling accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a job scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve scheduling accuracy in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a job scheduling method, including:

[0006] Obtain the jobs to be scheduled and determine the set of historical similar jobs corresponding to the jobs;

[0007] Based on the historical set of similar jobs, the predicted resource demand for the corresponding job is obtained;

[0008] Based on the predicted resource demand, determine the target job type corresponding to the job;

[0009] The target node corresponding to the task is determined based on the predicted resource demand for the task, the target task type, and the task type currently being processed in each node.

[0010] The job is assigned to the target node for processing.

[0011] In one embodiment, determining the set of historical similar jobs corresponding to the job includes:

[0012] Obtain the job characteristics corresponding to the job;

[0013] Based on the job characteristics, determine the set of historical similar jobs corresponding to the job from historical log data.

[0014] In this embodiment, the server utilizes the job characteristics corresponding to the job to obtain a set of historically similar jobs from historical log data, thereby improving the accuracy of obtaining the set of historically similar jobs.

[0015] In one embodiment, the job features include text features; the text features characterize the text type features in the job features;

[0016] The step of determining the set of historical similar jobs corresponding to the job from historical log data based on the job characteristics includes:

[0017] Based on the historical log data, a corresponding mapping dictionary is generated;

[0018] Based on the mapping dictionary, the historical log data and the text features corresponding to the job are mapped respectively to obtain each set of text-similar jobs corresponding to the historical log data, and the vector of the text features corresponding to the job;

[0019] Based on the hash value of the vector and each of the text similarity task sets, the corresponding text feature similarity task set is obtained;

[0020] The historical similar job set is obtained based on the similarity matching results between the job and the set of jobs with similar text features.

[0021] In this embodiment, the server can utilize historical log data, combined with a mapping dictionary and the textual features of the job, to identify historical similar jobs that are similar to the job, and form a set of historical similar jobs. This improves the matching degree between each historical similar job in the historical job set and the job to be scheduled, thereby improving the accuracy of obtaining the set of historical similar jobs.

[0022] In one embodiment, predicting the predicted resource demand for the job based on the historical similar job set includes:

[0023] Based on the number of historically similar jobs in the historical similar job set, determine the prediction strategy corresponding to the job;

[0024] Based on the prediction strategy, predict the required resources for the corresponding task.

[0025] In this embodiment, the server determines the corresponding prediction strategy based on the number of historical similar jobs in the historical similar job set to predict the required resource quantity, thereby improving the prediction accuracy of the required resource quantity.

[0026] In one embodiment, determining the prediction strategy corresponding to the job based on the number of historically similar jobs in the historically similar job set includes:

[0027] If the number of historically similar jobs in the historically similar job set is greater than the job count threshold, then the prediction strategy is determined to be prediction based on a regression model.

[0028] And / or,

[0029] If the number of historically similar jobs in the historically similar job set is less than or equal to the job count threshold, then the prediction strategy is determined to be prediction based on the mean method.

[0030] In this embodiment, the server determines the predicted amount of demand resources by comparing the number of historical similar jobs in the historical similar job set with the job number threshold, using both regression models and the mean method. Different methods are used for prediction when the number of historical similar jobs is different. For example, when the number is large, the accuracy of the model is higher, and when the number is small, the accuracy of the mean method is higher, thereby improving the prediction accuracy of demand resources.

[0031] In one embodiment, the predicted resource demand includes a target amount of processor resources and a target amount of memory resources;

[0032] The step of predicting the resource demand forecast corresponding to the operation according to the prediction strategy includes:

[0033] Obtain the processor resource request amount and memory resource request amount for the job input;

[0034] The processor resource request quantity is used as the processor resource target quantity in the predicted demand quantity;

[0035] The predicted amount of memory resources corresponding to the job is determined according to the prediction strategy, and the target amount of memory resources in the predicted amount of memory resources is determined according to the comparison result between the predicted amount of memory resources and the requested amount of memory resources.

[0036] In this embodiment, the server uses different methods to determine the target amount of processor resources and the target amount of memory resources when determining the predicted amount of various resource demands, thereby improving the prediction accuracy of various resource demands.

[0037] In one embodiment, the prediction strategy includes making predictions based on a regression model;

[0038] Determining the predicted memory resource amount corresponding to the job according to the prediction strategy includes:

[0039] Based on the target amount of each historical memory resource corresponding to each historical similar job in the historical similar job set, the regression model to be trained is trained to obtain the trained regression model.

[0040] The job is input into the trained regression model, which then outputs a predicted amount of memory resources based on the job's characteristics.

[0041] Through this embodiment, when there are a large number of similar historical jobs, the server can determine the prediction strategy to be based on a regression model, train the regression model, and use the regression model to predict the target amount of memory resources for the job, thereby improving the prediction accuracy of the target amount of memory resources.

[0042] In one embodiment, the prediction strategy includes prediction based on the mean method;

[0043] Determining the predicted memory resource amount corresponding to the job according to the prediction strategy includes:

[0044] Obtain the target amount of each historical memory resource corresponding to each historical similar job in the historical similar job set;

[0045] The predicted memory resource quantity for the job is obtained based on the average value of each of the historical memory resource target quantities.

[0046] In this embodiment, when the number of similar historical jobs is small, the server can determine the prediction strategy as a mean-based prediction method, using the average value of historical memory resource predictions to determine the memory resource prediction amount, thereby improving the prediction accuracy of the target memory resource amount.

[0047] In one embodiment, determining the target amount of memory resources in the predicted amount of memory resources based on the comparison result of the predicted amount of memory resources and the requested amount of memory resources includes:

[0048] If the predicted amount of memory resources is greater than the requested amount of memory resources, then the requested amount of memory resources will be used as the target amount of memory resources.

[0049] If the predicted amount of memory resources is less than or equal to the requested amount of memory resources, then the predicted amount of memory resources will be used as the target amount of memory resources.

[0050] In this embodiment, the server can determine the memory resources that better meet the resource requirements of the processing job by comparing the predicted memory resource quantity with the requested memory resource quantity, thereby improving the accuracy of memory resource prediction.

[0051] In one embodiment, the predicted resource demand includes a target amount of processor resources and a target amount of memory resources;

[0052] The step of determining the target job type corresponding to the job based on the predicted resource demand includes:

[0053] Obtain the ratio of the target amount of memory resources to the target amount of processor resources;

[0054] Based on the comparison result between the ratio and the ratio threshold, the target job type corresponding to the job is determined.

[0055] In this embodiment, the server can determine the target job type based on the ratio of the target amount of memory resources to the target amount of processor resources, thereby improving the accuracy of job allocation.

[0056] In one embodiment, determining the target node corresponding to the job based on the predicted resource demand for the job, the target job type, and the job type currently being processed in each node includes:

[0057] If the target job type is not included in the currently processed job type in the node, then the node is determined to be the target node;

[0058] If the target job type is included in the current job type being processed in each of the nodes, then the remaining amount of each required resource corresponding to each node is obtained, and the node with the largest remaining amount of required resources is determined as the target node.

[0059] In this embodiment, the server can prioritize job allocation to nodes with currently unprocessed target job types, and then allocate jobs based on the remaining amount of required resources. This avoids new jobs being the same as existing jobs in nodes, which would lead to resource contention and system performance degradation, and improves the balance of job scheduling.

[0060] In one embodiment, each of the nodes is marked with a type tag indicating the type of job currently being processed;

[0061] After assigning the job to the target node corresponding to the target job type for processing, the process further includes:

[0062] If the target job type is not currently being processed in the target node, then the type tag corresponding to the target job type marked on the target node is removed.

[0063] In this embodiment, the server removes the corresponding type marker after the target node has finished processing the above-mentioned job and the target job type does not exist in the target node, thereby improving the accuracy and real-time performance of detecting the job type processed in the node.

[0064] Secondly, this application also provides a job scheduling device, comprising:

[0065] The acquisition module is used to acquire the job to be scheduled and determine the set of historical similar jobs corresponding to the job;

[0066] The prediction module is used to predict the required resource quantity corresponding to the job based on the historical similar job set.

[0067] The classification module is used to determine the target job type corresponding to the job based on the predicted amount of demand resources.

[0068] The determination module is used to determine the target node corresponding to the job based on the predicted demand resources corresponding to the job, the target job type, and the job type currently being processed in each node;

[0069] The scheduling module is used to allocate the job to the target node for processing.

[0070] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0071] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0072] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0073] The aforementioned job scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product determine the historical similar job set corresponding to the job to be scheduled, predict the corresponding resource demand based on the historical similar job set, determine the target job type based on the resource demand forecast, and determine the target node corresponding to the job based on the resource demand forecast, the target job type, and the job type currently being processed by each node. The target node then processes the job. Compared to traditional job scheduling based on user-requested resources, this solution utilizes the historical similar job set to obtain the resource demand forecast corresponding to the job, making the resource demand allocation for the job more accurate. By using the resource demand forecast, the job is allocated to the target node that meets the job type requirements for processing, thereby improving the technical effect of job scheduling accuracy. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a flowchart illustrating a job scheduling method in one embodiment;

[0076] Figure 2 This is a flowchart illustrating the prediction steps in one embodiment;

[0077] Figure 3 This is a flowchart illustrating the job scheduling method in another embodiment;

[0078] Figure 4 This is a structural block diagram of a job scheduling device in one embodiment;

[0079] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0081] In related technologies, with the launch of supercomputing internet platforms, domestic computing power has become more accessible and easier to use, leading to a rapid increase in the number of platform users. Since these users come from various industries, their job resource requirements (such as CPU (Central Processing Unit), memory, etc.) exhibit diversity and imbalance. Furthermore, the accuracy of resource requests provided by most users is poor, further exacerbating resource fragmentation. This results in low cluster resource utilization, inefficient scheduling, and even long job wait times. These problems are particularly pronounced when dealing with jobs with high CPU and memory requirements.

[0082] Currently, resource-aware job allocation methods typically employ static resource partitioning scheduling. These methods divide the nodes in a cluster into fixed partitions based on their hardware configuration or resource characteristics (such as CPU and memory). The scheduling system then allocates jobs to specific resource partitions according to their resource requirements. Once a partition is set, the purpose of a node is fixed and typically does not adjust to the dynamic needs of a job.

[0083] However, classifying and scheduling jobs based on their resource requirements ignores the reliability of those resources, which are often far higher than the actual needs of the job. Therefore, the job classifications based on existing technologies may be inaccurate, leading to significant resource waste.

[0084] Based on this, this application obtains the predicted resource demand for a job by utilizing a set of historical similar jobs that are similar to the job, making the resource demand allocated to the job more accurate. The predicted resource demand is then used to allocate the job to the target node that meets the job type requirements for processing, thereby improving the technical effect of job scheduling accuracy.

[0085] In one embodiment, such as Figure 1 As shown, a job scheduling method is provided. This embodiment illustrates the method by applying it to a server. It is understood that this method can also be applied to a terminal, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server, including the following steps S202 to S210. Wherein:

[0086] Step S202: Obtain the job to be scheduled and determine the set of historical similar jobs corresponding to the job.

[0087] The jobs to be scheduled can be those on the server that require scheduling. These jobs can be allocated to corresponding target nodes for processing after the server predicts their resource requirements. To determine the resources needed for a job to be scheduled, the server can make predictions by combining the resources from a set of historically similar jobs. For example, the server can determine the set of historically similar jobs corresponding to the job to be scheduled. This set includes multiple historically similar jobs, which are those with a similarity greater than a similarity threshold. The server can extract features from the job to be scheduled, perform feature-based similarity matching in the historical jobs to obtain these historically similar jobs, and then generate the set of historically similar jobs based on these multiple similar jobs.

[0088] Step S204: Based on the historical set of similar jobs, predict the required resource quantity for each job.

[0089] After obtaining the aforementioned set of historical similar jobs, the server can use each historical similar job in the set to predict the resource requirements of the jobs to be scheduled, thus obtaining the corresponding predicted resource requirements. For example, the server predicts the predicted resource requirements for the jobs based on the aforementioned set of historical similar jobs. The predicted resource requirements represent the predicted amount of resources needed to process the jobs. The predicted resource requirements can be calculated using different prediction strategies based on the number of historical similar jobs in the set. The predicted resource requirements can include various types of resource requirements.

[0090] For example, in some embodiments, the aforementioned resource demand forecasts include, but are not limited to, target processor resources and target memory resources. The target processor resources represent the forecasted amount of CPU resources required to process the aforementioned job, and the target memory resources represent the forecasted amount of memory required to process the aforementioned job.

[0091] Step S206: Determine the target job type corresponding to the job based on the predicted demand resources.

[0092] Once the server determines the predicted resource requirements for the aforementioned tasks, it can determine the target task type based on these predicted resource requirements. For example, the predicted resource requirements may include the predicted quantities of various types of resources. The server can then determine the target task type based on a comparison of the predicted quantities of these different types of resources. Specifically, the predicted resource requirements may include target processor resources and target memory resources. The server can determine the target task type by comparing the target processor resources with the target memory resources.

[0093] Step S208: Determine the target node corresponding to the job based on the predicted demand resources, the target job type, and the job type currently being processed in each node.

[0094] After determining the predicted resource demand, the server can identify target nodes. A target node is a node used to process the aforementioned tasks. Target nodes can be nodes that meet both the task type requirements and the resource demand requirements. The server can have multiple nodes, and it can detect the task type currently being processed by each node. Each node can process one or more task types. The server determines the target node for each task based on the predicted resource demand, the target task type, and the task types currently being processed by each node.

[0095] The server can query each node based on the target job type represented by the aforementioned predicted resource demand, and can also compare the predicted resource demand with the remaining resource demand of each node. Thus, the server determines the target node corresponding to the target job type based on one or both of the above query and comparison methods. The remaining resource demand represents the amount of remaining resources in the node available for processing the job.

[0096] In related technologies, job scheduling only considers whether the available resources of a node meet the requirements of a new job, ignoring the marking of existing job load types on the node. Therefore, a new job may be of the same type as existing jobs on the node, leading to resource contention and system performance degradation. Based on this, when determining a target node, the server in this application can first detect the type of job currently being processed on the node, and then detect the remaining amount of the node's required resources.

[0097] In one embodiment, determining the target node corresponding to a job based on the predicted demand resources for the job, the target job type, and the job types currently being processed in each node includes: if the target job type is not included in the job types currently being processed in a node, then the node is determined as the target node; if the target job type is included in the job types currently being processed in each node, then the remaining demand resources for each node are obtained, and the node with the largest remaining demand resources is determined as the target node.

[0098] In this embodiment, when determining a target node, the server can first detect the type of job currently being processed in the node. For each node, if the server detects a target job type that does not include the aforementioned job to be scheduled among the currently processed job types in the node, the server can determine the node as the target node. The remaining resource requirement of the target node can be greater than the predicted resource requirement. The type of job currently being processed in the node can be obtained by the server marking the nodes with different types and detecting the currently marked types in each node; alternatively, it can be obtained by the server detecting the types of jobs being processed in each node.

[0099] If the server detects that the currently processed jobs on all the aforementioned nodes include the target job type, it means that each node on the server is processing a job corresponding to the target job type. The server can then obtain the remaining resources for each node. The server selects the node with the largest remaining resource requirement as the target node, thereby evenly distributing jobs and balancing the load across all nodes on the server.

[0100] In this embodiment, the server can prioritize job allocation to nodes with currently unprocessed target job types, and then allocate jobs based on the remaining amount of required resources. This avoids new jobs being the same as existing jobs in nodes, which would lead to resource contention and system performance degradation, and improves the balance of job scheduling.

[0101] Step S210: Assign the job to the target node for processing.

[0102] Once the server identifies the target node, it can assign the aforementioned jobs to that node for processing. The server then uses the target node's predicted resource requirements to allocate the appropriate resources for processing the job. After the target node completes processing the job, the server can also update the type of job currently being processed by that node.

[0103] In one embodiment, after assigning the job to the target node corresponding to the target job type for processing, the method further includes: if the job type currently being processed in the target node does not include the target job type, then removing the type tag corresponding to the target job type marked on the target node.

[0104] In this embodiment, each of the aforementioned nodes is marked with a type tag indicating the type of job currently being processed. Therefore, the server can determine the job type corresponding to the job being processed within a node by detecting the type tag on the node.

[0105] The server can update the type tag of the target node after the target node has finished processing the aforementioned job. For example, after the target node finishes processing the aforementioned job, the server can obtain the job type corresponding to the currently processed job in the target node. If the server detects that the currently processed job type in the target node does not include the target job type, the server removes the type tag corresponding to the target job type from the target node. This improves the real-time performance of the node and provides other jobs that call the node with real-time node processing status.

[0106] Specifically, the server can obtain the JT (Job Type) of a job. When allocating resources for a new job, the server prioritizes allocating it to nodes without JT type jobs. If none are available, the new job is allocated to a node that meets the resource requirements. After allocation, the server marks the allocated node as having a JT type job, forming a type tag. When the job completes, if there are no JT type jobs on the allocated node, the server clears the JT type tag and stores the job information in the database.

[0107] In this embodiment, the server removes the corresponding type marker after the target node has finished processing the above-mentioned job and the target job type does not exist in the target node, thereby improving the accuracy and real-time performance of detecting the job type processed in the node.

[0108] In the aforementioned job scheduling method, a set of historical similar jobs corresponding to the job to be scheduled is determined. Based on this set, the predicted resource demand is calculated. The target job type is then determined according to the predicted resource demand. Finally, based on the predicted resource demand, the target job type, and the job types currently being processed by each node, the target node is determined, and the target node processes the job. Compared to traditional job scheduling based on user-requested resources, this scheme utilizes a set of historical similar jobs to obtain the predicted resource demand for the job, making the resource demand allocation for the job more accurate. By using the predicted resource demand, the job is assigned to a target node that meets the job type requirements for processing, thus improving the accuracy of job scheduling.

[0109] In related technologies, job scheduling systems collect various indicators and store them in a database. However, most of the data is not fully utilized to analyze job characteristics and node resource consumption, making it impossible to rationally allocate queued jobs based on resource consumption characteristics, which will result in a waste of computing resources.

[0110] In one embodiment, determining the set of historical similar jobs corresponding to a job includes: obtaining job features corresponding to the job; and determining the set of historical similar jobs corresponding to the job from historical log data based on the job features.

[0111] In this embodiment, when detecting historical similar job sets, the server can identify and obtain them based on the job features corresponding to the jobs to be scheduled. For example, the server can obtain the job features corresponding to the jobs and determine the historical similar job set corresponding to the jobs from the server's historical log data based on the job features. The historical log data can be related data of various historical jobs processed within a historical time period stored in the server's database. The server can identify multiple historical similar jobs with a similarity greater than a similarity threshold from the historical log data based on the job features, forming a historical similar job set. The job features include various features, such as, but not limited to, text features and numerical features. The server can obtain the historical similar job set based on text features.

[0112] In this embodiment, the server utilizes the job characteristics corresponding to the job to obtain a set of historically similar jobs from historical log data, thereby improving the accuracy of obtaining the set of historically similar jobs.

[0113] In one embodiment, determining the historical similar job set corresponding to a job from historical log data based on job features includes: generating a corresponding mapping dictionary based on the historical log data; mapping the historical log data and the text features corresponding to the job based on the mapping dictionary to obtain each text similar job set corresponding to the historical log data and a vector of the text features corresponding to the job; obtaining the corresponding text feature similar job set based on the hash value of the vector and each text similar job set; and obtaining the historical similar job set based on the similarity matching result between the job and the text feature similar job set.

[0114] In this embodiment, the job features mentioned above include, but are not limited to, text features and numerical features. Text features refer to the text-type characteristics in the job; numerical features refer to the characteristics corresponding to the numerical values ​​in the job. For example, text features include, but are not limited to, five text features in the job: username, group name, queue name, job path, and application type; numerical features include, but are not limited to, four numerical features in the job: the number of CPUs requested by the user, the number of nodes requested by the user, the user's estimated job execution time, and the amount of memory resources requested.

[0115] Among them, the username represents the user account identifier that submits the job, the group name represents the system permission group to which the user belongs, the queue name represents the target scheduling queue for the job submission, the working path represents the file storage directory when the job runs, the application type represents the software or task category to which the job belongs, the number of CPUs requested by the user represents the number of CPU cores requested by the job (processor resource request), the number of nodes requested by the user represents the number of computing nodes that the job needs to occupy, and the memory resource request represents the user's estimated memory usage.

[0116] When retrieving historical similar jobs, the server can do so based on text features. For example, the server can extract the job features corresponding to the job, obtain the text features from these job features, and generate a corresponding mapping dictionary d based on historical log data. The server can then map the historical log data and the text features of the jobs according to the mapping dictionary d. That is, the server maps the historical log data based on the mapping dictionary d to obtain a set of text-similar jobs corresponding to the historical log data; and maps the text features of the jobs based on the mapping dictionary d to obtain a vector of text features corresponding to the jobs. The server can then combine the hash value of the vector with each set of text-similar jobs to obtain a set of text-feature-similar jobs. The set of text-similar jobs includes all historical jobs in the historical log data whose text similarity to the mapping dictionary d is greater than a similarity threshold; the set of text-feature-similar jobs includes all similar jobs whose text features have a similarity greater than a similarity threshold.

[0117] The server can then perform similarity matching between the aforementioned assignments and a set of assignments with similar text features, obtaining the corresponding similarity matching results. Based on these similarity matching results, the server obtains a set of historical assignments. For example, the server performs similarity matching between the assignments and each historical assignment in the set of assignments with similar text features, and identifies historical assignments with similarity scores greater than a similarity threshold as historical similar assignments. The aforementioned similarity thresholds can be set according to actual circumstances.

[0118] In this embodiment, the server can utilize historical log data, combined with a mapping dictionary and the textual features of the job, to identify historical similar jobs that are similar to the job, and form a set of historical similar jobs. This improves the matching degree between each historical similar job in the historical job set and the job to be scheduled, thereby improving the accuracy of obtaining the set of historical similar jobs.

[0119] In one embodiment, predicting the required resource quantity for a job based on a set of historical similar jobs includes: determining a prediction strategy for the job based on the number of historical similar jobs in the set of historical similar jobs; and predicting the required resource quantity for the job based on the prediction strategy.

[0120] In this embodiment, the server can predict the resource demand corresponding to a job based on a prediction strategy. This prediction strategy can include various methods and can be determined based on historical similar jobs. For example, the server can obtain the number of historical similar jobs in the historical similar job set and determine the prediction strategy corresponding to the job based on this number. Specifically, the server can compare the number of historical similar jobs in the similar job set with a job count threshold, and then determine the prediction strategy based on the comparison result. After determining the prediction strategy, the server can predict the resource demand corresponding to the job according to the prediction strategy.

[0121] In this embodiment, the server determines the corresponding prediction strategy based on the number of historical similar jobs in the historical similar job set to predict the required resource quantity, thereby improving the prediction accuracy of the required resource quantity.

[0122] In one embodiment, determining the prediction strategy for a job based on the number of historically similar jobs in the historically similar job set includes: if the number of historically similar jobs in the historically similar job set is greater than a job count threshold, then the prediction strategy is determined to be prediction based on a regression model. And / or, if the number of historically similar jobs in the historically similar job set is less than or equal to the job count threshold, then the prediction strategy is determined to be prediction based on the mean method.

[0123] In this embodiment, the server determines a prediction strategy based on the aforementioned historical similar job set, which can be based on the number of historical similar jobs. The server can preset a job count threshold, which can be set according to actual conditions. The server can determine whether the number of historical similar jobs in the aforementioned historical similar job set exceeds the aforementioned job count threshold. If so, the server determines that the prediction strategy is based on a regression model. If the server determines to use a regression model for prediction, it can train a trained regression model using the historical similar job set, and use the regression model to predict the aforementioned jobs to obtain the predicted resource demand. The regression model can be a neural network model obtained through iterative training based on a regression algorithm.

[0124] If the server detects that the number of historically similar jobs in the historical similar job set is less than or equal to the job count threshold, the server determines that the prediction strategy is based on the mean method. The mean method can be used to determine the predicted resource demand by calculating the average value of the predicted resource demand corresponding to historically similar jobs.

[0125] In this embodiment, the server determines the predicted amount of demand resources by comparing the number of historical similar jobs in the historical similar job set with the job number threshold, using both regression models and the mean method. Different methods are used for prediction when the number of historical similar jobs is different. For example, when the number is large, the accuracy of the model is higher, and when the number is small, the accuracy of the mean method is higher, thereby improving the prediction accuracy of demand resources.

[0126] In one embodiment, predicting the required resource quantity for a job according to a prediction strategy includes: obtaining the processor resource request quantity and memory resource request quantity for the job input; using the processor resource request quantity as the processor resource target quantity in the required resource quantity; determining the memory resource prediction quantity for the job according to the prediction strategy; and determining the memory resource target quantity in the required resource quantity based on the comparison result between the memory resource prediction quantity and the memory resource request quantity.

[0127] In this embodiment, the predicted resource demand can include various types of resources. For example, the predicted resource demand includes, but is not limited to, target amounts of processor resources and target amounts of memory resources. Furthermore, for different types of resource demand, the server can employ different strategies to determine the predicted amount for the corresponding resource type.

[0128] For example, the server can obtain the processor resource request amount and memory resource request amount from the user for the above job input. The processor resource request amount represents the processor resources requested by the user to process the job, and the memory resource request amount represents the memory resources requested by the user to process the job.

[0129] Regarding the target amount of processor resources, since the number of CPUs in the job scheduling system of the server is fixed, the predicted amount can be obtained based on the user's request volume when allocating processor resources to jobs. For example, the server can use the processor resource request volume as the target amount of processor resources in the predicted resource demand.

[0130] For the target amount of memory resources, the server can make predictions using the prediction strategies described above. For example, the server determines the predicted amount of memory resources for a job based on the prediction strategy, and then determines the target amount of memory resources in the predicted resource demand by comparing the predicted amount of memory resources with the memory resource request amount and based on the comparison result.

[0131] In this embodiment, the server uses different methods to determine the target amount of processor resources and the target amount of memory resources when determining the predicted amount of various resource demands, thereby improving the prediction accuracy of various resource demands.

[0132] In one embodiment, determining the predicted memory resource amount corresponding to a job according to a prediction strategy includes: training a regression model to be trained based on the target amount of each historical memory resource corresponding to each historical similar job in a historical similar job set, to obtain a trained regression model; inputting the job into the trained regression model, and having the regression model output the corresponding predicted memory resource amount based on the job features corresponding to the job.

[0133] In this embodiment, the server can predict the target amount of memory resources using the aforementioned prediction strategy. This prediction strategy includes prediction based on a regression model. After determining that the prediction strategy is based on a regression model, the server can obtain the target amounts of historical memory resources for each historically similar job in the historical similar job set. Here, the historical memory resource target amount represents the target amount of memory resources corresponding to each historically similar job. The server can train the regression model to be trained based on the target amounts of historical memory resources for each historically similar job in the historical similar job set, obtaining a trained regression model. Specifically, the server can use each historically similar job and each historical memory resource target amount as samples to iteratively train the regression model, obtaining the trained regression model.

[0134] The server can input the jobs to be scheduled into a trained regression model. The regression model extracts features from the jobs to obtain corresponding job features, and then performs feature recognition based on these job features, outputting the corresponding predicted memory resource quantity. Here, the predicted memory resource quantity represents the target amount of memory resources that requires further evaluation to determine whether the predicted memory resource quantity can be used as the target amount of memory resources.

[0135] Through this embodiment, when there are a large number of similar historical jobs, the server can determine the prediction strategy to be based on a regression model, train the regression model, and use the regression model to predict the target amount of memory resources for the job, thereby improving the prediction accuracy of the target amount of memory resources.

[0136] In one embodiment, determining the predicted memory resource amount corresponding to a job according to a prediction strategy includes: obtaining the target amount of each historical memory resource corresponding to each historical similar job in a historical similar job set; and obtaining the predicted memory resource amount corresponding to the job based on the average value of each historical memory resource target amount.

[0137] In this embodiment, the server can use the aforementioned determined prediction strategy to predict the target amount of memory resources. This prediction strategy includes prediction based on the mean method. The mean method refers to a method for predicting the target amount of memory resources based on an average value. After determining that the prediction strategy is based on the mean method, the server can obtain the target amount of each historical memory resource corresponding to each historical similar job in the historical similar job set. Here, the historical memory resource target amount represents the target amount of memory resources corresponding to each historical similar job. The server can obtain the average value of each historical memory resource target amount based on the historical memory resource target amounts corresponding to each historical similar job in the historical similar job set, and obtain the predicted amount of memory resources corresponding to the job to be scheduled based on the average value.

[0138] In this embodiment, when the number of similar historical jobs is small, the server can determine the prediction strategy as a mean-based prediction method, using the average value of historical memory resource predictions to determine the memory resource prediction amount, thereby improving the prediction accuracy of the target memory resource amount.

[0139] In one embodiment, determining the target amount of memory resources in the predicted amount of memory resources based on the comparison result of the predicted amount of memory resources and the requested amount of memory resources includes: if the predicted amount of memory resources is greater than the requested amount of memory resources, then the requested amount of memory resources is used as the target amount of memory resources; if the predicted amount of memory resources is less than or equal to the requested amount of memory resources, then the predicted amount of memory resources is used as the target amount of memory resources.

[0140] In this embodiment, after the server predicts the amount of memory resources based on the prediction strategy, it needs to further determine the predicted amount of memory resources. Specifically, a user can request the use of corresponding memory resources on the server; the memory resources requested by the user are represented as the memory resource request amount. The server can compare the predicted amount of memory resources with the memory resource request amount from the user, and based on the comparison result, determine the target amount of memory resources.

[0141] If the server detects that the predicted amount of memory resources is greater than the requested amount of memory resources, it means that the user does not need too much memory resources, and the server will use the requested amount of memory resources as the target amount of memory resources. If the server detects that the predicted amount of memory resources is less than or equal to the requested amount of memory resources, it means that the user has requested too much memory resources, and the server can use the more accurate predicted amount of memory resources as the target amount of memory resources.

[0142] Specifically, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating the prediction steps in one embodiment. The server can predict the memory required for a job based on historical log data. Specifically, the server combines similarity search and machine learning, searching for historically similar jobs in the historical job set based on textual and numerical features, and training a regression model on this set for prediction. To ensure prediction accuracy, the server can also periodically update the training data and retrain the regression model.

[0143] In the process of predicting the target amount of memory resources, the server can obtain historical log data D and the aforementioned job. new The server can generate a mapping dictionary d based on the text features in the historical log data D. The server uses the mapping dictionary d to map the historical log data D and the job. new The text features are mapped to 0 and 1 respectively to obtain the matrix F corresponding to the historical log data D, and the job. new The corresponding vector v.

[0144] The server can hash the matrix F to obtain a signature matrix S, and then hash the signature matrix S to obtain m subsets of text-similar jobs, i.e., the text-similar job set, which can be specifically represented as {s1, s2, ..., sm}. The server can also obtain the job based on the hash value of the vector v. new The corresponding text feature similarity task set s i The server can use Euclidean distance to determine the above job. new Similar to text features, the task set s i The similarity is used to obtain the above-mentioned historical similar job set L.

[0145] The server can determine whether the number of historically similar jobs in the aforementioned historical similar job set L exceeds the job count threshold K, thereby determining the corresponding prediction strategy. If yes, a regression model is used for prediction; otherwise, the mean method is used. The server obtains the job based on the prediction strategy. newThe corresponding predicted memory resource quantity `pre_mem` can also be used to obtain the predicted execution time `pre_time`. The server can check whether the predicted memory resource quantity is greater than the user's requested memory resource quantity `req_mem`. If so, the server determines the target memory resource quantity `mem` to be the requested memory resource quantity `req_mem`, specifically expressed as `mem = req_mem`; otherwise, the server determines the target memory resource quantity `mem` to be the predicted memory resource quantity `pre_mem`, specifically expressed as `mem = pre_mem`, and updates the job's target memory resource quantity to `mem`.

[0146] In this embodiment, the server can determine the memory resources that better meet the resource requirements of the processing job by comparing the predicted memory resource quantity with the requested memory resource quantity, thereby improving the accuracy of memory resource prediction.

[0147] In one embodiment, determining the target job type corresponding to the job based on the predicted demand resources includes: obtaining the ratio of the target amount of memory resources to the target amount of processor resources; and determining the target job type corresponding to the job based on the comparison result of the ratio and the ratio threshold.

[0148] In this embodiment, the predicted resource demand includes target processor resources and target memory resources. When determining the target job type, the server can combine the target processor resources and target memory resources from the predicted resource demand to determine the target job type. For example, the server can obtain the ratio of the target memory resources to the target processor resources and compare the ratio with a ratio threshold. Based on the comparison result, the server determines the target job type corresponding to the job. The ratio threshold can be set according to actual conditions. When the ratio is greater than the ratio threshold, the server can determine the target job type as a first job type; when the ratio is less than or equal to the ratio threshold, the server can determine the target job type as a second job type. The first job type and the second job type are different.

[0149] Specifically, the first job type mentioned above can be memory-intensive, and the second job type mentioned above can be processor-intensive (CPU-intensive). Memory-intensive jobs primarily consume large amounts of memory resources, while processor-intensive jobs primarily rely on high CPU computing resources. The server can classify jobs based on their computing resource requirements (the number of CPUs requested by the job and the predicted amount of memory), categorizing them into CPU-intensive and memory-intensive types, specifically expressed as JT = mem / cpu. Here, JT represents the job type, mem represents the target amount of memory resources, and cpu represents the target amount of processor resources, which can be the number of cores requested by the user. When JT is greater than a certain ratio threshold, the job is considered memory-intensive; otherwise, it is considered CPU-intensive. This ratio threshold can be determined through multiple experiments to achieve the optimal performance for the scheduling system.

[0150] In this embodiment, the server can determine the target job type based on the ratio of the target amount of memory resources to the target amount of processor resources, thereby improving the accuracy of job allocation.

[0151] In one exemplary embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating the job scheduling method in another embodiment. In this embodiment, the server can perform predictions based on historical log data from its job scheduling system. Since the number of CPUs in the server's job scheduling system is fixed, the server can predict the target amount of memory resources required for each job. The job features used by the server for prediction include, but are not limited to, five textual features: username, group name, queue name, work path, and application type; and four numerical features: the number of CPUs requested by the user, the number of nodes requested by the user, the user's estimated job execution time, and the user's estimated memory usage.

[0152] After the server predicts the target amount of memory resources, it can classify the job type based on the CPU and memory resource requirements and the ratio threshold.

[0153] The server can also perform resource allocation and job scheduling. The allocation rule can be that the server prioritizes assigning jobs to nodes that do not already have the same job type. After allocation, the server marks the existing job types on the nodes. After a job completes, if there are no jobs of the same type on the node, the server removes that job type from the node's markings. The server can also write completed job information to the database and periodically update job data to the prediction module to retrain the regression model and ensure prediction accuracy.

[0154] Through the above embodiments, the server utilizes a set of historical similar jobs to obtain the predicted resource requirements for each job, making the resource allocation for jobs more accurate. By using these predicted resource requirements, jobs are assigned to target nodes that meet the job type requirements for processing, thereby improving the accuracy of job scheduling. Furthermore, by predicting the memory required by a job, the classification of job types based on resource requirements becomes more accurate, while avoiding unnecessary resource consumption. Based on the job type and existing job type tags on nodes, jobs of the same type are isolated on different nodes as much as possible, while different types of jobs are assigned to the same node, thus rationally allocating resources and improving the resource utilization of large-scale clusters in the supercomputing internet.

[0155] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0156] Based on the same inventive concept, this application also provides a job scheduling apparatus for implementing the job scheduling method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more job scheduling apparatus embodiments provided below can be found in the limitations of the job scheduling method described above, and will not be repeated here.

[0157] In one exemplary embodiment, such as Figure 4 As shown, a job scheduling device is provided, including: an acquisition module 500, a prediction module 502, a classification module 504, a determination module 506, and a scheduling module 508, wherein:

[0158] The acquisition module 500 is used to acquire jobs to be scheduled and determine the set of historical similar jobs corresponding to the jobs.

[0159] The prediction module 502 is used to predict the required resource quantity for a job based on a set of historical similar jobs.

[0160] The classification module 504 is used to determine the target job type corresponding to the job based on the predicted amount of demand resources.

[0161] The determination module 506 is used to determine the target node corresponding to the job based on the predicted amount of demand resources corresponding to the job, the target job type, and the job type currently being processed in each node.

[0162] The scheduling module 508 is used to allocate jobs to target nodes for processing.

[0163] In one embodiment, the acquisition module 500 is used to acquire the job characteristics corresponding to the job; and determine the set of historical similar jobs corresponding to the job from historical log data based on the job characteristics.

[0164] In one embodiment, the acquisition module 500 is used to generate a corresponding mapping dictionary based on historical log data; map the historical log data and the text features corresponding to the jobs according to the mapping dictionary to obtain each set of text-similar jobs corresponding to the historical log data and the vector of the text features corresponding to the jobs; obtain the corresponding set of text feature-similar jobs based on the hash value of the vector and each set of text-similar jobs; and obtain the historical similar job set based on the similarity matching result between the jobs and the set of text feature-similar jobs.

[0165] In one embodiment, the prediction module 502 is used to determine the prediction strategy corresponding to the job based on the number of historical similar jobs in the historical similar job set; and to predict the required resource quantity corresponding to the job based on the prediction strategy.

[0166] In one embodiment, the prediction module 502 is used to determine the prediction strategy as prediction based on a regression model if the number of historical similar jobs in the historical similar job set is greater than the job number threshold.

[0167] In one embodiment, the prediction module 502 is used to determine the prediction strategy as a mean-based prediction if the number of historical similar jobs in the historical similar job set is less than or equal to the job number threshold.

[0168] In one embodiment, the prediction module 502 is configured to obtain the processor resource request amount and memory resource request amount for the job input; use the processor resource request amount as the processor resource target amount in the demand resource prediction amount; determine the memory resource prediction amount corresponding to the job according to the prediction strategy; and determine the memory resource target amount in the demand resource prediction amount according to the comparison result of the memory resource prediction amount and the memory resource request amount.

[0169] In one embodiment, the prediction module 502 is used to train the regression model to be trained based on the target amount of each historical memory resource corresponding to each historical similar job in the historical similar job set, so as to obtain the trained regression model; input the job into the trained regression model, and the regression model outputs the corresponding memory resource prediction amount based on the job characteristics corresponding to the job.

[0170] In one embodiment, the prediction module 502 is used to obtain the target amount of each historical memory resource corresponding to each historical similar job in the historical similar job set; and to obtain the predicted amount of memory resources corresponding to the job based on the average value of each historical memory resource target amount.

[0171] In one embodiment, the prediction module 502 is configured to use the memory resource request as the memory resource target if the predicted memory resource quantity is greater than the memory resource request quantity; and to use the predicted memory resource quantity as the memory resource target quantity if the predicted memory resource quantity is less than or equal to the memory resource request quantity.

[0172] In one embodiment, the classification module 504 is used to obtain the ratio of the target amount of memory resources to the target amount of processor resources; and to determine the target job type corresponding to the job based on the comparison result of the ratio and the ratio threshold.

[0173] In one embodiment, the determining module 506 is used to determine a node as a target node if the current job type being processed in a node does not include the target job type; and if the current job type being processed in each node includes the target job type, then obtain the remaining amount of each required resource corresponding to each node and determine the node with the largest remaining amount of required resources as the target node.

[0174] In one embodiment, the above apparatus further includes: a marking module, configured to remove the type mark corresponding to the target job type marked on the target node if the currently processed job type in the target node does not include the target job type.

[0175] Each module in the aforementioned job scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0176] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores job data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a job scheduling method.

[0177] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0178] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the job scheduling method described above.

[0179] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the job scheduling method described above.

[0180] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the job scheduling method described above.

[0181] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0183] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0184] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A job scheduling method, characterized in that, The method includes: Obtain the jobs to be scheduled and determine the set of historical similar jobs corresponding to the jobs; Based on the historical set of similar jobs, the predicted resource demand for the corresponding job is obtained; Based on the predicted resource demand, determine the target job type corresponding to the job; The target node corresponding to the task is determined based on the predicted resource demand for the task, the target task type, and the task type currently being processed in each node. The job is assigned to the target node for processing.

2. The method according to claim 1, characterized in that, Determining the set of historical similar jobs corresponding to the job includes: Obtain the job characteristics corresponding to the job; Based on the job characteristics, determine the set of historical similar jobs corresponding to the job from historical log data.

3. The method according to claim 2, characterized in that, The task features include text features; the text features characterize the text type features within the task features. The step of determining the set of historical similar jobs corresponding to the job from historical log data based on the job characteristics includes: Based on the historical log data, a corresponding mapping dictionary is generated; Based on the mapping dictionary, the historical log data and the text features corresponding to the job are mapped respectively to obtain each set of text-similar jobs corresponding to the historical log data, and the vector of the text features corresponding to the job; Based on the hash value of the vector and each of the text similarity task sets, the corresponding text feature similarity task set is obtained; The historical similar job set is obtained based on the similarity matching results between the job and the set of jobs with similar text features.

4. The method according to claim 1, characterized in that, The step of predicting the resource demand forecast corresponding to the job based on the historical similar job set includes: Based on the number of historically similar jobs in the historical similar job set, determine the prediction strategy corresponding to the job; Based on the prediction strategy, predict the required resources for the corresponding task.

5. The method according to claim 4, characterized in that, The step of determining the prediction strategy corresponding to the job based on the number of historically similar jobs in the historically similar job set includes: If the number of historically similar jobs in the historically similar job set is greater than the job count threshold, then the prediction strategy is determined to be prediction based on a regression model. And / or, If the number of historically similar jobs in the historically similar job set is less than or equal to the job count threshold, then the prediction strategy is determined to be prediction based on the mean method.

6. The method according to claim 4, characterized in that, The predicted resource demand includes target amounts for processor resources and target amounts for memory resources; The step of predicting the resource demand forecast corresponding to the operation according to the prediction strategy includes: Obtain the processor resource request amount and memory resource request amount for the job input; The processor resource request quantity is used as the processor resource target quantity in the predicted demand quantity; The predicted amount of memory resources corresponding to the job is determined according to the prediction strategy, and the target amount of memory resources in the predicted amount of memory resources is determined according to the comparison result between the predicted amount of memory resources and the requested amount of memory resources.

7. The method according to claim 6, characterized in that, The prediction strategy includes prediction based on a regression model; Determining the predicted memory resource amount corresponding to the job according to the prediction strategy includes: Based on the target amount of each historical memory resource corresponding to each historical similar job in the historical similar job set, the regression model to be trained is trained to obtain the trained regression model. The job is input into the trained regression model, which then outputs a predicted amount of memory resources based on the job's characteristics.

8. The method according to claim 6, characterized in that, The prediction strategy includes prediction based on the mean method; Determining the predicted memory resource amount corresponding to the job according to the prediction strategy includes: Obtain the target amount of each historical memory resource corresponding to each historical similar job in the historical similar job set; The predicted memory resource quantity for the job is obtained based on the average value of each of the historical memory resource target quantities.

9. The method according to claim 6, characterized in that, The step of determining the target amount of memory resources in the predicted amount of memory resources based on the comparison result of the predicted amount of memory resources and the requested amount of memory resources includes: If the predicted amount of memory resources is greater than the requested amount of memory resources, then the requested amount of memory resources will be used as the target amount of memory resources. If the predicted amount of memory resources is less than or equal to the requested amount of memory resources, then the predicted amount of memory resources will be used as the target amount of memory resources.

10. The method according to claim 1, characterized in that, The predicted resource demand includes target amounts for processor resources and target amounts for memory resources; The step of determining the target job type corresponding to the job based on the predicted resource demand includes: Obtain the ratio of the target amount of memory resources to the target amount of processor resources; Based on the comparison result between the ratio and the ratio threshold, the target job type corresponding to the job is determined.

11. The method according to any one of claims 1 to 10, characterized in that, The step of determining the target node corresponding to the job based on the predicted resource demand for the job, the target job type, and the job type currently being processed in each node includes: If the target job type is not included in the currently processed job type in the node, then the node is determined to be the target node; If the target job type is included in the current job type being processed in each of the nodes, then the remaining amount of each required resource corresponding to each node is obtained, and the node with the largest remaining amount of required resources is determined as the target node.

12. The method according to any one of claims 1 to 10, characterized in that, Each of the nodes is marked with a type tag indicating the type of job currently being processed; After assigning the job to the target node corresponding to the target job type for processing, the process further includes: If the target job type is not currently being processed in the target node, then the type tag corresponding to the target job type marked on the target node is removed.

13. A job scheduling device, characterized in that, The device includes: The acquisition module is used to acquire the job to be scheduled and determine the set of historical similar jobs corresponding to the job; The prediction module is used to predict the required resource quantity corresponding to the job based on the historical similar job set. The classification module is used to determine the target job type corresponding to the job based on the predicted amount of demand resources. The determination module is used to determine the target node corresponding to the job based on the predicted demand resources corresponding to the job, the target job type, and the job type currently being processed in each node; The scheduling module is used to allocate the job to the target node for processing.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images