Slurm resource scheduling optimization method based on HPC cluster and medium

Abstract

The invention discloses a Slurm resource scheduling optimization method based on an HPC cluster and a medium. A target data lake corresponding to a target resource scheduling task is obtained in real time; respectively inputting each piece of to-be-scheduled cluster data into a multi-task machine learning prediction module, and respectively calculating to obtain a predicted operation time length, a predicted resource utilization rate and a job waiting time interval of each job; respectively carrying out calculation through a job priority calculation method to obtain a target job priority ranking result; writing a target job priority ranking result into a target Slurm job queue; and performing scheduling task distribution and processing on the target Slurm job queue through a node job resource scheduling processing and feedback method to obtain a resource scheduling task processing result, and returning the resource scheduling task processing result to the Slurm controller database. The problems of low scheduling efficiency, low resource utilization rate and high multi-task prediction complexity in the prior art are solved, and the scheduling efficiency and the resource utilization rate are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cluster data processing, and particularly relates to a Slurm resource scheduling optimization method based on an HPC cluster and a medium. BACKGROUND

[0002] High Performance Computing (HPC) clusters are widely used in scientific computing, engineering simulation, and artificial intelligence tasks. Among them, Slurm, as a mainstream scheduling system, uses rule-based priority and backfill algorithms for task scheduling.

[0003] The inventor found that the prior art has the following defects in the process of implementing the present application: At present, the task scheduling is performed by using the rule-based priority and backfill algorithm, which is difficult to dynamically optimize according to the job characteristics and historical data. And although there are attempts to predict resources based on statistical modeling, most of them lack the prediction ability of deep learning, and lack deep integration with Slurm scheduling, which will lead to low scheduling efficiency, low resource utilization, high multi-task prediction complexity, and lack of adaptive optimization problems. SUMMARY

[0004] The present application provides a Slurm resource scheduling optimization method based on an HPC cluster and a medium to improve scheduling efficiency and resource utilization.

[0005] According to one aspect of the present application, a Slurm resource scheduling optimization method based on an HPC cluster is provided, comprising:

[0006] Real-time acquisition of target data lake corresponding to target resource scheduling task; wherein the target data lake includes at least one group of to-be-scheduled cluster data;

[0007] Each of the to-be-scheduled cluster data is input into a pre-trained multi-task machine learning prediction module, and each job estimated running time, estimated resource utilization and job waiting time interval are calculated;

[0008] Through a pre-set job priority calculation method, each job estimated running time, estimated resource utilization and job waiting time interval are calculated to obtain a target job priority sorting result;

[0009] Obtaining and according to the target application programming interface for interaction between the target resource scheduling task and the Slurm controller, the target job priority sorting result is written into the target Slurm job queue;

[0010] The target Slurm job queue is distributed and processed by a preset node job resource scheduling processing and feedback method, a resource scheduling task processing result corresponding to the target resource scheduling task is obtained, and the resource scheduling task processing result is returned to the Slurm controller database.

[0011] According to another aspect of the present application, a Slurm resource scheduling optimization device based on an HPC cluster is provided, which comprises:

[0012] A target data lake acquisition module is configured to acquire a target data lake corresponding to a target resource scheduling task in real time, wherein the target data lake comprises at least one group of cluster data to be scheduled.

[0013] A job predicted running time, predicted resource usage rate and job waiting time interval calculation module is configured to input each of the cluster data to be scheduled into a pre-trained multi-task machine learning prediction module, and calculate a job predicted running time, a predicted resource usage rate and a job waiting time interval for each job.

[0014] A target job priority sorting result determination module is configured to calculate a job predicted running time, a predicted resource usage rate and a job waiting time interval for each job by a preset job priority calculation method, and obtain a target job priority sorting result.

[0015] A target job priority sorting result writing module is configured to obtain a target application programming interface for interaction between the target resource scheduling task and a Slurm controller, and write the target job priority sorting result into a target Slurm job queue.

[0016] A resource scheduling task processing result determination and return module is configured to distribute and process a target Slurm job queue by a preset node job resource scheduling processing and feedback method, obtain a resource scheduling task processing result corresponding to the target resource scheduling task, and return the resource scheduling task processing result to a Slurm controller database.

[0017] According to another aspect of the present application, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the Slurm resource scheduling optimization method based on an HPC cluster according to any one of the embodiments of the present application when executing the computer program.

[0018] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for causing a processor to implement the HPC cluster-based Slurm resource scheduling optimization method according to any of the embodiments of the present application when executed.

[0019] The technical solution of the embodiment of the present application solves the problems of low scheduling efficiency, low resource utilization and high complexity of multi-task prediction in the prior art, improves the scheduling efficiency and resource utilization, reduces the complexity of multi-task prediction and scheduling, and improves the flexibility of resource scheduling.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0022] Figure 1 is a flowchart of an HPC cluster-based Slurm resource scheduling optimization method according to the first embodiment of the present application;

[0023] Figure 2 is a structural schematic diagram of an HPC cluster-based Slurm resource scheduling optimization device according to the second embodiment of the present application;

[0024] Figure 3It is a structural schematic diagram of an electronic device provided according to an embodiment three of the present application. DETAILED DESCRIPTION

[0025] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0026] It should be noted that the terms "target", "current" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] It is worth noting that in the technical scheme of the present application, the collected information is information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal; if the user chooses to refuse, enter the expert decision-making process.

[0028] Embodiment one

[0029] Figure 1 A flowchart of a Slurm resource scheduling optimization method based on an HPC cluster is provided for embodiment one of the present application. The present embodiment is applicable to the case of optimizing the resource scheduling of task data in an HPC cluster through Slurm. The method can be executed by a Slurm resource scheduling optimization device based on an HPC cluster, which can be realized in the form of hardware and / or software.

[0030] Correspondingly, as shown in Figure 1 the method comprises:

[0031] S110: Real-time acquisition of the target data lake corresponding to the target resource scheduling task.

[0032] The target data lake includes at least one set of cluster data to be scheduled.

[0033] In this embodiment, the data source for the cluster data to be scheduled can be a statistical database, a monitoring system, or system logs. For a statistical database, this may include job submission information, runtime, or resource allocation. For a monitoring system, this may include CPU utilization, memory usage, or I / O bandwidth. For system logs, this may include job failure information and waiting times.

[0034] Specifically, the target data lake includes one or more sets of cluster data to be scheduled, and the data in the target data lake can be multi-source data.

[0035] S120. Input the data of each cluster to be scheduled into the pre-trained multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization and job waiting time interval of each job.

[0036] The multi-task machine learning prediction module can be a prediction model module constructed using a multi-task learning framework. It can share a feature extraction layer and can separately predict the parameter sizes for the estimated job runtime, estimated resource utilization, and job waiting time interval. The estimated job runtime can be the estimated runtime required for processing each cluster of data to be scheduled. The estimated resource utilization can be the estimated resource utilization required for processing each cluster of data to be scheduled. The job waiting time interval can be the specific details of the waiting time interval required for processing each cluster of data to be scheduled.

[0037] Optionally, the step of inputting the data of each cluster to be scheduled into a pre-trained multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval for each job includes: extracting features from the data of each cluster to be scheduled using a static feature extraction method to obtain static joint descriptive features for each data; wherein the static joint descriptive features include job scale descriptive features, resource request parameter descriptive features, and user submission history statistical features; extracting features from the data of each cluster to be scheduled using a dynamic feature extraction method to obtain dynamic joint descriptive features for each data; wherein the dynamic joint descriptive features include scheduling queue status features and real-time load monitoring index features; processing the job scale descriptive features, resource request parameter descriptive features, user submission history statistical features, scheduling queue status features, and real-time load monitoring index features for each cluster to be scheduled using a pre-set feature standard and normalization processing method to obtain unified cluster data numerical features corresponding to each cluster data; and inputting the unified cluster data numerical features into the multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval for each job.

[0038] In this embodiment, the first step is to extract features from each cluster data to be scheduled using a static feature extraction method to obtain static joint descriptive features. Specifically, job scale descriptive features may include the number of jobs, the number of requesting nodes, and the number of CPUs. Resource request parameter descriptive features may include memory and time limits. User submission history statistical features may include average job duration and submission frequency.

[0039] Furthermore, features are extracted using a dynamic feature extraction method to obtain dynamic joint descriptive features for each data point. These dynamic joint descriptive features include scheduling queue status features and real-time load monitoring metric features. Specifically, scheduling queue status features can include waiting queue length and idle resource ratio.

[0040] Furthermore, by using feature standards and normalization methods, we can first standardize the characteristics of each job scale description, resource request parameter description, user submission history statistics, scheduling queue status, and real-time load monitoring indicators, and then perform normalization to transform heterogeneous features into a unified numerical feature vector, which is a unified cluster data numerical feature, to generate the model input.

[0041] Correspondingly, the numerical features of the unified cluster data are input into the multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval for each job.

[0042] Additionally, after a job is completed, runtime data (actual job runtime, actual resource utilization, and actual job wait time interval) is written back to the data lake for iterative retraining of the multi-task machine learning prediction module. This improves the prediction accuracy of the multi-task machine learning prediction module.

[0043] Optionally, before inputting the numerical features of each unified cluster data into the multi-task machine learning prediction module to calculate the estimated runtime, estimated resource utilization, and job waiting time interval for each job, the method further includes: acquiring the historical data lake corresponding to each historical resource scheduling task, and the historical cluster data to be scheduled corresponding to each historical data lake; acquiring the historical estimated runtime, historical estimated resource utilization, and historical job waiting time interval corresponding to each group of historical cluster data to be scheduled; inputting the historical cluster data to be scheduled, the historical estimated runtime, the historical estimated resource utilization, and the historical job waiting time interval into the initial multi-task machine learning prediction module for model training, and optimizing the model parameters according to the pre-set multi-task loss function to obtain the trained multi-task machine learning prediction module; wherein, the multi-task loss function is obtained by formula... To be determined; among which, This represents the calculated value of the multi-task loss function. Indicates the runtime prediction loss. This indicates the predicted loss due to resource utilization. Indicates the predicted loss based on the waiting time; , , This represents an adjustable weight parameter.

[0044] Specifically, for adjustable weight parameters, initial parameter values ​​can be set, and then the weight parameters can be adjusted using pre-set parameter adjustment and optimization methods. During adjustment, it is necessary to standardize the various loss units to avoid the impact of dimensional differences on optimization. The runtime prediction loss can be calculated using mean squared error; the resource utilization prediction loss can be calculated using multi-dimensional mean squared error; and the waiting time prediction loss can be calculated using cross-entropy loss.

[0045] In this embodiment, the initial multi-task machine learning prediction module needs to be trained based on the historical data lakes corresponding to multiple historical resource scheduling tasks, the historical data lakes corresponding to each group of historical clusters to be scheduled, and the historical job expected runtime, historical expected resource utilization rate, and historical job waiting time intervals corresponding to each group of historical clusters to be scheduled. The model parameters are then optimized based on the multi-task loss function (specifically, by minimizing the total loss through an optimizer) to obtain the trained multi-task machine learning prediction module, which can support online learning to achieve fine-tuning of the model.

[0046] The advantage of this setup is that by training the multi-task machine learning prediction module, the estimated runtime of the job, the estimated resource utilization, and the job waiting time interval can be obtained more accurately and conveniently, which can better assist resource scheduling and processing operations.

[0047] S130. Using a pre-set job priority calculation method, calculate the expected runtime, expected resource utilization, and job waiting time interval for each job to obtain the target job priority ranking result.

[0048] The target job priority ranking result can be obtained by sorting the calculated target job priority weights. The job priority calculation method can be to determine the job priority corresponding to each cluster of data to be scheduled.

[0049] Optionally, the step of calculating the expected runtime, expected resource utilization, and job waiting time interval for each job using a pre-set job priority calculation method to obtain the target job priority ranking result includes: obtaining the cluster data configuration information corresponding to the cluster data to be scheduled, and obtaining the strategy score for each target cluster data using a pre-set strategy score calculation method; and, based on the expected runtime, expected resource utilization, and job waiting time interval for each job, as well as the strategy score for each target cluster data, calculating the target cluster data strategy score using the formula corresponding to the job priority calculation method. Perform job priority weight calculation to obtain the target job priority weight P corresponding to the data of the cluster to be scheduled; where... Indicates the estimated runtime of the operation; Indicates the projected resource utilization rate; Indicates the waiting time range for the task; This indicates the calculation of the normalization function; , , and This indicates that the weights are adjusted dynamically. The target cluster data strategy score is represented; the priority weights of each target job are sorted to obtain the target job priority ranking result.

[0050] The cluster data configuration information can be a description of the configuration information corresponding to each cluster data to be scheduled. Specifically, the configuration description information can include Policy type; specifically, the Fairness mode uses Gini coefficients based on... Calculate fairness; Throughput pattern uses and This is determined by the normalization function used to ensure that the input dimensions are consistent.

[0051] Among them, the strategy score calculation method can be a method that obtains different target cluster data strategy scores based on different configuration description information.

[0052] In this embodiment, it is assumed that after obtaining the specific target cluster data policy score, and determining the expected job runtime, expected resource utilization, and job waiting time interval, corresponding dynamically adjusted weight values ​​can be set to calculate the weights using the formula. The priority weight of the target task is obtained.

[0053] Suppose there are 5 groups of cluster data to be scheduled, and the calculated target job priority weights are P1, P2, P3, P4, and P5. These weights (P1, P2, P3, P4, and P5) can be sorted by value to determine the priority. The typical priority range is set as follows: The larger the value, the higher the priority, and thus the earlier the resource scheduling and processing operations will be performed.

[0054] Alternatively, it can be calculated... To determine the target backfill mechanism, small jobs are scheduled to run in idle windows. This allows for compressed allocation and resource matching. Deviations can be monitored to trigger priority recalculation and update the scheduling process.

[0055] S140. Obtain the target application programming interface for interaction between the target resource scheduling task and the Slurm controller, and write the target job priority sorting result into the target Slurm job queue.

[0056] In this embodiment, the scheduling execution interface acts as a bridge between the system and the Slurm controller management system. It is responsible for efficiently implementing the priority and resource allocation schemes generated by the scheduling decision engine, while ensuring the real-time feedback of runtime data, thus forming a closed-loop optimization. The scheduling execution interface can refer to the target application programming interface.

[0057] Optionally, the step of acquiring and writing the target job priority ranking result into the target Slurm job queue based on the target application programming interface (API) of the target resource scheduling task interacting with the Slurm controller includes: acquiring the target API of the target resource scheduling task interacting with the Slurm controller; writing the target job priority ranking result into the Slurm job queue according to the received interface control instructions through the target API, and dynamically adjusting the job ranking in the Slurm job queue to obtain the target Slurm job queue; the step of acquiring and writing the target job priority ranking result into the target Slurm job queue based on the target API of the target resource scheduling task interacting with the Slurm controller also includes: generating a target backfill strategy based on the expected job runtime, and allocating node jobs to the target Slurm job queue in conjunction with the expected resource utilization rate to obtain the target node allocation result.

[0058] In this embodiment, the Slurm controller can first be interacted with via a REST application programming interface or a remote procedure call to ensure compatibility with the Slurm architecture. Further, upon receiving interface control instructions (specifically, the scontrol command or application programming interface instructions), the priority value calculated by the scheduling decision engine, i.e., the target job priority ranking result, is written into the Slurm job queue. The job ranking in the Slurm job queue needs to be dynamically adjusted to obtain the target Slurm job queue.

[0059] Furthermore, a target backfill strategy can be generated based on the expected job runtime, and node job allocation can be performed on the target Slurm job queue in conjunction with the expected resource utilization rate to obtain the target node allocation result. Specifically, compute nodes can be specified through Slurm's sbatch or slurm_allocate_resources interface, and asynchronous request mechanisms (such as HTTP POST or gRPC) can be used to reduce scheduling latency and support high-concurrency job submission scenarios. Security can also be ensured by using OAuth2 or Kerberos authentication to ensure the security of application programming interface calls and prevent unauthorized access.

[0060] S150. Using a pre-set node job resource scheduling and feedback method, the target Slurm job queue is scheduled and processed to obtain the resource scheduling task processing result corresponding to the target resource scheduling task, and the resource scheduling task processing result is sent back to the Slurm controller database.

[0061] Among them, the node job resource scheduling and feedback method can be a method for scheduling node job resources and feeding back the calculated actual operating parameters.

[0062] Optionally, the step of distributing and processing scheduling tasks to the target Slurm job queue using a pre-set node job resource scheduling processing and feedback method to obtain resource scheduling task processing results corresponding to the target resource scheduling tasks, and then sending the resource scheduling task processing results back to the Slurm controller database, includes: allocating each node job to the corresponding target node according to the target node allocation results, the expected runtime of each job, and the expected resource utilization rate, using the node job resource scheduling processing and feedback method; and instructing the Slurm controller's health check interface to check the node status of each target node. The system monitors the target node and, if its status is faulty, provides rescheduling information and adds the node job to the job retry queue. It also optimizes the retry priority of the retry queue based on the job waiting time interval. The system uses the node job success confirmation interface of the Slurm controller to determine the target node's status. If successful, it obtains the node job processing result. After each node job completes processing, the system obtains the resource scheduling task processing result corresponding to the target resource scheduling task based on the processing results of each node job, and sends the resource scheduling task processing result back to the Slurm controller database.

[0063] In this embodiment, based on the target node allocation result, the expected runtime and expected resource utilization of each job, each node job can be allocated to the corresponding target node through a node job resource scheduling and feedback method. The target node can be a central processing unit node or a graphics processing unit cluster, and is executed through Slurm's slurmd daemon.

[0064] Furthermore, load balancing can be configured, which involves prioritizing nodes with lower loads based on the estimated job runtime and expected resource utilization to avoid node overload. A node fault detection mechanism can also be configured, instructing the Slurm controller's health check interface to monitor the node status of each target node. If a target node is found to be faulty, rescheduling information is sent back, and the node's job is added to the job retry queue. The retry priority in the job retry queue is then optimized based on the job waiting time interval.

[0065] Additionally, an interface can be configured to indicate whether the node job of the Slurm controller has been successfully acknowledged. This allows for the determination of the target node; if successful, the node job processing result is obtained. If unsuccessful, error feedback processing is required.

[0066] Correspondingly, after each node job completes its processing, the resource scheduling task processing result corresponding to the target resource scheduling task is obtained based on the processing result of each node job task, and the resource scheduling task processing result is sent back to the Slurm controller database.

[0067] Optionally, the node job task processing result includes the actual runtime of each job, the actual resource utilization rate, and the actual job waiting time interval; the step of sending the resource scheduling task processing result back to the Slurm controller database includes: performing integrity verification on the actual runtime of each job, the actual resource utilization rate, and the actual job waiting time interval respectively; after meeting the verification conditions, sending the actual runtime of each job, the actual resource utilization rate, and the actual job waiting time interval back to the Slurm controller database; instructing the Slurm controller database to obtain the received data volume in real time; if the received data volume meets the preset data volume threshold, adjusting the parameters of the multi-task machine learning prediction module to obtain the multi-task machine learning prediction module after parameter adjustment.

[0068] In this embodiment, before data is sent back, the integrity of each job's actual runtime, actual resource utilization, and actual job waiting time interval needs to be verified. If these parameters are satisfied, the data can be asynchronously written to the data lake via Kafka or a message queue (such as RabbitMQ), ensuring high throughput and low latency. Job status data can also be sent back, but this also requires integrity verification. Integrity verification may include missing value detection and format normalization, preventing dirty data from affecting model parameter adjustments.

[0069] After the data is transmitted to the Slurm controller database, the Slurm controller database needs to be used to count the amount of data. If the amount of new received data reaches the threshold, the parameters of the multi-task machine learning prediction module need to be adjusted based on the returned data. The adjusted parameters of the multi-task machine learning prediction module can improve the prediction accuracy.

[0070] Additionally, the frequency of application programming interface (API) calls needs to be dynamically adjusted based on cluster load changes (e.g., reducing priority update frequency under high load) to prevent Slurm controller overload. Extensible interfaces can also be configured to support compatibility with non-Slurm schedulers, achieving cross-platform scheduling through an abstract interface layer (such as gRPC services). For monitoring and log data, Prometheus can be integrated to monitor interface performance (e.g., latency, throughput) and error rates in real time, generating traceable scheduling logs for easy fault diagnosis and patent technology verification. Furthermore, batch job submission and scheduling are supported, improving the processing efficiency of large-scale clusters through transactional operations (such as Slurm's job_array).

[0071] The technical solution of this invention involves: acquiring the target data lake corresponding to the target resource scheduling task in real time; inputting the data of each cluster to be scheduled into a pre-trained multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval for each job; calculating the expected runtime, expected resource utilization, and job waiting time interval for each job using a pre-set job priority calculation method to obtain the target job priority ranking result; acquiring and writing the target job priority ranking result into the target Slurm job queue according to the target application programming interface for interaction between the target resource scheduling task and the Slurm controller; distributing and processing the scheduling tasks in the target Slurm job queue using a pre-set node job resource scheduling processing and feedback method to obtain the resource scheduling task processing result corresponding to the target resource scheduling task, and sending the resource scheduling task processing result back to the Slurm controller database. This solves the problems of low scheduling efficiency, low resource utilization, and high complexity of multi-task prediction in the prior art, improves scheduling efficiency and resource utilization, reduces the complexity of multi-task prediction and scheduling, and improves the flexibility of resource scheduling.

[0072] Example 2

[0073] Figure 2 This is a schematic diagram of a Slurm resource scheduling optimization device based on an HPC cluster, provided in Embodiment 2 of the present invention. The Slurm resource scheduling optimization device based on an HPC cluster provided in this embodiment can be implemented through software and / or hardware, and can be configured in a terminal device or server to implement a Slurm resource scheduling optimization method based on an HPC cluster according to an embodiment of the present invention. Figure 2As shown, the device includes: a target data lake acquisition module 210, a module 220 for calculating the estimated runtime of the job, the estimated resource utilization rate, and the job waiting time interval, a module 230 for determining the target job priority ranking result, a module 240 for writing the target job priority ranking result, and a module 250 for determining and transmitting the resource scheduling task processing result.

[0074] The target data lake acquisition module 210 is used to acquire the target data lake corresponding to the target resource scheduling task in real time; wherein, the target data lake includes at least one set of cluster data to be scheduled.

[0075] The job estimated runtime, estimated resource utilization and job waiting time interval calculation module 220 is used to input the data of each cluster to be scheduled into the pre-trained multi-task machine learning prediction module to calculate the estimated runtime, estimated resource utilization and job waiting time interval of each job respectively.

[0076] The target job priority ranking result determination module 230 is used to calculate the expected runtime, expected resource utilization and job waiting time interval of each job according to the pre-set job priority calculation method, and obtain the target job priority ranking result.

[0077] The target job priority ranking result writing module 240 is used to obtain and write the target job priority ranking result into the target Slurm job queue according to the target application programming interface of the target resource scheduling task interacting with the Slurm controller.

[0078] The resource scheduling task processing result determination and feedback module 250 is used to distribute and process the scheduling task of the target Slurm job queue through a pre-set node job resource scheduling processing and feedback method, obtain the resource scheduling task processing result corresponding to the target resource scheduling task, and send the resource scheduling task processing result back to the Slurm controller database.

[0079] The technical solution of this invention involves: acquiring the target data lake corresponding to the target resource scheduling task in real time; inputting the data of each cluster to be scheduled into a pre-trained multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval for each job; calculating the expected runtime, expected resource utilization, and job waiting time interval for each job using a pre-set job priority calculation method to obtain the target job priority ranking result; acquiring and writing the target job priority ranking result into the target Slurm job queue according to the target application programming interface for interaction between the target resource scheduling task and the Slurm controller; distributing and processing the scheduling tasks in the target Slurm job queue using a pre-set node job resource scheduling processing and feedback method to obtain the resource scheduling task processing result corresponding to the target resource scheduling task, and sending the resource scheduling task processing result back to the Slurm controller database. This solves the problems of low scheduling efficiency, low resource utilization, and high complexity of multi-task prediction in the prior art, improves scheduling efficiency and resource utilization, reduces the complexity of multi-task prediction and scheduling, and improves the flexibility of resource scheduling.

[0080] Based on the above embodiments, the job estimated runtime, estimated resource utilization, and job waiting time interval calculation module 220 can be specifically used for: extracting features from each of the scheduled cluster data using a static feature extraction method to obtain static joint descriptive features for each data; wherein, the static joint descriptive features include job scale descriptive features, resource request parameter descriptive features, and user submission history statistical features; extracting features from each of the scheduled cluster data using a dynamic feature extraction method to obtain dynamic joint descriptive features for each data; wherein, the dynamic joint descriptive features include scheduling queue status features and real-time load monitoring index features; processing each of the job scale descriptive features, resource request parameter descriptive features, user submission history statistical features, scheduling queue status features, and real-time load monitoring index features using a pre-set feature standard and normalization processing method to obtain unified cluster data numerical features corresponding to each of the scheduled cluster data; and inputting each of the unified cluster data numerical features into the multi-task machine learning prediction module to calculate the estimated runtime, estimated resource utilization, and job waiting time interval for each job.

[0081] Based on the above embodiments, a multi-task machine learning prediction module training module is further included, which can be specifically used for: before inputting the numerical features of each unified cluster data into the multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval of each job, respectively, acquiring the historical data lake corresponding to each historical resource scheduling task, and the historical cluster data to be scheduled corresponding to each historical data lake; acquiring the historical expected runtime, historical expected resource utilization, and historical job waiting time interval corresponding to each group of historical cluster data to be scheduled; inputting the historical cluster data to be scheduled, historical expected runtime, historical expected resource utilization, and historical job waiting time interval into the initial multi-task machine learning prediction module for model training, and optimizing the model parameters according to the pre-set multi-task loss function to obtain the trained multi-task machine learning prediction module; wherein, the multi-task loss function is obtained by formula To be determined; among which, This represents the calculated value of the multi-task loss function. Indicates the runtime prediction loss. This indicates the predicted loss due to resource utilization. Indicates the predicted loss based on the waiting time; , , This represents an adjustable weight parameter.

[0082] Based on the above embodiments, the target job priority ranking result determination module 230 can be specifically used to: obtain cluster data configuration information corresponding to the cluster data to be scheduled, and obtain the strategy score of each target cluster data through a pre-set strategy score calculation method; and, based on the expected runtime, expected resource utilization, and job waiting time interval of each job, as well as the strategy score of each target cluster data, calculate the target cluster data strategy score using the formula corresponding to the job priority calculation method. Perform job priority weight calculation to obtain the target job priority weight P corresponding to the data of the cluster to be scheduled; where... Indicates the estimated runtime of the operation; Indicates the projected resource utilization rate; Indicates the waiting time range for the task; This indicates the calculation of the normalization function; , , and This indicates that the weights are adjusted dynamically. The target cluster data strategy score is represented; the priority weights of each target job are sorted to obtain the target job priority ranking result.

[0083] Based on the above embodiments, the target job priority ranking result writing module 240 can be specifically used to: obtain the target application programming interface for the interaction between the target resource scheduling task and the Slurm controller; write the target job priority ranking result into the Slurm job queue according to the target application programming interface and through the received interface control instructions, and dynamically adjust the job ranking in the Slurm job queue to obtain the target Slurm job queue.

[0084] Based on the above embodiments, a target node allocation result determination module is also included, which can be specifically used to: after obtaining and interacting with the Slurm controller through the target application programming interface of the target resource scheduling task, and writing the target job priority sorting result into the target Slurm job queue, generate a target backfilling strategy based on the expected runtime of the job, and allocate node jobs to the target Slurm job queue in combination with the expected resource utilization rate to obtain the target node allocation result.

[0085] Based on the above embodiments, the resource scheduling task processing result determination and feedback module 250 can be specifically used for: allocating each node job to the corresponding target node according to the target node allocation result, the expected runtime of each job, and the expected resource utilization rate, through the node job resource scheduling processing and feedback method; instructing the health check interface of the Slurm controller to monitor the node status of each target node; if the node status of the target node is faulty, feedback rescheduling information is provided, and the node job is added to the job retry queue, and the retry priority corresponding to the job retry queue is optimized in combination with the job waiting time interval; instructing the node job success confirmation interface of the Slurm controller to judge the target node; if successful, the node job task processing result is obtained; after each node job completes processing, the resource scheduling task processing result corresponding to the target resource scheduling task is obtained according to the node job task processing result, and the resource scheduling task processing result is fed back to the Slurm controller database.

[0086] Based on the above embodiments, the node job task processing results include the actual runtime of each job, the actual resource utilization rate, and the actual job waiting time interval.

[0087] Based on the above embodiments, the resource scheduling task processing result determination and feedback module 250 may specifically be further configured to: perform integrity verification on each of the actual operation duration, actual resource utilization rate, and actual job waiting time interval of the jobs respectively, and after meeting the verification conditions, transmit each of the actual operation duration, actual resource utilization rate, and actual job waiting time interval to the Slurm controller database; instruct the Slurm controller database to obtain the received data volume in real time, and if the received data volume meets the preset data volume threshold, adjust the parameters of the multi-task machine learning prediction module to obtain the multi-task machine learning prediction module after parameter adjustment.

[0088] The Slurm resource scheduling optimization device based on the HPC cluster provided by the embodiments of the present invention can execute the Slurm resource scheduling optimization method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0089] Embodiment III

[0090] Figure 3 The structural schematic diagram of the electronic device 10 that can be used to implement Embodiment III of the present invention is shown. The electronic device aims to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0091] As Figure 3 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by at least one processor, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0092] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the Slurm resource scheduling optimization method based on HPC clusters.

[0094] In some embodiments, the HPC cluster-based Slurm resource scheduling optimization method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the HPC cluster-based Slurm resource scheduling optimization method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the HPC cluster-based Slurm resource scheduling optimization method by any other suitable means (e.g., by means of firmware).

[0095] The method includes: acquiring the target data lake corresponding to the target resource scheduling task in real time; wherein the target data lake includes at least one set of cluster data to be scheduled; inputting each of the cluster data to be scheduled into a pre-trained multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval for each job; calculating the expected runtime, expected resource utilization, and job waiting time interval for each job using a pre-set job priority calculation method to obtain the target job priority ranking result; acquiring and writing the target job priority ranking result into the target Slurm job queue according to the target application programming interface for interaction between the target resource scheduling task and the Slurm controller; distributing and processing the scheduling task in the target Slurm job queue using a pre-set node job resource scheduling processing and feedback method to obtain the resource scheduling task processing result corresponding to the target resource scheduling task, and sending the resource scheduling task processing result back to the Slurm controller database.

[0096] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0098] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0100] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0101] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0102] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0103] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0104] Example 4

[0105] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions, when executed by a computer processor, are used to execute a Slurm resource scheduling optimization method based on an HPC cluster. The method includes: real-time acquisition of a target data lake corresponding to a target resource scheduling task; wherein the target data lake includes at least one set of cluster data to be scheduled; inputting each set of cluster data to be scheduled into a pre-trained multi-task machine learning prediction module to calculate the expected runtime, expected resource utilization, and job waiting time interval for each job; calculating the expected runtime, expected resource utilization, and job waiting time interval for each job using a pre-set job priority calculation method to obtain a target job priority ranking result; acquiring and writing the target job priority ranking result into a target Slurm job queue according to the target application programming interface (API) for interaction between the target resource scheduling task and the Slurm controller; distributing and processing scheduling tasks in the target Slurm job queue using a pre-set node job resource scheduling processing and feedback method to obtain a resource scheduling task processing result corresponding to the target resource scheduling task, and sending the resource scheduling task processing result back to the Slurm controller database.

[0106] Of course, the computer-executable instructions provided in the embodiments of the present invention, which include a computer-readable storage medium, are not limited to the method operations described above, but can also perform related operations in the Slurm resource scheduling optimization based on HPC clusters provided in any embodiment of the present invention.

[0107] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0108] It is worth noting that in the above embodiments of Slurm resource scheduling optimization based on HPC clusters, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this invention.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A Slurm resource scheduling optimization method based on an HPC cluster, characterized in that, The method comprises the following steps: real-time acquisition of target data lake corresponding to target resource scheduling task; wherein the target data lake comprises at least one group of to-be-scheduled cluster data; inputting each to-be-scheduled cluster data into a pre-trained multi-task machine learning prediction module to respectively calculate each job estimated running time, estimated resource usage and job waiting time interval; calculating each job estimated running time, estimated resource usage and job waiting time interval by a pre-set job priority calculation method to obtain a target job priority ranking result; acquiring a target application programming interface for interaction between the target resource scheduling task and a Slurm controller, and writing the target job priority ranking result into a target Slurm job queue; scheduling task distribution and processing of the target Slurm job queue by a pre-set node job resource scheduling processing and feedback method to obtain a resource scheduling task processing result corresponding to the target resource scheduling task, and returning the resource scheduling task processing result to a Slurm controller database.

2. The method of claim 1, wherein, The method further comprises the following steps before the inputting step: extracting features of each to-be-scheduled cluster data by a static feature extraction method to obtain data static joint description features; wherein the data static joint description features comprise job size description features, resource request parameter description features and user submission history statistical features; extracting features of each to-be-scheduled cluster data by a dynamic feature extraction method to obtain data dynamic joint description features; wherein the data dynamic joint description features comprise scheduling queue state features and real-time load monitoring index features; processing each job size description feature, resource request parameter description feature, user submission history statistical feature, scheduling queue state feature and real-time load monitoring index feature by a pre-set feature standard and normalization processing method to obtain a unified cluster data numerical feature corresponding to each to-be-scheduled cluster data; inputting each unified cluster data numerical feature into the multi-task machine learning prediction module to respectively calculate each job estimated running time, estimated resource usage and job waiting time interval.

3. The method of claim 2, wherein, The method further comprises the following steps before the inputting step: acquiring historical data lake corresponding to each historical resource scheduling task, and each group of historical to-be-scheduled cluster data corresponding to each historical data lake; acquiring historical job estimated running time, historical estimated resource usage and historical job waiting time interval corresponding to each group of historical to-be-scheduled cluster data; The historical scheduled cluster data, the historical job estimated running time, the historical estimated resource usage, and the historical job waiting time interval are input into an initial multi-task machine learning prediction module for model training, and model parameter optimization is performed according to a pre-set multi-task loss function, to obtain a trained multi-task machine learning prediction module; Wherein, the multi-task loss function is determined by formula ; wherein, represents the multi-task loss function calculation value, represents the running time prediction loss, represents the resource utilization prediction loss, represents the waiting time prediction loss; , , represents the adjustable weight parameter.

4. The method of claim 3, wherein, The job priority calculation method is used to calculate the estimated running time, the estimated resource usage, and the job waiting time interval of each job, to obtain a target job priority ranking result, including: The cluster data configuration information corresponding to the scheduled cluster data is obtained, and a pre-set strategy score calculation method is used to obtain a target cluster data strategy score; According to the expected running time, the expected resource usage rate and the job waiting time interval of each job, and the target cluster data policy score, the job priority weight is calculated by the formula corresponding to the job priority calculation method to obtain the target job priority weight P corresponding to the to-be-scheduled cluster data. wherein, represents a job expected run length; represents an expected resource usage; represents a job wait time interval; represents a normalization function calculation; , , and represents a dynamic adjustment weight; represents a target cluster data policy score; The target job priority weights are ranked to obtain the target job priority ranking result.

5. The method of claim 4, wherein, The target application programming interface for interacting between the target resource scheduling task and the Slurm controller is obtained, and the target job priority ranking result is written into a target Slurm job queue, including: The target application programming interface for interacting between the target resource scheduling task and the Slurm controller is obtained; According to the target application programming interface, the target job priority ranking result is written into the Slurm job queue through the received interface control instruction, and the job ranking in the Slurm job queue is dynamically adjusted to obtain a target Slurm job queue; In the step of obtaining the target application programming interface for interacting between the target resource scheduling task and the Slurm controller, and writing the target job priority ranking result into the target Slurm job queue, the step further includes: A target backfill strategy is generated according to the job estimated running time, and the target Slurm job queue is allocated with node jobs in combination with the estimated resource usage, to obtain a target node allocation result.

6. The method of claim 5, wherein, The target Slurm job queue is scheduled and processed through the pre-set node job resource scheduling processing and feedback method, to obtain a resource scheduling task processing result corresponding to the target resource scheduling task, and the resource scheduling task processing result is fed back to the Slurm controller database, including: According to the target node allocation result, the job estimated running time, and the estimated resource usage, each node job is allocated to a corresponding target node through the node job resource scheduling processing and feedback method; The health check interface of the Slurm controller is instructed to monitor the node state of each target node, if the node state of the target node is a fault state, feedback information for rescheduling is fed back, and the node job is added to a job retry queue, and a job waiting time interval is combined to optimize the retry priority of the job retry queue corresponding to the job retry queue; The node job success confirmation interface of the Slurm controller is instructed to judge the target node, if successful, a node job task processing result is obtained; After each node job is completed, a resource scheduling task processing result corresponding to the target resource scheduling task is obtained according to a task processing result of each node job, and the resource scheduling task processing result is returned to a Slurm controller database.

7. The method of claim 6, wherein, The node job task processing result includes actual running time, actual resource usage and actual job waiting time interval of each job; The resource scheduling task processing result is returned to the Slurm controller database, including: The actual running time, actual resource usage and actual job waiting time interval of each job are respectively subjected to integrity check, and after the check condition is met, the actual running time, actual resource usage and actual job waiting time interval of each job are returned to the Slurm controller database; The Slurm controller database is instructed to obtain the received data amount in real time, and if the received data amount meets a preset data amount threshold, the multi-task machine learning prediction module is adjusted to obtain the multi-task machine learning prediction module after parameter adjustment.

8. An HPC cluster-based Slurm resource scheduling optimization apparatus, characterized by, It includes: A target data lake acquisition module is configured to acquire a target data lake corresponding to a target resource scheduling task in real time, wherein the target data lake includes at least one group of to-be-scheduled cluster data; A job estimated running time, estimated resource usage and job waiting time interval calculation module is configured to input each to-be-scheduled cluster data into a pre-trained multi-task machine learning prediction module to calculate an estimated running time, an estimated resource usage and a job waiting time interval of each job, respectively; A target job priority sorting result determination module is configured to calculate an estimated running time, an estimated resource usage and a job waiting time interval of each job by using a pre-set job priority calculation method to obtain a target job priority sorting result; A target job priority sorting result writing module is configured to acquire a target application programming interface for interaction between the target resource scheduling task and a Slurm controller, and write the target job priority sorting result into a target Slurm job queue; A resource scheduling task processing result determination and return module is configured to distribute and process a scheduling task of the target Slurm job queue by using a pre-set node job resource scheduling processing and feedback method to obtain a resource scheduling task processing result corresponding to the target resource scheduling task, and return the resource scheduling task processing result to a Slurm controller database.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the HPC cluster-based Slurm resource scheduling optimization method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute the HPC cluster-based Slurm resource scheduling optimization method of any one of claims 1-7.

Images