A job scheduling and resource allocation system in a high-performance computing cluster

Abstract

The application discloses a kind of job scheduling and resource allocation system in high-performance computing cluster, it is related to high-performance computing technical field, the system includes job scheduler, resource manager and newly added job DAG parser, cluster data distribution monitor, cache preheating priority calculator, collaborative decision engine, distributed cache preheating executor and performance optimization module, job DAG parser constructs job dependent directed acyclic graph and analyzes critical path, cache preheating priority calculator determines data preheating priority using dynamic priority drift algorithm, collaborative decision engine coordinates the resource allocation of scheduling and preheating, and performance optimization module dynamically adjusts system parameters, the system realizes job scheduling and cache preheating depth collaboration, reduces job access delay, improves cluster computing efficiency and resource utilization, and is suitable for processing complex job workflow high-performance computing cluster.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing technology, specifically to a job scheduling and resource allocation system in a high-performance computing cluster. Background Technology

[0002] High-performance computing clusters serve as critical infrastructure for solving complex scientific computing and engineering simulation problems, and their job scheduling and resource allocation efficiency directly impacts overall computing performance. With the expansion of computing scale and the increase in application complexity, the dependencies between jobs within the cluster become increasingly complex, and the impact of data locality on computing performance becomes more significant. Traditional job scheduling systems primarily focus on the dynamic allocation and load balancing of computing resources, failing to fully consider the guiding role of job dependency topology in data prefetching strategies. This results in computing nodes frequently remaining idle while waiting for remote data, leading to a waste of valuable computing resources.

[0003] In existing technologies, many studies have attempted to optimize cluster resource allocation strategies. For example, Hu Yahong et al. proposed a resource scheduling scheme that considers application and node characteristics in their paper "Heterogeneous Cluster Node and Job Characteristic Awareness Resource Allocation Algorithm" (Computer Engineering and Applications, No. 18, 2022). This scheme analyzes the CPU-intensive and memory-intensive characteristics of jobs, and combines static and dynamic performance indicators of nodes to allocate jobs to the nodes most suitable for their type. Although this scheme shortens job execution time to some extent, its optimization focus is still limited to matching computing resources with job types, and it does not involve data caching and warming based on inter-job dependencies.

[0004] Another related paper, "An Application Parallel Parameter Optimization Method Integrated into a Supercomputing Job Scheduling System" (ChinaXiv:202208.00107), aims to automatically determine the optimal application parallel parameters and hardware usage to improve computational scalability. However, its core method lies in the optimization of parallel parameters and does not solve the problem of proactive pre-fetching and caching of data required before job execution. In particular, it lacks a collaborative mechanism for data preheating of job workflows with complex dependencies.

[0005] The core problem that urgently needs to be addressed is the disconnect between job scheduling and data caching / warming processes in existing high-performance computing clusters. When allocating computing resources, the job scheduler typically assumes that the data required by the job is already available or readily accessible, failing to deeply integrate dependency awareness based on directed acyclic graphs into the scheduling decisions and dynamically adjust cache warming priorities accordingly. This disconnect leads to significant access delays at the start of job execution, often due to the failure to cache required data in a timely manner on local or neighboring nodes, thus hindering further improvements in the overall computing efficiency of the cluster.

[0006] Therefore, there is an urgent need for a new technical solution that can deeply coordinate the cache preheating process with job scheduling. By parsing the DAG structure of the job workflow and dynamically adjusting the data preheating priority based on the real-time status of the cluster, precise synchronization between job execution and data availability can be achieved, effectively reducing the idle waiting time of computing nodes. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a job scheduling and resource allocation system for high-performance computing clusters. By integrating job DAG parsing, cluster data distribution monitoring, dynamic priority drift algorithm, collaborative decision-making, and distributed cache preheating execution functions, it achieves job scheduling collaboration with job dependency awareness and real-time cluster state adaptation for data preheating. It synchronizes data availability with job execution rhythm, reduces idle time on computing nodes due to waiting for data, overcomes the shortcomings of job scheduling and data cache preheating disconnect in existing technologies, and improves the overall operating efficiency of high-performance computing clusters.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a job scheduling and resource allocation system in a high-performance computing cluster, comprising a job scheduler and a resource manager, and further comprising:

[0009] The job DAG parser is used to parse the job workflow submitted by the user and construct a directed acyclic graph representing the dependencies between jobs.

[0010] Cluster data distribution monitor is used to monitor the data storage status and network topology status of each node in the cluster in real time.

[0011] The cache preheating priority calculator calculates the preheating priority of the data blocks required by each job based on the job dependencies in the directed acyclic graph and the real-time status of the cluster.

[0012] The collaborative decision engine coordinates resource allocation between the job scheduler and the cache preheating process based on the priority sequence output by the cache preheating priority calculator.

[0013] The distributed cache preheating executor performs cross-node data prefetching operations according to the instructions of the collaborative decision engine;

[0014] The cache preheating priority calculator mentioned above uses a dynamic priority drift algorithm to adjust the data preheating priority of subsequent jobs in real time according to the job completion status in the directed acyclic graph.

[0015] The collaborative decision engine coordinates the cache preheating process with the job scheduling process for synchronization between data preheating and job execution.

[0016] Furthermore, the job DAG parser specifically includes:

[0017] The dependency extraction module is used to extract data dependencies and control dependencies between jobs from the job workflow description file;

[0018] The Directed Acyclic Graph (DAG) construction module abstracts jobs as nodes in the graph and the dependencies between jobs as directed edges, thus constructing a complete DAG.

[0019] The critical path analysis module identifies the critical path in a directed acyclic graph using the critical path analysis method, and calculates the earliest start time, latest start time, and time difference for each task node.

[0020] The metadata association module associates each job node with the metadata of the data blocks it needs to access;

[0021] The critical path analysis module uses the following formula to calculate the criticality of a job node:

[0022]

[0023] in Indicates the work node The criticality, Indicates the work node The latest start time, Indicates the work node The earliest start time, Indicates the work node The degree of exit, This represents the maximum value among the earliest start times of all job nodes. This represents the minimum earliest start time among all job nodes.

[0024] Furthermore, the cluster data distribution monitor includes:

[0025] The data location tracking module records the storage location and replica distribution of each data block in the cluster;

[0026] The network topology discovery module obtains the topology of the cluster network through the link layer discovery protocol, including the switch hierarchy and link bandwidth information.

[0027] The real-time load monitoring module continuously collects CPU utilization, memory usage, network bandwidth utilization, and storage input / output performance metrics for each node.

[0028] The data popularity statistics module calculates the access frequency and access patterns of each data block within a preset time window.

[0029] The cache status maintenance module maintains the list of data blocks in the cache of each node, as well as their cache time and access time information;

[0030] The network topology discovery module employs an improved topology discovery algorithm, calculating the data transmission cost between nodes using the following formula:

[0031]

[0032] in Indicates from node To the node Data transmission cost, Represents a node To the node Network hop count, Represents a node To the node Available bandwidth, Represents a node To the node Average link utilization on the path Represents a node To the node Data transmission delay, , , These are the weighting coefficients.

[0033] Furthermore, the cache preheating priority calculator employs a dynamic priority drift algorithm, specifically including:

[0034] The initial priority calculation module calculates the initial warm-up priority of each job data block based on the job's position in the directed acyclic graph and the job's estimated execution time.

[0035] The event response module monitors job completion events and triggers priority recalculation when job completion is detected.

[0036] The priority adjustment module adjusts the data preheating priority of subsequent tasks based on the status of completed tasks.

[0037] The conflict resolution module arbitrates requests based on priority when multiple jobs compete for data warm-up resources.

[0038] The dynamic priority drift algorithm described above calculates the preheating priority of job data blocks using the following formula:

[0039]

[0040] in Indicate homework At any moment Preheating priority Indicate homework Initial preheating priority, This represents the dependency weight coefficient. Indicate homework The set of all direct predecessor operations, Indicates the preceding operation The criticality, Indicates the time decay coefficient. Indicates the preceding operation Completion time, This represents the resource demand weighting coefficient. Indicate homework Total resource demand Indicate homework Data block size, This represents the queue delay penalty coefficient. Indicate homework Waiting time in the queue.

[0041] Furthermore, the collaborative decision-making engine includes:

[0042] The resource allocation arbitration module allocates computing and network resources between the job scheduler and the cache preheating process;

[0043] The progress synchronization module is used to keep the data preheating progress consistent with the job scheduling progress.

[0044] The deadlock prevention module detects and resolves potential deadlock situations during resource allocation.

[0045] The Quality of Service (QoS) assurance module provides differentiated QoS assurance for jobs and data warm-up requests of different priorities.

[0046] The resource allocation arbitration module employs a multi-objective optimization method, making resource allocation decisions using the following formula:

[0047]

[0048] The constraints are satisfied:

[0049]

[0050] in Indicate homework priority, Indicate whether it is an assignment Decision variables for allocating computing resources Indicates data preheating task The delay cost, Indicates whether it is a data preheating task. Decision variables for allocating network resources Indicates operation Energy consumption cost Indicates whether to perform the operation. Decision variables, , , These are the weighting coefficients. , , These represent the upper limits for computing resources, network resources, and the number of operations, respectively.

[0051] Furthermore, the distributed cache preheating executor includes:

[0052] The data transmission planning module plans the optimal data transmission path based on the location information of the data blocks and the network topology.

[0053] The parallel transmission control module controls the simultaneous transmission of multiple data blocks, optimizing network bandwidth utilization.

[0054] The fault tolerance module handles network and node failures that occur during data transmission.

[0055] The cache replacement module evicts infrequently used data blocks based on cache space usage and using the least recently used algorithm.

[0056] The transmission progress monitoring module monitors the transmission progress and rate of each data block in real time.

[0057] The data transmission planning module employs a network-aware transmission path selection algorithm, evaluating the merits of each transmission path using the following formula:

[0058]

[0059] in This indicates the score of the transmission path. Indicates link bandwidth capacity, Indicates link Current utilization rate Indicates link The number of jumps, Indicates link The delay Indicates link The variance of delay jitter.

[0060] Furthermore, the dynamic priority drift algorithm also includes a priority decay mechanism, specifically implemented through the following formula:

[0061]

[0062] in Indicates the operation after attenuation adjustment At any moment Preheating priority Indicates the time decay rate. Indicates the initial calculation time of the priority. Represents the activation coefficient. Indicate homework At any moment Activation signal;

[0063] The activation signal The calculation formula is:

[0064]

[0065] in Indicate homework The set of sub-jobs, Indicate homework The set of dependent jobs, For indicator functions, Indicate homework The start time, Indicate homework The estimated execution time, Indicates from the assignment To the homework The weight, Indicates dependent jobs Completion time, Indicates dependent jobs The estimated cleanup time, Indicates from dependent jobs To the homework The influence weight.

[0066] Furthermore, the system also includes a performance optimization module, which dynamically adjusts system parameters using the following formula:

[0067]

[0068] in This represents a system parameter vector, including buffer size, warm-up lead time, and number of parallel transfers. Indicates the learning rate. Represents the gradient of performance metrics;

[0069] The performance indicators The calculation formula is:

[0070]

[0071] in This indicates the number of jobs within the monitoring time window. Indicate homework Execution time, Indicate homework Resource utilization rate during the execution period Indicate homework energy consumption , , These are the weighting coefficients;

[0072] The performance optimization module continuously monitors the system's operating status and automatically adjusts system parameters to optimize overall performance.

[0073] Compared with existing technologies, this job scheduling and resource allocation system in a high-performance computing cluster has the following advantages:

[0074] I. This invention, by setting up a job DAG parser, a cluster data distribution monitor, a cache preheating priority calculator, a collaborative decision engine, and a distributed cache preheating executor, firstly, the job DAG parser parses the job workflow and constructs a directed acyclic graph of dependencies between jobs. Then, the cluster data distribution monitor obtains the storage, network, and load status of cluster nodes in real time, and calculates the data preheating priority using a dynamic priority drift algorithm. Finally, the collaborative decision engine coordinates the resource allocation between the job scheduler and the cache preheating process, achieving synchronization between data preheating and job execution. This effectively solves the problem of job scheduling and data cache preheating being disconnected in existing technologies, avoids access delays caused by the failure to cache required data in a timely manner during job execution, reduces idle waiting time of computing nodes, and improves the overall computing efficiency of high-performance computing clusters.

[0075] Second, this invention establishes a performance optimization module that constructs comprehensive performance indicators based on job execution time, resource utilization, and energy consumption. It then uses a gradient adjustment method to dynamically optimize system parameters such as cache size, warm-up lead time, and number of parallel transmissions. Simultaneously, the distributed cache warm-up executor has a fault-tolerant processing mechanism and a cache replacement function based on the least recently used algorithm. The cluster data distribution monitor can accurately calculate the data transmission cost between nodes by improving the topology discovery algorithm. This enables adaptive optimization of system parameters, improves cluster resource utilization, and ensures the stability of data transmission and caching processes.

[0076] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0078] Figure 1 This is a schematic diagram illustrating the system architecture and data collaboration of the present invention;

[0079] Figure 2 This is a flowchart of the dynamic priority calculation and update process of the present invention;

[0080] Figure 3 This is a flowchart illustrating the operation of the present invention. Detailed Implementation

[0081] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0082] Example 1

[0083] like Figure 1 and Figure 2 As shown, this embodiment discloses a specific implementation of a job scheduling and resource allocation system in a high-performance computing cluster, aiming to solve the problem of idle computing nodes and low resource utilization caused by the disconnect between job scheduling and data cache preheating in the prior art. By deploying a job DAG parser, a cluster data distribution monitor, a cache preheating priority calculator, a collaborative decision engine, a distributed cache preheating executor, and a performance optimization module, it achieves job dependency parsing, real-time cluster status monitoring, dynamic cache preheating priority calculation, collaborative allocation of scheduling and preheating resources, and adaptive optimization of system parameters. Among them, the critical path analysis algorithm quantifies the criticality of jobs, the improved topology discovery algorithm calculates node transmission costs, the dynamic priority drift algorithm adjusts preheating priorities, and the multi-objective optimization algorithm achieves resource arbitration, ultimately achieving precise synchronization between job execution and data availability, improving the overall computing efficiency and resource utilization of the high-performance computing cluster.

[0084] The specific implementation process is as follows:

[0085] This embodiment applies to a high-performance computing cluster composed of several compute nodes, storage nodes, and tiered switches. The cluster is used to process scientific computing and engineering simulation workflows with complex dependencies. The deployment and workflow of each system component are described in detail below:

[0086] Deployment and workflow of the DAG parser for the job:

[0087] The job DAG parser is deployed on the cluster's scheduling and management node. It receives job workflow description files submitted by users through the cluster client. Its process includes four steps: dependency extraction, DAG construction, critical path analysis, and metadata association.

[0088] Dependency extraction:

[0089] The dependency extraction module reads the job workflow description file and identifies two types of dependencies between jobs through syntax parsing: data dependencies and control dependencies. The module determines data-dependent job pairs by matching the "input data identifier" and "output data identifier" in the description file; and determines control-dependent job pairs by identifying the "execution trigger condition" field. Both types of dependencies are stored in structured data format.

[0090] Construction of directed acyclic graph:

[0091] The Directed Acyclic Graph (DAG) construction module abstracts each job as a node in the DAG and the dependencies between jobs as directed edges: if job A is a predecessor job of job B, then a directed edge is constructed from node A to node B. The module traverses all jobs and dependencies to generate a complete DAG structure, and uses a loop detection algorithm to ensure that the DAG is free of closed loops. Finally, the DAG structure is stored in the metadata database of the scheduling management node.

[0092] Critical path analysis:

[0093] The critical path analysis module uses the critical path method to identify the critical path in the DAG and calculates the earliest start time, latest start time, and time difference for each task node. It also calculates the criticality of each task node using the following formula to quantify the impact of the task on the workflow completion time:

[0094]

[0095] In the formula: : Work Node The criticality value ranges from (0, 2]. The larger the number of work nodes The more significant the impact on workflow completion time, the more priority should be given to ensuring its resource and data needs; : Work Node The latest start time refers to the work node. If execution starts after this time, the latest start time that will cause a delay in the completion time of the entire workflow can be derived by traversing the DAG backward from the final job node. : Work Node The earliest start time refers to the work node. The earliest start time that can be started immediately after all predecessor jobs have been completed can be derived by traversing the DAG in a forward direction from the initial job node; The maximum value among the earliest start times of all job nodes, reflecting the upper limit of the entire workflow's time span; The minimum of the earliest start times of all job nodes, usually the earliest start time of the initial job node; : Work Node The out-degree refers to the distance from the work node. The number of directed edges from which a point originates. The larger the number of work nodes The more pronounced the "bottleneck effect"; : Natural constant, to ensure that the denominator is always positive, so that the second term of the formula takes values ​​in the range of (0,1], which is used to strengthen the influence of out-degree on criticality.

[0096] Metadata association:

[0097] The metadata association module reads the "data access requirements" field of each job node, extracts the data block identifier required for job execution, and obtains the metadata information of the corresponding data block from the cluster metadata database. It then binds the data block metadata information with the job node attributes to form a "job-data" association table, which is used by the cluster data distribution monitor and the cache preheating priority calculator.

[0098] The deployment and workflow of the cluster data distribution monitor are as follows:

[0099] The cluster data distribution monitor adopts a distributed deployment approach, deploying monitoring agents on each compute node, storage node, and core switch to collect cluster status data in real time. Its working process includes five steps: data location tracking, network topology discovery, real-time load monitoring, data heat statistics, and cache status maintenance.

[0100] Data location tracking:

[0101] The data location tracking module interacts with the cluster storage management system to record the storage location of each data block in real time and monitor data block migration and replica updates. The module stores the mapping relationship between data block identifiers and storage node IPs in a distributed hash table for quick lookup by other components.

[0102] Network topology discovery:

[0103] The network topology discovery module collects the connection relationships between switches and nodes through the link layer discovery protocol, constructs the hierarchical topology of the cluster network, and calculates the data transmission cost between any two nodes using the following improved topology discovery algorithm, providing a basis for data transmission path selection:

[0104]

[0105] In the formula: From node To the node The data transmission cost; the higher the value, the more data is transmitted from the node. Transmit to node The higher the cost, the more you should avoid choosing that transmission path; , , Weighting coefficients are preset based on the characteristics of the cluster network and satisfy the following conditions: ; :node To the node Network hops, referring to the number of network hops from which data travels from a node. Transmit to node The more switches and hops a packet needs to pass through, the higher the risk of transmission delay and packet loss. :node To the node Available bandwidth refers to the minimum available bandwidth of each link on the transmission path, measured in Mbps. The larger the available bandwidth, the higher the data transmission rate. :node To the node The average utilization rate of links on the path refers to the average ratio of the bandwidth used by each link on the path to the total bandwidth. The value ranges from [0,1). The higher the utilization rate, the higher the risk of transmission congestion. :node To the node Data transmission latency refers to the latency of data transmission from node Send to node The average reception time, measured in milliseconds (ms), indicates the timeliness of data transmission. The smaller the delay, the more timely the data transmission.

[0106] The module recalculates the transmission cost between each node at preset intervals to ensure that the data transmission cost reflects the real-time status of the cluster network.

[0107] Real-time load monitoring:

[0108] The real-time load monitoring module collects CPU utilization, memory utilization, network bandwidth utilization, and storage I / O performance indicators of each computing node through the node monitoring agent, and collects storage utilization and read / write response time of each storage node. The module stores the collected indicators in a time series to the monitoring database, with a sampling period of 10 seconds, and performs anomaly detection on the indicators.

[0109] Data popularity statistics:

[0110] The data popularity statistics module counts the access frequency and access patterns of each data block within a preset time window. By analyzing the job's "data access log," the module accumulates the number of times each data block is accessed within the time window, marking data blocks with access frequencies exceeding a preset threshold as "high-popularity data blocks" and prioritizing their inclusion in the cache preheating range; it also records the access patterns of data blocks to provide a reference for cache replacement strategies.

[0111] Cache state maintenance:

[0112] The cache status maintenance module monitors the list of data blocks in the local cache of each compute node in real time, and records the cache write time, the last access time and cache space occupied for each data block. When the cache space is insufficient, the module sends a cache space warning to the distributed cache warm-up executor and triggers the cache replacement operation. At the same time, the module synchronizes the cache status to the cluster metadata database to ensure that other components can obtain cache information in real time.

[0113] The deployment and workflow of the cache preheating priority calculator are as follows:

[0114] The cache preheating priority calculator is deployed on the scheduling management node and interacts in real time with the job DAG parser and cluster data distribution monitor. It uses a dynamic priority drift algorithm to calculate the preheating priority of the data blocks required by each job. Its working process includes four steps: initial priority calculation, event response, priority adjustment, and conflict resolution.

[0115] Initial priority calculation:

[0116] The initial priority calculation module is based on the "job-data" association table output by the job DAG parser and the criticality of job nodes. Based on the data block heat output by the cluster data distribution monitor, the initial preheating priority of the data blocks required by each job is calculated. The module presets initial priority calculation rules: task criticality. The higher the initial priority, the higher the popularity of the corresponding data block. The larger; for example, will Set as This ensures that the initial priority comprehensively reflects the importance of the task and the data access requirements.

[0117] Incident Response:

[0118] The event response module monitors the job execution status. When it detects that a job has been completed, it immediately triggers the recalculation of the data preheating priority for subsequent jobs. At the same time, the module monitors cluster status abnormal events. When an abnormality is detected, it triggers the recalculation of the priority of the affected data blocks.

[0119] Priority adjustment:

[0120] The priority adjustment module uses a dynamic priority drift algorithm, combined with the completion status of the predecessor job, the real-time cluster load, and queue waiting status, to adjust the data warm-up priority of subsequent jobs. The specific formula is as follows:

[0121]

[0122] In the formula: :Operation At any moment The preheating priority is determined by the value; the larger the value, the higher the priority for data blocks to be prefetched. :Operation The initial preheating priority is obtained by the initial priority calculation module; Dependency weighting coefficient, preset to 0.3~0.5, is used to adjust the degree of influence of predecessor tasks on the priority of subsequent tasks. The larger the value, the more significant the increase in priority for subsequent tasks after the precursor task is completed; :Operation The set of all direct predecessor jobs, i.e., jobs that need to be completed first. The set of jobs that must be started before it can be launched; Precursor Operations The criticality is determined by the job DAG parser. The larger the size, the more advanced the operation. For subsequent work The more significant the impact; Time decay coefficient, preset to 0.1~0.2, is used to represent the rate at which the impact of the completion time of the preceding task on the priority of subsequent tasks decays. The larger the value, the faster the decay. : The current moment; Precursor Operations The completion time is pushed by the job scheduler; Resource demand weighting coefficient, preset to 0.2~0.4, is used to adjust the degree of influence of task resource demand on priority. The larger the value, the more significant the increase in the priority of tasks with high resource requirements; :Operation The total resource demand refers to the operation The combined quantitative value of the number of CPU cores and memory capacity required for execution; :Operation Data block size refers to the size of the job. The total capacity of all required data blocks, in GB. The smaller the data block, the higher the prefetch efficiency and the smaller its proportion in priority calculation. Queue delay penalty coefficient, preset to 0.1~0.3, used to penalize jobs that wait for a long time. The larger the value, the more significantly the priority of tasks with longer waiting times is reduced. :Operation The waiting time in the cache preheating request queue, in minutes, from the job. The preheating request is submitted to the queue and the timer starts.

[0123] Meanwhile, to avoid resource consumption caused by excessively high priority over a long period, the module introduces a priority decay mechanism, which is implemented using the following formula. Adjustments will be made:

[0124]

[0125] In the formula: Operation after attenuation adjustment At any moment The preheating priority is set to ensure that the priority changes dynamically over time. Time decay rate, preset to 0.05~0.1, is used to control the natural decay rate of priority. The larger the value, the faster the priority decays. :Operation The initial calculation time for priority, i.e., the first calculation. The moment; Activation coefficient, preset to 0.2~0.3, used to adjust the effect of the activation signal on priority. The larger the value, the more significant the priority boost from the activation signal;

[0126] :Operation At any moment The activation signal is used to increase priority under specific conditions, and its calculation formula is:

[0127]

[0128] In the formula: :Operation The set of sub-jobs, i.e., dependent jobs The set of subsequent jobs to be executed;

[0129] Indicator function, when The value is 1 if the condition is met, and 0 otherwise. As a sub-homework The start time, For sub-homework The estimated execution time; From sub-jobs To the homework The weight, preset to 0.1~0.2, is used to indicate the impact of subjob initiation on the job. The degree of influence of priority; :Operation The set of dependent jobs, i.e., jobs The set of predecessor jobs that need to be completed; Indicator function, when The value is 1 if the condition is met, and 0 otherwise. For dependent jobs Completion time, For dependent jobs Estimated cleanup time; From dependent jobs To the homework The influence weight, preset to 0.2~0.3, is used to represent the impact of the completion of the dependent job cleanup on the job. The degree of impact of priority.

[0130] Conflict resolution:

[0131] When multiple jobs' data warm-up requests compete for the same network or storage resources, the conflict resolution module uses the adjusted warm-up priority. Arbitration is conducted: resources are allocated to preheating requests with higher priority; if priorities are the same, resources are allocated to requests with smaller data blocks; the arbitration result is pushed to the collaborative decision engine in real time as the basis for resource allocation.

[0132] The deployment and workflow implementation of the collaborative decision-making engine are as follows:

[0133] The collaborative decision-making engine is deployed on the scheduling management node, serving as the core interaction between the job scheduler and the distributed cache preheating executor. It realizes resource allocation arbitration, progress synchronization, deadlock prevention, and service quality assurance. Its working process includes the following four steps:

[0134] Resource allocation arbitration:

[0135] The resource allocation arbitration module employs a multi-objective optimization method to coordinate resources between job scheduling and cache preheating, ensuring that resource allocation takes into account job priority, preheating delay, and energy consumption. This is specifically achieved through the following objective function:

[0136]

[0137] Constraints:

[0138]

[0139] In the formula; , , Weighting coefficients are preset based on the cluster's operational goals and satisfy the following conditions: ; : Number of jobs to be scheduled; :Operation Priority; Is this homework? The decision variable for allocating computing resources takes the value 1 or 0; : Number of cache preheating tasks to be executed; Data preheating task The delay cost refers to the preheating task The time lost due to job delays, measured in minutes, is estimated by the cluster data distribution monitor based on transmission costs. Is this a data preheating task? The decision variable for allocating network resources takes the value 1 or 0; : Number of resource operations to be executed; :operate Energy consumption cost refers to the cost of performing the operation. The required energy consumption, in kWh, is estimated by energy consumption indicators collected by the node monitoring agent. : Whether to perform the operation The decision variable takes the value 1 or 0; The maximum amount of computing resources that can be allocated to a cluster refers to the combined maximum of the total number of CPU cores and the total memory capacity of all computing nodes in the cluster. The maximum network resources that can be allocated to the cluster refers to the sum of the total available bandwidth of all links in the cluster. The maximum number of operations that the cluster can execute is preset by the processing capacity of the cluster management node.

[0140] The module solves the above multi-objective optimization problem using a linear programming algorithm, derives a resource allocation scheme, and pushes the scheme to the job scheduler and the distributed cache warm-up executor, respectively.

[0141] Progress synchronization:

[0142] The progress synchronization module collects job scheduling progress and cache preheating progress in real time, and synchronizes the two through the following mechanism:

[0143] If the execution progress of a certain job reaches 90%, the module notifies the cache preheating executor to prefetch the data blocks required by subsequent jobs of this job.

[0144] If the preheating progress of a certain data block lags behind the job scheduling progress, the module notifies the resource allocation arbitration module to adjust the resource allocation.

[0145] The module generates a "Scheduling-Preheating Progress Synchronization Report" every 30 seconds and stores it in the scheduling log for subsequent troubleshooting and performance analysis.

[0146] Deadlock prevention:

[0147] The deadlock prevention module employs an "ordered resource allocation" strategy to avoid deadlocks during resource allocation.

[0148] The module assigns a unique priority number to cluster resources;

[0149] All components are required to request resources in ascending order of resource number and release resources in descending order of resource number.

[0150] The module monitors the resource request sequence in real time. If a request that violates the "ordered allocation" rule is detected, the request is immediately rejected and the component is prompted to adjust the request order, thus preventing deadlock at the source.

[0151] Service quality assurance:

[0152] The Quality of Service (QoS) assurance module provides differentiated QoS guarantees based on the priority of the job and preheating tasks:

[0153] For high-priority jobs, the module requires the resource allocation arbitration module to reserve at least 80% of the required computing resources and ensure that the preheating rate of the required data blocks reaches 100% before allowing the job to start.

[0154] For medium-priority jobs, the module requires that the preheating completion rate reach 80% before the job can be started, and the resource allocation priority is lower than that of high-priority jobs.

[0155] For low-priority jobs, the module allows preheating to be executed in parallel with the job, with the lowest priority resource allocation.

[0156] The module periodically evaluates the QoS guarantee performance; if it fails to meet the preset standards, the weighting coefficients are adjusted. , , .

[0157] The deployment and workflow of the distributed cache preheating executor are as follows:

[0158] The distributed cache pre-fetch executor adopts a distributed deployment approach, deploying a pre-fetch agent on each storage node. Based on instructions from the collaborative decision engine, it executes cross-node data pre-fetch operations. Its workflow includes five steps: data transmission planning, parallel transmission control, fault tolerance processing, cache replacement, and transmission progress monitoring.

[0159] Data transmission planning:

[0160] The data transmission planning module is based on the "data block storage location" and "node transmission cost" output by the cluster data distribution monitor. The resource allocation scheme of the collaborative decision-making engine evaluates the merits of each transmission path and selects the optimal path using the following formula:

[0161]

[0162] In the formula: The transmission path score is as follows: the higher the value, the better the path. The transmission path from the source node to the destination node; A single link on the path; :link The bandwidth capacity refers to the total bandwidth of the link, measured in Mbps. :link The current utilization rate, with a value range of [0,1); :link The hop count contribution value refers to the link The percentage of hops in the path; :link The delay is measured in milliseconds (ms). :link The variance of delay jitter, in milliseconds. 2 The smaller the variance, the more stable the link delay, and the more reliable the data transmission.

[0163] The module iterates through all possible transmission paths and calculates the cost of each path. The path with the highest score is selected as the data transmission path; if multiple paths have the same score, the path with the fewest hops is selected.

[0164] Parallel transmission control:

[0165] The parallel transmission control module controls the parallel transmission of multiple data blocks based on the network resources allocated by the collaborative decision engine, thereby optimizing network bandwidth utilization.

[0166] The module determines the maximum number of data blocks to be transmitted in parallel based on the network bandwidth utilization of the target node;

[0167] The module employs "traffic shaping" technology to control the transmission rate of each data block, ensuring that the total transmission rate does not exceed the available bandwidth of the target node;

[0168] For high-priority data blocks, the module assigns them a higher transmission rate weight.

[0169] Fault tolerance:

[0170] The fault tolerance module takes the following fault tolerance measures to address potential network and node failures during data transmission:

[0171] Network Failure: The module monitors the transmission link status in real time. If a link interruption is detected, it immediately recalculates the optimal path based on the data transmission planning module and restores data transmission.

[0172] Node failure: If the source node fails, the module queries the replica storage node of the data block from the cluster data distribution monitor, switches to the replica node as the new source node, and restarts the transmission; if the target node fails, the module notifies the collaborative decision engine to reassign the job execution node, and then plans the transmission path based on the new target node.

[0173] The module records the fault tolerance process in the fault tolerance log, including the fault type, handling measures, and recovery time, for subsequent optimization of fault tolerance strategies.

[0174] Cache replacement:

[0175] When the target node's cache space is insufficient, the cache replacement module uses the least recently used algorithm to evict cached data blocks:

[0176] The module reads the most recent access time of all data blocks in the target node's cache and sorts them to form an "access time list";

[0177] The module prioritizes evicting the data block with the earliest access time to free up cache space; if multiple data blocks have the same access time, the data block with the smallest size is evicted.

[0178] Before eviction, the module checks whether the data block is a "high-frequency data block". If it is a high-frequency data block, it will prioritize eviction of low-frequency data blocks to ensure that high-frequency data blocks remain in the cache.

[0179] Transmission progress monitoring:

[0180] The transmission progress monitoring module collects the transmission progress, transmission rate, and transmission delay of each data block in real time, and presents and provides feedback in the following ways:

[0181] The module updates the transmission progress every 10 seconds, generates a "data preheating progress table", and pushes it to the collaborative decision engine and user client.

[0182] If the transmission rate of a certain data block is lower than the preset threshold, the module analyzes the cause and sends optimization suggestions to the data transmission planning module or the parallel transmission control module.

[0183] Once the data block transfer is complete, the module notifies the cluster data distribution monitor to update the cache status and simultaneously notifies the job scheduler that "data is ready," allowing the job to start execution.

[0184] The deployment and workflow of the performance optimization module are as follows:

[0185] The performance optimization module is deployed on the scheduling and management node. By continuously monitoring the system's operating status and dynamically adjusting system parameters, it optimizes the overall performance of the cluster. Its working process includes three steps: performance index calculation, parameter gradient solution, and parameter tuning.

[0186] Performance index calculation:

[0187] The performance optimization module calculates performance metrics using the following formula, based on job execution data, resource usage data, and energy consumption data from the cluster monitoring database, within a preset monitoring time window. :

[0188]

[0189] In the formula: System performance index; the larger the value, the better the overall system performance. : The number of jobs monitored within the time window; , , Weighting coefficients are preset based on the cluster optimization objective and satisfy the following conditions: ; :Operation The execution time, in minutes. The smaller the value, the higher the efficiency of task execution. The larger; :Operation Resource utilization rate during the execution period refers to the operation The ratio of actual CPU and memory resources used to allocated resources, with a value range of [0,1]. The larger the size, the less resources are wasted; :Operation Energy consumption refers to the energy consumption of the operation The total electrical energy consumed during the execution process, in kWh. The smaller the value, the lower the system energy consumption.

[0190] Parameter gradient solution:

[0191] The performance optimization module will use the system parameter vector. As optimization variables, the performance index is solved using the finite difference method. gradient with respect to each parameter :

[0192] Regarding cache size Slight adjustments Calculate before and after adjustment The change in quantity, to obtain ;

[0193] For the preheating lead time Slight adjustments Calculate the change and obtain ;

[0194] For parallel transmission numbers Slight adjustments Calculate the change and obtain .

[0195] gradient It reflects the direction and degree of influence of each parameter on the performance index.

[0196] Parameter adjustment:

[0197] The module is based on the gradient descent method and dynamically adjusts system parameters using the following formula:

[0198]

[0199] In the formula: : The adjusted system parameter vector; : The system parameter vector before adjustment; The learning rate, preset to 0.01~0.05, is used to control the step size for parameter adjustment. : Gradient of performance metrics.

[0200] The module performs parameter adjustments once every monitoring time window. After adjustment, the new parameters are pushed to the corresponding system components, and the changes in performance indicators before and after parameter adjustment are recorded to form a "parameter optimization report" for continuous iterative optimization of system performance.

[0201] In summary, this embodiment achieves deep collaboration between job scheduling and data cache preheating in a high-performance computing cluster by deploying a job DAG parser, a cluster data distribution monitor, a cache preheating priority calculator, a collaborative decision engine, a distributed cache preheating executor, and a performance optimization module. It quantifies job criticality through a critical path analysis algorithm to ensure critical jobs receive resources first; optimizes data transmission paths and dynamically adjusts preheating priorities through improved topology discovery and dynamic priority drift algorithms; achieves efficient resource allocation and stable system operation through multi-objective optimization algorithms and deadlock prevention mechanisms; and enables adaptive adjustment of system parameters through the performance optimization module. This embodiment effectively solves the problem of job scheduling and cache preheating being disconnected in existing technologies, reduces idle time on computing nodes due to waiting for data, improves cluster resource utilization and job execution efficiency, and meets the needs of high-performance computing clusters in processing complex job workflows.

[0202] Example 2

[0203] like Figure 3 As shown in Example 1, this example elaborates on the specific steps of a job scheduling and resource allocation system in a high-performance computing cluster during operation. The specific steps are as follows:

[0204] Step 1: The user submits the job workflow description file through the cluster client.

[0205] Step 2: The job DAG parser receives the job workflow description file, parses the data dependencies and control dependencies between jobs, and constructs a directed acyclic graph representing the dependencies between jobs.

[0206] Step 3: The job DAG parser identifies the critical path in the directed acyclic graph using the critical path analysis method, calculates the earliest start time, latest start time, and criticality of each job node, and associates the job nodes with the required data block metadata.

[0207] Step 4: The cluster data distribution monitor monitors the data storage location, network topology, node load indicators, and data block access frequency of each node in the cluster in real time, and maintains cache status information.

[0208] Step 5: The cache preheating priority calculator calculates the initial preheating priority of the data blocks required by each job based on the job dependencies, job criticality, and real-time cluster status in the directed acyclic graph.

[0209] Step 6: The cache preheating priority calculator monitors job completion events, dynamically adjusts the data preheating priority of subsequent jobs based on the status of completed jobs, and arbitrates when multiple preheating requests compete for resources.

[0210] Step 7: The collaborative decision engine arbitrates resource allocation between job scheduling and cache preheating based on the priority sequence output by the cache preheating priority calculator, ensuring that the data preheating progress is synchronized with the job scheduling progress.

[0211] Step 8: The collaborative decision engine detects and prevents deadlocks that may occur during resource allocation, and provides differentiated quality of service assurance for jobs and data warm-up requests of different priorities.

[0212] Step 9: The distributed cache preheating executor plans the optimal data transmission path according to the instructions of the collaborative decision engine, controls the parallel transmission of multiple data blocks, handles faults during the transmission process, and manages the cache space.

[0213] Step 10: The job scheduler starts job execution after the data preheating is completed, based on the resource allocation scheme of the collaborative decision engine.

[0214] Step 11: The performance optimization module continuously monitors job execution time, resource utilization, and energy consumption, and dynamically adjusts system parameters to optimize overall performance.

[0215] Step 12: The system repeats steps 4 to 11 until all jobs are completed.

[0216] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A system for job scheduling and resource allocation in a high performance computing cluster, comprising a job scheduler and a resource manager, characterized in that, Also comprising: a job DAG parser for parsing the user-submitted job workflow and constructing a directed acyclic graph representing the inter-job dependency; a cluster data distribution monitor for monitoring the data storage state and network topology state of each node in the cluster in real time; a cache warm-up priority calculator for calculating the warm-up priority of each job-required data block according to the dependency in the directed acyclic graph and the real-time state of the cluster; the cache warm-up priority calculator employs a dynamic priority drift algorithm, which specifically comprises: an initial priority calculation module for calculating the initial warm-up priority of each job-required data block according to the position of the job in the directed acyclic graph and the estimated execution time of the job; an event response module for monitoring the job completion event and triggering priority recalculation when a job is detected to be completed; a priority adjustment module for adjusting the data warm-up priority of the subsequent jobs according to the state of the completed jobs; a conflict resolution module for arbitrating based on priority when the data warm-up requests of multiple jobs compete for resources; wherein the dynamic priority drift algorithm calculates the warm-up priority of the job data block using the following formula: wherein representing a job at time a preheat priority, representing a job an initial preheat priority, representing a dependency weight coefficient, representing a job a set of all direct predecessor jobs, representing a criticality of a predecessor job representing a time decay coefficient, representing a completion time of a predecessor job representing a resource demand weight coefficient, representing a resource demand total of a job representing a data block size of a job representing a queue delay penalty coefficient, representing a waiting time of a job in a queue;​​​​ a collaborative decision engine for coordinating the resource allocation of the job scheduler and the cache warm-up process according to the priority sequence output by the cache warm-up priority calculator; a distributed cache warm-up executor for performing cross-node data prefetching operations according to the instructions of the collaborative decision engine; wherein the cache warm-up priority calculator employs a dynamic priority drift algorithm to adjust the data warm-up priority of the subsequent jobs in real time according to the completion state of the jobs in the directed acyclic graph; the collaborative decision engine makes collaborative decisions on the cache warm-up process and the job scheduling process for synchronization between data warm-up and job execution.

2. The system for job scheduling and resource allocation in a high performance computing cluster of claim 1, wherein, the job DAG parser specifically comprises: a dependency extraction module for extracting the data dependency and control dependency between jobs from the job workflow description file; a directed acyclic graph construction module for abstracting jobs as nodes in the graph and abstracting the dependency between jobs as directed edges to construct a complete directed acyclic graph; a critical path analysis module for identifying the critical path in the directed acyclic graph through critical path analysis and calculating the earliest start time, latest start time, and time difference of each job node; a metadata association module for associating each job node with the metadata information of the data block it needs to access; wherein the critical path analysis module calculates the criticality of the job node using the following formula: wherein denotes the criticality of a job node , denotes the latest start time of a job node , denotes the earliest start time of a job node , denotes the out-degree of a job node , denotes the maximum of the earliest start times of all job nodes, denotes the minimum of the earliest start times of all job nodes.

3. The system for job scheduling and resource allocation in a high performance computing cluster of claim 1, wherein, the cluster data distribution monitor comprises: a data location tracking module for recording the storage location and replica distribution of each data block in the cluster; a network topology discovery module for obtaining the topology structure of the cluster network, including the switch hierarchy structure and link bandwidth information, through the link layer discovery protocol; a real-time load monitoring module for continuously collecting the central processing unit utilization rate, memory usage, network bandwidth utilization rate, and storage input / output performance indicators of each node; a data popularity statistics module for counting the access frequency and access pattern of each data block within a preset time window; a cache state maintenance module for maintaining the data block list in the cache of each node and its cache time and access time information; The network topology discovery module adopts an improved topology discovery algorithm, and calculates the data transmission cost between nodes by the following formula: wherein denotes the data transfer cost from node to node , denotes the network hop count from node to node , denotes the available bandwidth from node to node , denotes the average utilization of links on the path from node to node , denotes the data transfer delay from node to node , , , are weight factors.

4. The system of claim 1, wherein, The collaborative decision engine includes: The resource allocation arbitration module allocates computing resources and network resources between the job scheduler and the cache warm-up process; The progress synchronization module is used to keep the data warm-up progress consistent with the job scheduling progress; The deadlock prevention module detects and solves the deadlock situation that may occur in the resource allocation process; The quality of service guarantee module provides differentiated quality of service guarantee for jobs and data warm-up requests of different priorities; The resource allocation arbitration module adopts a multi-objective optimization method, and makes resource allocation decisions by the following formula: Satisfy the constraint condition: wherein denotes the priority of the job, denotes a decision variable whether to allocate computing resources for the job, denotes a delay cost of the data warm-up task, denotes a decision variable whether to allocate network resources for the data warm-up task, denotes an energy consumption cost of the operation, denotes a decision variable whether to perform the operation, , , are weight coefficients, , , denote upper limits of the number of computing resources, network resources and operations, respectively.​​​​​​ 5. The system for job scheduling and resource allocation in a high performance computing cluster of claim 1, wherein, The distributed cache warm-up executor includes: The data transmission planning module plans the optimal data transmission path according to the location information of the data block and the network topology structure; The parallel transmission control module controls the simultaneous transmission of multiple data blocks, optimizing network bandwidth utilization; The fault tolerance processing module handles network failures and node failures that occur during data transmission; The cache replacement module eliminates infrequently used data blocks according to the cache space usage according to the least recently used algorithm; The transmission progress monitoring module monitors the transmission progress and transmission rate of each data block in real time; The data transmission planning module adopts a network-aware transmission path selection algorithm, and evaluates the pros and cons of each transmission path by the following formula: wherein denotes a score of a transmission path, denotes a bandwidth capacity of a link , denotes a current utilization of a link , denotes a hop count of a link , denotes a delay of a link , denotes a delay jitter variance of a link .

6. The system for job scheduling and resource allocation in a high performance computing cluster of claim 1, wherein, The dynamic priority drift algorithm also includes a priority decay mechanism, which is implemented by the following formula: wherein represents a work adjusted by decay at time a preheat priority, represents a work at time a preheat priority, represents a time decay rate, represents a priority initial calculation time, represents an activation coefficient, represents a work at time an activation signal; The activation signal The calculation formula is: in Indicate homework The set of sub-jobs, Indicate homework The set of dependent jobs, For indicator functions, Indicate homework The start time, Indicate homework The estimated execution time, Indicates from the assignment To the homework The weight, Indicates dependent jobs Completion time, Indicates dependent jobs The estimated cleanup time, Indicates from dependent jobs To the homework The influence weight.

7. The system of claim 1, wherein, The system also includes a performance optimization module, which dynamically adjusts system parameters by the following formula: wherein : adjusted system parameter vector; : unadjusted system parameter vector; denotes a system parameter vector, including cache size, warm-up lead time, number of parallel transmissions, denotes a learning rate, denotes a gradient of the performance indicator; The performance indicator The formula for calculating the performance indicator is: wherein represents the number of jobs within a monitoring time window, represents the execution time of a job , represents the resource utilization during the execution of a job , represents the energy consumption of a job , , , is a weight coefficient; The performance optimization module automatically adjusts system parameters to optimize overall performance by continuously monitoring system operation status.

Images