HPC cluster adaptive job scheduling method and system

Abstract

The application discloses an HPC cluster adaptive job scheduling method and system, and the scheduling method comprises the following steps: calculating the multi-dimensional resource comprehensive resource entropy of the computing nodes of the HPC cluster; constructing a weighted heterogeneous topology graph by taking the computing nodes as graph nodes and the high-speed interconnection links between the computing nodes as graph edges, calculating and outputting the graph structure entropy based on the weighted heterogeneous topology graph; performing multi-scale spectral graph convolution, edge type perception residual GAT double-path topology embedding processing on the weighted heterogeneous topology graph, taking the graph structure entropy as a gating parameter, and outputting node-level fusion embedding features; performing heat risk estimation on the graph nodes by using an adaptive Kalman filter based on the physical temperature time sequence data of the computing nodes, outputting node-level heat risks, and determining a global heat risk; determining a scheduling strategy mode based on the comprehensive resource entropy and the global heat risk; and performing job scheduling based on the scheduling strategy mode and the comprehensive score of the graph nodes. The application realizes multi-dimensional collaborative optimization scheduling.

Description

Technical Field

[0001] This invention relates to the field of HPC cluster job scheduling technology, and in particular to an adaptive job scheduling method and system for HPC clusters. Background Technology

[0002] High-performance computing (HPC) clusters typically consist of hundreds or thousands of computing nodes connected by high-speed interconnect networks, forming a large-scale distributed computing system with a complex topology. User-submitted jobs have diverse resource requirements (number of CPU cores, memory capacity, number of GPUs, network bandwidth, etc.). The cluster job scheduler is responsible for allocating jobs in the waiting queue to suitable computing nodes for execution. Its scheduling quality directly affects cluster resource utilization and average job waiting time. Regarding resource fragmentation measurement, existing methods calculate the ratio of allocated CPU cores and memory size to the total amount of resources on each physical machine, obtaining the cluster's CPU resource fragmentation rate and memory resource fragmentation rate respectively, and then combining them into a unified cluster resource fragmentation metric index. While this method achieves unified quantification of multi-dimensional resource fragmentation, it is essentially based on a proportional calculation of resource allocation, belonging to a total quantity metric, i.e., only focusing on the ratio of allocated resources to total resources. The limitation of this method is that two completely different fragmentation distribution patterns may yield the same fragmentation rate value. For example, in a cluster of 100 nodes, if fragments are evenly distributed across all nodes, it's different from having fragments concentrated on 10 nodes while the remaining nodes have complete resources. In both cases, the total resource allocation ratio is the same, but the former hinders the scheduling of all types of jobs, while the latter can still provide complete nodes for large jobs. Proportional metrics cannot distinguish between these two fundamentally different cluster states. Regarding cluster topology awareness, existing HPC schedulers typically employ a greedy strategy when allocating nodes, selecting the first set of nodes that meets resource requirements without considering the physical topology of the selected nodes. This topology-ignoring allocation method incurs two efficiency losses: first, if the selected nodes are scattered across multiple switch domains or racks, inter-process communication within a job needs to traverse multiple network hops, leading to a significant increase in communication latency; second, topology-aware allocation may cause bandwidth contention and hotspot congestion on interconnect links, reducing the overall parallel computing efficiency of the cluster. Existing topology-aware scheduling research mainly focuses on hierarchical awareness allocation based on fat tree or dragonfly topologies. However, most of these methods are based on static topology hierarchy rules (such as priority for racks and switches), lacking mathematical models that can quantify the global structural characteristics of the cluster topology. This makes it difficult to adaptively balance communication locality and resource utilization under dynamically changing cluster loads. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings and defects of the prior art and provide an adaptive job scheduling method and system for HPC clusters. It is an adaptive job scheduling method for HPC clusters that integrates multi-dimensional resource entropy assessment, graph structure awareness and uncertainty thermal prediction. By modeling the cluster topology as a graph structure, introducing equipment temperature constraints and dual-path topology embedding of multi-scale spectral graph convolution and edge type-aware residual graph network, it achieves multi-dimensional collaborative optimization from resource fragmentation assessment, topology embedding matching to uncertainty thermal prediction avoidance.

[0004] According to a first aspect of this application, an adaptive job scheduling method for HPC clusters is provided, comprising:

[0005] Real-time acquisition of multi-dimensional resource usage data and physical temperature time-series data of computing nodes in HPC clusters;

[0006] The comprehensive resource entropy is calculated based on the multidimensional resource usage data; the comprehensive resource entropy is determined by aggregating the single-dimensional resource entropy of the multidimensional resource usage data, and the multidimensional resource usage data includes at least CPU utilization, memory usage, network bandwidth utilization, and I / O throughput.

[0007] A weighted heterogeneous topology graph is constructed, with each computing node as a graph node and the high-speed interconnect links between computing nodes as graph edges; the graph structure entropy is calculated and output based on the weighted heterogeneous topology graph.

[0008] The weighted heterogeneous topology graph is subjected to multi-scale spectral graph convolution and edge type-aware residual GAT dual-path topology embedding processing, with graph structure entropy as the gating parameter, and node-level fusion embedding features are output.

[0009] Based on the physical temperature time series data, an adaptive Kalman filter is used to estimate the thermal risk of each graph node, outputting the node-level thermal risk, and the global thermal risk is determined based on the node-level thermal risk.

[0010] Based on the comprehensive resource entropy and global thermal risk, the scheduling strategy mode is determined;

[0011] Job scheduling is performed based on the aforementioned scheduling strategy model and the comprehensive score of each graph node determined by the cosine similarity between the resource demand vector and the available resource vector of the graph node, the topology matching degree determined by the topology-level or node-level fusion embedding features, and the node-level hot risk.

[0012] Preferably, the comprehensive resource entropy is determined based on the aggregation of single-dimensional resource entropy from multi-dimensional resource usage data, including:

[0013] ; ;

[0014] in, Represents the overall resource entropy. For resource dimensions weights, Represents a set of multidimensional resource dimensions. For resource dimensions fragment entropy; For resource dimensions The node in the following diagram falls into the first... The probability of a resource interval; For resource dimensions The number of resource interval divisions; It is a constant.

[0015] Preferably, the weighted heterogeneous topology graph includes edge weights determined by link bandwidth and link delay, wherein the edge weights are determined by dividing the physical bandwidth by the link delay.

[0016] Preferably, the graph edges include multiple edge types for labeling the hierarchical affiliation of links, and the edge types include at least leaf-spine links, spine-core links, intra-rack links, and cross-rack links.

[0017] Preferably, the calculation and output of graph structure entropy based on weighted heterogeneous topological graphs involves determining and outputting the graph structure entropy based on the eigenvalue spectrum of the normalized Laplacian matrix of the weighted heterogeneous topological graph, including:

[0018] Determine the eigenvalues ​​of the normalized Laplace matrix and the probability distribution of the normalized eigenvalues;

[0019] ; Represents the probability distribution of normalized eigenvalues. Indicates the number of nodes in the graph; , These are the eigenvalues ​​of the normalized Laplace matrix;

[0020] Calculate the graph structure entropy based on the normalized eigenvalue probability distribution:

[0021] ;

[0022] in, Represents graph structure entropy; It is a constant.

[0023] Preferably, the step of performing multi-scale spectral graph convolution and edge type-aware residual GAT dual-path topological embedding processing on the weighted heterogeneous topological graph, using graph structure entropy as a gating parameter, and outputting node-level fused embedding features includes:

[0024] A multi-scale Chebyshev spectral convolutional network is employed, performing spectral domain filtering simultaneously across local, mid-range, and global receptive fields before feature fusion to obtain the first embedding feature of the graph node. The graph node embedding features at different scales before feature fusion are represented as follows:

[0025] ;

[0026] in, Representing scale The nodes in the following figure exist Layer embedding features This represents the activation function. Representing scale The nodes in the following figure exist Layer embedding features express Chebyshev polynomial of order 1 This represents a scaled Laplacian matrix; For scale Next Layer Order of learnable coefficients, Representing scale The corresponding Chebyshev polynomial order;

[0027] We employ edge-type-aware residual GAT, learning independent attention parameters for each edge type, and using residual connections and layer normalization to obtain the second embedding features of the graph nodes.

[0028] ;

[0029] in, Represents graph nodes exist The second embedding feature of the layer, Attention parameters representing edge types, Presentation layer normalization processing, Indicates weight, Represents graph nodes exist Features of the layer Represents graph nodes neighboring nodes exist Features of the layer Represents the set of neighboring nodes;

[0030] Based on the feature fusion weight coefficients determined by graph structure entropy, the first embedded feature and the second embedded feature are fused to output a node-level fused embedded feature; wherein, the feature fusion weight coefficients are determined by the following formula:

[0031] ;

[0032] in, Indicates the feature fusion weight coefficients. Indicates the gating weight, This indicates average pooling. Represents graph nodes The first embedding feature, Represents graph nodes The second embedding feature, This represents the graph structure entropy.

[0033] Preferably, the node-level thermal risk is calculated as follows:

[0034] ;

[0035] in, Represents graph nodes Node-level thermal risks Represents the standard normal cumulative distribution function. This represents the standard deviation of the predicted temperature at each node in the graph. For safe temperature threshold, This indicates the predicted temperature at the graph nodes. Indicates time, As a forward-looking window, This represents the minimum standard deviation.

[0036] Preferably, determining the scheduling strategy mode based on the comprehensive resource entropy and global hot risk includes:

[0037] If satisfied or Execute the active defragmentation mode, performing defragmentation while simultaneously... and The compute nodes trigger pre-migration, pausing backfilling;

[0038] If the conditions for the active cleaning mode are not met, but are met or Implement a conservative reservation mode to avoid The computing nodes;

[0039] If neither the conditions for the proactive cleanup mode nor the conditions for the conservative reservation mode are met, and the conditions are met... and Execute backfill mode;

[0040] in, Represents the overall resource entropy. Indicates global thermal risk. , These represent the lower and upper thresholds of the overall resource entropy. , The lower and upper thresholds represent the global thermal risk. Indicates node-level thermal risk. The standard deviation of the predicted temperature is represented by the nodes in the graph.

[0041] Preferably, the overall score of the graph node is calculated using the following formula:

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] in, This represents the overall score of the graph nodes. The weighted coefficients represent the cosine similarity between the resource demand vector and the available resource vector of the graph nodes. The weighting coefficients represent the degree of topological matching. The weighting coefficients representing node-level thermal risk are summed to 1. This represents the cosine similarity between the resource demand vector and the available resource vector of the graph nodes. Indicates the topology matching degree. This represents the graph structure entropy.

[0047] According to a second aspect of this application, an adaptive job scheduling system for an HPC cluster is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the adaptive job scheduling method for the HPC cluster when executing the computer program.

[0048] The HPC cluster adaptive job scheduling method and system of this application integrates three elements—multidimensional resource entropy assessment, graph structure entropy perception, and thermal risk perception—into a collaborative scheduling system. This adaptive job scheduling method for HPC clusters solves the shortcomings of existing technologies, which rely solely on a single dimension of resource status for scheduling decisions and lack quantitative assessment of the overall health of the cluster's multidimensional resources. It can be applied to scenarios such as large supercomputing centers, cloud computing platforms, and AI training clusters that require efficient scheduling and management of large-scale computing resources. Attached Figure Description

[0049] Figure 1 This is a flowchart of the HPC cluster adaptive job scheduling method of the present invention. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0051] This invention introduces the concept of "entropy" from information theory into HPC cluster resource status assessment. Simultaneously, it performs graph modeling of the cluster topology and extracts graph structure features. It learns topology embedding using a dual-path architecture of multi-scale spectral graph convolution and edge type-aware residual graph network. It models device temperature as a state estimation problem of adaptive Kalman filtering to obtain predicted temperature and prediction uncertainty. Finally, through the joint decision-making of three factors of "resource entropy-graph structure-thermal uncertainty" and the entropy weight adaptive scoring mechanism, it achieves the optimal matching between scheduling strategy and multi-dimensional cluster state.

[0052] See Figure 1 As shown in the exemplary embodiment of this application, the HPC cluster adaptive job scheduling method includes the following steps:

[0053] S1. Real-time acquisition of multi-dimensional resource usage data and physical temperature time-series data of computing nodes in the HPC cluster;

[0054] S2. Calculate the comprehensive resource entropy based on the multidimensional resource usage data; the comprehensive resource entropy is determined by aggregating the single-dimensional resource entropy of the multidimensional resource usage data, and the multidimensional resource usage data includes at least CPU utilization, memory usage, network bandwidth utilization, and I / O throughput;

[0055] S3. Construct a weighted heterogeneous topology graph with each computing node as a graph node and the high-speed interconnect links between computing nodes as graph edges; calculate and output the graph structure entropy based on the weighted heterogeneous topology graph;

[0056] S4. Perform multi-scale spectral graph convolution and edge type-aware residual GAT dual-path topological embedding on the weighted heterogeneous topological graph, using graph structure entropy as the gating parameter, and output node-level fusion embedding features.

[0057] S5. Based on the physical temperature time series data, an adaptive Kalman filter is used to estimate the thermal risk of each graph node, output the node-level thermal risk, and determine the global thermal risk based on the node-level thermal risk;

[0058] S6. Based on the comprehensive resource entropy and global hot risk, determine the scheduling strategy mode;

[0059] S7. Job scheduling is performed based on the aforementioned scheduling strategy mode and the comprehensive score of each graph node determined by the cosine similarity between the resource demand vector and the available resource vector of the graph node, the topology matching degree determined by the topology level or node level fusion embedding features, and the node level hot risk.

[0060] In this application, the comprehensive resource entropy is determined based on the aggregation of single-dimensional resource entropy from multi-dimensional resource usage data. Specifically, single-dimensional resource entropy, such as CPU entropy, memory entropy, network entropy, and I / O entropy, is calculated separately first. Then, multiple single-dimensional resource entropies are aggregated into a comprehensive resource entropy, including:

[0061] ;

[0062] ;

[0063] in, Represents the overall resource entropy. For resource dimensions weights, Represents a set of multidimensional resource dimensions. For resource dimensions fragment entropy; For resource dimensions The node in the following diagram falls into the first... The probability of a resource interval; For resource dimensions The number of resource interval divisions; This is a constant, specifically a small constant to prevent zero logarithm. Taking the four-dimensional resource dimension as an example, the default value for all four dimensions is 0.25, but it can be adjusted according to business priorities. This is a normalization factor that ensures the entropy values ​​of each dimension fall within the [0,1] interval.

[0064] Information entropy naturally distinguishes the differences in fragment distribution structure that proportional measures cannot differentiate. Experimental verification shows that proportional measures remain unchanged at 0.64 when fragments evolve from global dispersion to local aggregation, while the proposed method, using comprehensive resource entropy, can reduce it from 0.81 to 0.37, accurately tracking the real improvement process of cluster scheduling space.

[0065] In this application, the weighted heterogeneous topology graph includes edge weights determined by link bandwidth and link delay, wherein the edge weights are determined by dividing the physical bandwidth by the link delay. Furthermore, the graph edges include various edge types used to label the hierarchical affiliation of links, and the edge types at least include leaf-spine links (edge ​​links between leaf switches and spine switches), spine-core links (edge ​​links between spine switches and core switches), intra-rack links, and cross-rack links. For example, the constructed weighted heterogeneous topology graph is G=(v,e,w,t), where v represents the set of graph nodes, e represents the set of edges, w represents the set of edge weights, and t represents the set of edge types. For example, the edge weights between two adjacent graph nodes... , Physical bandwidth, in Gbps. Link delay, in μs, corresponding to the edge type. This indicates the hierarchical affiliation of the link. Among them, leaf-spine is the leaf-spine edge, which represents the internal backbone link within a Pod between leaf switches and spine switches; spine-core is the spine-core edge, which represents the cross-Pod interconnection link between spine switches and core switches.

[0066] In one embodiment, the step of calculating and outputting graph structure entropy based on a weighted heterogeneous topological graph, and determining and outputting graph structure entropy based on the eigenvalue spectrum of the normalized Laplacian matrix of the weighted heterogeneous topological graph, includes:

[0067] Determine the eigenvalues ​​of the normalized Laplace matrix and the probability distribution of the normalized eigenvalues;

[0068] ; Represents the probability distribution of normalized eigenvalues. Indicates the number of nodes in the graph; The first normalized Laplace matrix One eigenvalue; The first normalized Laplace matrix One eigenvalue;

[0069] Calculate the graph structure entropy based on the normalized eigenvalue probability distribution:

[0070] ;

[0071] in, Represents graph structure entropy; It is a constant. .

[0072] Among them, the normalized Laplace matrix , This represents the identity matrix of the corresponding dimension, with dimension 1. , Equal to the total number of nodes, This represents the graph adjacency weight matrix, which is a matrix composed of edge weights obtained by dividing the physical bandwidth by the link delay; This represents a diagonal matrix, also known as a degree matrix, whose elements... .right Eigenvalue decomposition yields the eigenvalue spectrum { },in, , It is an eigenvalue in the eigenvalue spectrum.

[0073] This application explicitly models the HPC cluster topology as a heterogeneous topology graph labeled with four link types (leaf-spine, spine-core, intra-rack, cross-rack). The graph structure entropy is defined by the eigenvalue spectrum of the normalized Laplacian matrix, providing the scheduler with an independent and interpretable quantitative indicator of topology quality.

[0074] In one embodiment, the multi-scale spectral graph convolution and edge-type-aware residual GAT dual-path topological embedding processing of the weighted heterogeneous topological graph, using graph structure entropy as a gating parameter, outputs node-level fused embedding features, including:

[0075] A multi-scale Chebyshev spectral graph convolutional network is employed. Spectral domain filtering is performed simultaneously across three receptive fields: local (within the switch domain), mid-range (rack-level), and global (across racks), followed by feature fusion to obtain the first embedding feature of the graph node. The graph node embedding features at different scales before feature fusion are represented as follows:

[0076] ;

[0077] in, Representing scale The nodes in the following figure exist Layer embedding features This represents the activation function. Representing scale The nodes in the following figure exist Layer embedding features express Chebyshev polynomial of order 1 This represents a scaled Laplacian matrix. ; For scale Next Layer Order of learnable coefficients, Representing scale The corresponding Chebyshev polynomial order, such as , , k≥2, specifically, for example This corresponds to three scales: local, medium-range, and global. =2 (for 1-2 hop neighbors, used to capture local structure within the switch domain). =4 (for 1-4 hop neighbors, used to capture rack-level topology features). =6 (for 1-6 hop neighbors, used to capture cross-rack global connectivity patterns).

[0078] Based on the graph node embedding feature representations at different scales before feature fusion, the first embedding feature of the graph node is obtained by fusing with learnable scale attention weight coefficients, as shown in the following example:

[0079] ;

[0080] in, This represents the first embedding feature of a graph node. This represents the graph node embedding feature representations at different scales before feature fusion. The attention weight coefficient represents the learnable scale.

[0081] in, ;

[0082] In the formula, Represents the learnable attention vector, superscript Indicates transpose. This represents the learnable projection matrix. This indicates average pooling. Represents the normalization function. This represents the hyperbolic tangent function.

[0083] By using multi-scale convolution processing, the system utilizes local scale to perceive the adjacent connection patterns within the switch domain (key to leaf layer matching in fat tree topology), mid-scale to perceive routing characteristics across switches within the rack, and global scale to perceive remote communication bottlenecks across racks. The complementary information from these three scales avoids information loss in receptive field selection in single-order Chebyshev.

[0084] We employ an edge-type-aware residual GAT (Graph Attention Network) to learn independent attention parameters based on each edge type. Residual connections and layer normalization are then used to obtain the second embedding features of the graph nodes.

[0085] ;

[0086] in, Represents graph nodes exist The second embedding feature of the layer, Attention parameters representing edge types, Presentation layer normalization processing, Indicates weight, Represents graph nodes exist Features of the layer Represents graph nodes neighboring nodes exist Features of the layer The set of neighboring nodes is represented; the attention parameter for the edge type is determined by the following formula:

[0087] ;

[0088] in, This is a type-specific attention vector, corresponding to 4 sets of independent parameters. edge type Learnable embedding vectors, Embed the projection matrix for the edge type. Indicates the transpose operation; , Representing edge type The embedding feature representation of the corresponding adjacent graph nodes; This represents a linear activation function with leakage correction.

[0089] The attention mechanism distinguishes link types. For example, the four edge types of leaf-spine, spine-core, intra-rack, and cross-rack learn independent attention parameters, enabling the model to distinguish the essential communication differences between high-bandwidth, low-latency links within the same rack and congestion-prone links across racks.

[0090] Based on the feature fusion weight coefficients determined by graph structure entropy, the first embedded feature and the second embedded feature are fused to output a node-level fused embedded feature; wherein, the feature fusion weight coefficients are determined by the following formula:

[0091] ;

[0092] in, Indicates the feature fusion weight coefficients. Indicates the gating weight, This indicates average pooling, and [·;·;·] indicates vector concatenation. Represents the graph nodes obtained by multi-scale spectral graph convolution processing. The first embedding feature, Represents the graph nodes obtained by edge type-aware residual GAT. The second embedding feature, This represents the graph structure entropy.

[0093] The optimal feature fusion weight coefficients are automatically learned by the gating matrix, rather than relying on manually set threshold parameters, enabling the dual-path architecture to adapt to the topological distribution characteristics of different clusters.

[0094] The final fused features are represented as follows:

[0095] ,

[0096] in, This represents the output node-level fused embedding features. , This represents the feature fusion weight coefficient.

[0097] In one embodiment, the edge type-aware residual GAT processing of this application can be configured to when Activated at time This represents the graph structure entropy threshold.

[0098] In this application, graph structure entropy is used as a gating signal for gated fusion: when When the representation graph structure is topologically regular, spectral features should dominate, and the model will automatically learn to incorporate them through training. Push towards a larger value; when When the representation graph structure is high (and topologically complex), attention features should dominate, and the model should automatically apply them. The threshold is pushed towards a larger value. Compared to fixed sigmoid gating, which relies on only a single threshold parameter β, this application uses learnable gating that can adapt to the distribution differences of different cluster topology characteristics.

[0099] In summary, this application employs a dual-path architecture consisting of K=2, 4, and 6 three-scale parallel Chebyshev spectral graph convolution (global path) combined with edge type-aware residual GAT (local path). The three-scale spectral convolution simultaneously captures topological features from three receptive fields: local switch domain, mid-range rack level, and global cross-rack level. Through learnable-scale attention fusion, it avoids the receptive field information loss of single-order Chebyshev, achieving information richness far exceeding that of single-order Chebyshev or single-layer GAT. The edge type-aware GAT learns independent attention parameters for four link types, distinguishing between high-bandwidth links within the same rack and congested links across racks. The two paths are fused using a learnable gating matrix. The gating automatically learns the optimal fusion weights for different topological states from the data, exhibiting cluster adaptability through learnable gating.

[0100] In one embodiment, the node-level thermal risk is calculated as follows:

[0101] ;

[0102] in, Represents graph nodes Node-level thermal risks Represents the standard normal cumulative distribution function. This represents the standard deviation of the predicted temperature at each node in the graph. The default safe temperature threshold is 55℃. This indicates the predicted temperature at the graph nodes. Indicates time, As a forward-looking window, The minimum standard deviation is 0.5℃ by default.

[0103] The global thermal risk index is counted as follows:

[0104] ;

[0105] In the formula: This is the global thermal risk index; The total number of nodes in the cluster, which can be greater than, less than or equal to. Preferred ; For the computation of graph nodes The thermal risk index. The physical temperature time-series data includes at least one of the following: CPU core temperature, GPU temperature, and node inlet temperature.

[0106] Specifically, in practice, a two-dimensional adaptive Kalman filter is maintained for each graph node, and temperature is modeled as a state estimation problem with an unknown rate of change:

[0107] Constructing the state vector: ;

[0108] in The current temperature of the graph node. The temperature change rate of the nodes in the graph. This represents the current state vector of a graph node, i.e., the temperature state vector.

[0109] Based on the state transition matrix, construct a state transition model (constant rate assumption):

[0110] Estimated state vector: ;

[0111] Covariance matrix: ;

[0112] in Here is the state transition matrix. The sampling interval is... It is a 2×2 state covariance matrix. This is the noise covariance matrix for the adaptive process.

[0113] Perform measurement update (receive actual temperature) back):

[0114] ;

[0115] ;

[0116] ;

[0117] Where the observation matrix , To measure the noise covariance, This represents the Kalman gain matrix.

[0118] Adaptation process noise covariance matrix Adjust dynamically based on the recent drastic temperature changes:

[0119] ;

[0120] in For sliding windows Temperature variance within, For adaptive coefficients, Indicates the baseline. Indicates the temperature of the graph nodes. This represents a 2-order identity matrix.

[0121] When the compute node temperature fluctuates drastically (e.g., during intensive job startup), adaptively increase... This makes the filter more trusting of new observations and faster at tracking changes; when the temperature is stable. Back to baseline Smooth noise reduction.

[0122] exist Always Make predictions ( (Forward window, take 1 / 3 of the average job execution time) ,in Represents continuous application state transitions step.

[0123] Node-based temperature prediction: ;

[0124] Standard deviation of predicted temperature at graph nodes (uncertainty): .

[0125] This application employs an adaptive Kalman filter to separate state estimation from noise in temperature observations, outputting prediction uncertainty. This prevents the scheduler from being misled when temperature sensor noise is high and enables a faster response when temperature changes drastically. Furthermore, this application introduces prediction uncertainty quantification into HPC scheduling. Specifically, the thermal risk index automatically lowers the risk value when uncertainty is high, preventing overly conservative scheduling caused by prediction noise, such as mistakenly marking nodes with stable temperatures but high sensor noise as high-risk.

[0126] This application models the temperature of graph nodes as a two-dimensional Kalman state estimation problem, simultaneously estimating temperature and the rate of temperature change, and outputting the predicted temperature and prediction covariance (uncertainty). Process noise is adaptively adjusted based on the severity of temperature fluctuations: new observations are trusted during severe fluctuations, while noise is smoothed during stable periods. The thermal risk index integrates the predicted temperature and prediction uncertainty, calculating the probability of the predicted temperature exceeding the limit under a given uncertainty using a normal CDF: the risk is determined when the predicted temperature is high and the uncertainty is low; when the predicted temperature is high but the uncertainty is high, the risk is automatically suppressed (to prevent over-conservatism). Since the Kalman filter is an independent and complete prediction-estimation-decision subsystem, it possesses adaptive noise adjustment and uncertainty quantification capabilities.

[0127] This application describes a scheduling strategy mode determined based on the comprehensive resource entropy and global hot risk, prioritizing the following steps: First, the conditions for the proactive consolidation mode are assessed; if these are not met, the conditions for the conservative reservation mode are assessed; if neither is met, the backfilling mode is executed. Specifically, this includes:

[0128] If satisfied or Execute the active defragmentation mode, performing defragmentation while simultaneously... and The compute nodes trigger pre-migration, pausing backfilling;

[0129] If the conditions for the active cleaning mode are not met, but are met or Implement a conservative reservation mode to avoid Computation nodes, such as reserving contiguous resource blocks for large jobs;

[0130] If neither the conditions for the proactive cleanup mode nor the conditions for the conservative reservation mode are met, and the conditions are met... and Execute backfill mode, such as executing an aggressive backfill strategy, prioritizing the execution of aggressive backfill strategies in low-temperature computing node regions;

[0131] Specifically, in backfill mode, the search space includes all free computing nodes and allows for fragmented utilization; in conservative reservation mode, only computing nodes that satisfy the contiguous resource block constraint are searched and excluded. The nodes; in the active defragmentation mode, defragmentation and pre-migration are triggered first, and then matching is performed on the defragmented resources. A hysteresis mechanism is used between the various control switching thresholds to avoid frequent oscillations and switching of scheduling modes.

[0132] In one embodiment of this application, the overall score of the graph nodes is calculated using the following formula:

[0133] ;

[0134] ;

[0135] ;

[0136] ;

[0137] in, This represents the overall score of the graph nodes. , , These represent their respective weighting coefficients, and the sum of the three is 1. This represents the cosine similarity between the resource demand vector and the available resource vector of the graph nodes. Indicates the topology matching degree. This indicates node-level thermal risk. Among them, Determined based on topological hierarchy or based on The cosine similarity calculation enables topological embeddings to truly participate in the final matching decision.

[0138] Specifically, such as At that time, the topological hierarchy is used to determine the value, such as 1.0 for the same domain, 0.8 for the neighboring domain, and 0.3 for the distant domain. At that time, based on The cosine similarity is determined.

[0139] In this application, after determining the scheduling strategy mode based on the comprehensive resource entropy and global thermal risk, the calculation of the comprehensive score of the graph nodes determines which nodes to select under the scheduling strategy mode. The two are connected through the strategy mode variable, and finally form the optimal matching node set control executed by the scheduler.

[0140] Specifically, after obtaining the comprehensive score of the graph nodes, the comprehensive scores are sorted in descending order. Starting from the graph node with the highest score, the job resource requirements are mapped to the available resources of the nodes in sequence until all resource requirements of the job (such as CPU utilization, memory usage, network bandwidth utilization, I / O throughput, etc.) are met, forming a set of graph nodes that meet the requirements, which constitutes the optimal matching node set S*. Based on this, the scheduled job is sent to the computing nodes corresponding to the graph nodes in the optimal matching node set S* for processing. At the same time, the measured temperature of the computing nodes corresponding to the graph nodes in the optimal matching node set S* is fed back to the Kalman filter as the observation input for the next round of temperature status update, forming a closed-loop mechanism of evaluation-decision-execution-feedback.

[0141] This application calculates a comprehensive score, which automatically focuses the scoring on the most pressing optimization goals of the cluster at present, namely, resource matching when resource fragmentation is severe, communication affinity when topology is complex, and temperature safety when thermal risk is high, thereby improving the adaptability and adaptability of the method.

[0142] According to a second aspect of this application, an adaptive job scheduling system for an HPC cluster is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the adaptive job scheduling method for the HPC cluster.

[0143] Application scenario example:

[0144] A supercomputing center cluster comprises 200 compute nodes, each configured with a 64-core CPU, 256GB of memory, and a 100Gbps InfiniBand network (fat tree topology). The cluster simultaneously runs compute-intensive jobs (such as WRF weather simulation) and memory-intensive jobs (such as genome assembly). Traditional solutions only backfill based on CPU core count, ignoring the accumulation of memory fragmentation and the overhead of topology communication. After one week of operation, approximately 15% of nodes are unable to accept new jobs due to insufficient memory, and inter-process communication (MPI, Message Passing Interface) latency across switch domains extends the execution time of some jobs by approximately 12%. This new solution simultaneously monitors CPU and memory entropy, compute memory entropy, and collects node temperature data to assess compute node thermal risk. When memory entropy exceeds a threshold, defragmentation is automatically triggered; when the thermal risk of nodes in a certain area exceeds a thermal risk threshold, hot migration is initiated. After implementation, the cluster's overall resource utilization improved by approximately 8%, average job latency decreased by approximately 18%, MPI latency decreased by approximately 15%, and the frequency of cluster hotspot events decreased by approximately 60%.

[0145] This invention relates to the joint entropy assessment and adaptive switching of scheduling strategies for multi-dimensional resources such as CPU, network I / O, and storage I / O in HPC clusters. By weighted aggregation of multi-dimensional comprehensive resource entropy and inter-dimensional coupling analysis, it solves the problem that existing height-based methods only focus on the occupancy of a single resource dimension such as the number of CPU cores or memory capacity, ignoring the matching relationship and balance between multi-dimensional resources such as CPU, memory, network bandwidth, and I / O bandwidth in the cluster. When CPU resources are sufficient but memory fragmentation is severe, traditional schedulers cannot perceive this "barrel effect" and may schedule memory-sensitive jobs to unsuitable nodes, resulting in low job execution efficiency or even failure.

[0146] This invention introduces information entropy into resource fragmentation measurement, using the distribution probability of nodes in each resource sub-interval instead of the resource allocation ratio as the measurement basis. It naturally possesses the ability to distinguish the fragmentation distribution structure, solving the problem that existing resource fragmentation measurement methods lack structural perception capabilities. That is, methods based on resource allocation ratios, represented by existing resource fragmentation measurement methods, only measure the degree of fragmentation from the perspective of the total amount of resources, and cannot capture the distribution structure characteristics of resource fragments.

[0147] This invention utilizes an entropy-graph structure-temperature three-factor adaptive strategy decision-making mechanism and automatically learns the importance weights of neighboring nodes through an attention mechanism. This enables real-time embedding learning and scheduling collaborative optimization of HPC cluster topology, solving the problem that existing methods typically use preset fixed thresholds for strategy switching, and scheduling strategy switching relies on static thresholds, making it impossible to dynamically adjust according to the real-time status of the cluster.

[0148] This invention achieves topology-aware node aggregation and allocation by using cluster topology graph modeling, graph structure entropy calculation, and GAT node embedding learning. During topology regularization, it prioritizes the allocation of nodes for communication-intensive jobs within the same switch domain. This solves the problems of existing scheduling methods lacking cluster topology awareness and ignoring the positional relationship of selected nodes in the physical topology during node allocation. These problems may lead to MPI communication within a job spanning multiple network hops, causing communication delays and significantly increasing bandwidth contention with interconnect links.

[0149] With the continuous increase in single-node computing density (multi-core CPUs, multiple GPUs, high-bandwidth memory), node heat dissipation has become one of the bottlenecks in supercomputing center operations. When a large number of jobs are concentrated and scheduled to adjacent nodes, local hotspots may form, increasing cooling power consumption and potentially triggering node temperature protection throttling, leading to extended job execution times or even node downtime. However, existing HPC scheduling methods generally do not incorporate real-time node physical temperature into scheduling decisions; they treat the cluster as a temperature-uniform isothermal system, ignoring the substantial impact of spatiotemporal temperature non-uniformity on scheduling quality. This invention transforms node temperature into a dynamic scheduling constraint through a temperature penalty factor, suppressing high-temperature nodes in the matching score and guiding the scheduler to prioritize low-temperature nodes, achieving heat-aware load balancing. This solves the problems of existing scheduling methods lacking device temperature constraints, failing to incorporate real-time node physical temperature into scheduling decisions, potentially leading to local hotspots, increased cooling power consumption, and temperature protection throttling.

[0150] This invention addresses the problems of existing methods lacking joint entropy assessment and coupling analysis of multi-dimensional resources, failing to establish a joint entropy assessment model for multi-dimensional resources, and being unable to capture the coupling relationships between various dimensions by weighted aggregation of four-dimensional entropy of CPU, memory, network I / O, and storage I / O and cross-dimensional difference analysis.

[0151] The present invention provides an adaptive job scheduling method for HPC clusters that integrates multi-dimensional resource entropy assessment, graph structure awareness, and thermal awareness. This method quantifies the topology connectivity quality by modeling the cluster topology as a graph structure and calculating the graph structure entropy. It introduces GAT to learn the topology embedding vector of nodes and constructs a temperature penalty factor in combination with the real-time temperature of the equipment. Together with multi-dimensional resource entropy, it forms a three-factor decision mechanism, realizing the upgrade from single-dimensional resource-driven scheduling to a three-dimensional collaborative scheduling paradigm of resource-topology-temperature, thereby improving the cluster resource utilization, communication efficiency, and heat dissipation balance.

[0152] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0153] Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of the equivalents of the claims be included within the invention.

[0154] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An adaptive job scheduling method for HPC clusters, characterized in that, Including the following steps: Real-time acquisition of multi-dimensional resource usage data and physical temperature time-series data of computing nodes in HPC clusters; The comprehensive resource entropy is calculated based on the multidimensional resource usage data; the comprehensive resource entropy is determined by aggregating the single-dimensional resource entropy of the multidimensional resource usage data, and the multidimensional resource usage data includes at least CPU utilization, memory usage, network bandwidth utilization, and I / O throughput. A weighted heterogeneous topology graph is constructed, with each computing node as a graph node and the high-speed interconnect links between computing nodes as graph edges. Calculate and output graph structure entropy based on weighted heterogeneous topological graphs; The weighted heterogeneous topology graph is subjected to multi-scale spectral graph convolution and edge type-aware residual GAT dual-path topology embedding processing, with graph structure entropy as the gating parameter, and node-level fusion embedding features are output. Based on the physical temperature time series data, an adaptive Kalman filter is used to estimate the thermal risk of each graph node, outputting the node-level thermal risk, and the global thermal risk is determined based on the node-level thermal risk. Based on the comprehensive resource entropy and global thermal risk, the scheduling strategy mode is determined; Job scheduling is performed based on the aforementioned scheduling strategy model and the comprehensive score of each graph node determined by the cosine similarity between the resource demand vector and the available resource vector of the graph node, the topology matching degree determined by the topology-level or node-level fusion embedding features, and the node-level hot risk.

2. The HPC cluster adaptive job scheduling method according to claim 1, characterized in that, The comprehensive resource entropy is determined based on the aggregation of single-dimensional resource entropy from multi-dimensional resource usage data, including: ； ； in, Represents the overall resource entropy. For resource dimensions weights, Represents a set of multidimensional resource dimensions. For resource dimensions fragment entropy; For resource dimensions The node in the following diagram falls into the first... The probability of a resource interval; For resource dimensions The number of resource interval divisions; It is a constant.

3. The HPC cluster adaptive job scheduling method according to claim 1, characterized in that, The weighted heterogeneous topology graph includes edge weights determined by link bandwidth and link delay, wherein the edge weights are determined by dividing the physical bandwidth by the link delay.

4. The HPC cluster adaptive job scheduling method according to claim 1, characterized in that, The graph edges include various edge types used to label the hierarchical affiliation of links. The edge types include at least leaf-spine links, spine-core links, intra-rack links, and cross-rack links.

5. The HPC cluster adaptive job scheduling method according to claim 1, characterized in that, The calculation and output of graph structure entropy based on weighted heterogeneous topological graphs involves determining and outputting the graph structure entropy based on the eigenvalue spectrum of the normalized Laplacian matrix of the weighted heterogeneous topological graph, including: Determine the eigenvalues ​​of the normalized Laplace matrix and the probability distribution of the normalized eigenvalues; ; Represents the probability distribution of normalized eigenvalues. Indicates the number of nodes in the graph; , These are the eigenvalues ​​of the normalized Laplace matrix; Calculate the graph structure entropy based on the normalized eigenvalue probability distribution: ； in, Represents graph structure entropy; It is a constant.

6. The HPC cluster adaptive job scheduling method according to claim 1, characterized in that, The weighted heterogeneous topological graph is subjected to multi-scale spectral graph convolution and edge-type-aware residual GAT dual-path topological embedding processing, using graph structure entropy as the gating parameter, to output node-level fused embedding features, including: A multi-scale Chebyshev spectral convolutional network is employed, performing spectral domain filtering simultaneously across local, mid-range, and global receptive fields before feature fusion to obtain the first embedding feature of the graph node. The graph node embedding features at different scales before feature fusion are as follows: ； in, Representing scale The nodes in the following figure exist Layer embedding features This represents the activation function. Representing scale The nodes in the following figure exist Layer embedding features, express Chebyshev polynomial of order 1 This represents a scaled Laplacian matrix; For scale Next Layer Order of learnable coefficients, Representing scale The corresponding Chebyshev polynomial order; We employ edge-type-aware residual GAT, learning independent attention parameters based on each edge type. Residual connections and layer normalization are then used to obtain the second embedding features of the graph nodes. ； in, Represents graph nodes exist The second embedding feature of the layer, Attention parameters representing edge types, Presentation layer normalization processing, Indicates weight, Represents graph nodes exist Features of the layer Represents graph nodes neighboring nodes exist Features of the layer This represents the number of neighboring nodes in the graph; Based on the feature fusion weight coefficients determined by graph structure entropy, the first embedded feature and the second embedded feature are fused to output a node-level fused embedded feature; wherein, the feature fusion weight coefficients are determined by the following formula: ； in, Indicates the feature fusion weight coefficients. Indicates the gating weight, This indicates average pooling. Represents graph nodes The first embedding feature, Represents graph nodes The second embedding feature, This represents the graph structure entropy.

7. The HPC cluster adaptive job scheduling method according to claim 1, characterized in that, The node-level thermal risk is calculated as follows: ； in, Represents graph nodes Node-level thermal risks Represents the standard normal cumulative distribution function. This represents the standard deviation of the predicted temperature at each node in the graph. For safe temperature threshold, This indicates the predicted temperature at the graph nodes. Indicates time, As a forward-looking window, This represents the minimum standard deviation.

8. The HPC cluster adaptive job scheduling method according to claim 1, characterized in that, The process of determining the scheduling strategy mode based on the comprehensive resource entropy and global hot risk includes: If satisfied or Execute the active defragmentation mode, performing defragmentation while simultaneously... and The compute nodes trigger pre-migration, pausing backfilling; If the conditions for the active cleaning mode are not met, but are met or Implement a conservative reservation mode to avoid The computing nodes; If neither the conditions for the proactive cleanup mode nor the conditions for the conservative reservation mode are met, and the conditions are met... and Execute backfill mode; in, Represents the overall resource entropy. Indicates global thermal risk. , These represent the lower and upper thresholds of the overall resource entropy. , The lower and upper thresholds represent the global thermal risk. Indicates node-level thermal risk. The standard deviation of the predicted temperature is represented by the nodes in the graph.

9. The HPC cluster adaptive job scheduling method according to claim 8, characterized in that, The overall score of the graph nodes is calculated using the following formula: ； ； ； ； in, This represents the overall score of the graph nodes; The weighted coefficients represent the cosine similarity between the resource demand vector and the available resource vector of the graph nodes. The weighting coefficients represent the degree of topological matching. The weighting coefficients representing node-level thermal risk are summed to 1. This represents the cosine similarity between the resource demand vector and the available resource vector of the graph nodes. Indicates the topology matching degree. This represents the graph structure entropy.

10. An HPC cluster adaptive job scheduling system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the HPC cluster adaptive job scheduling method as described in any one of claims 1 to 9.

Images