Server GPU resource and heat dissipation collaborative scheduling method and device, and medium

Abstract

The application provides a server GPU resource and heat dissipation cooperative scheduling method and device and medium, the method maps the candidate GPU resource action to the predicted heat generation sequence of each heat area, and maps the candidate heat dissipation action to the predicted heat dissipation effect sequence of the same heat area in the same predicted time domain to combine a candidate action pair, and predicts the heat area temperature track; the target action pair is screened according to the temperature safety margin, the cooling capacity margin and the joint evaluation index; the execution sequence and the time offset of the resource scheduling command and the heat dissipation control command are determined according to the GPU heat generation response time and the heat dissipation response time; and the online correction of the coupling parameter is carried out by using the temperature prediction residual error. The application can inhibit local hot spots, reduce invalid migration and excessive heat dissipation energy consumption.

Description

Technical Field

[0001] This invention relates to the field of server resource scheduling and server thermal management technology, and in particular to a method, apparatus and medium for coordinated scheduling of server GPU resources and heat dissipation. Background Technology

[0002] As the scale of AI training, online inference, and high-performance computing jobs continues to increase, the integration density and power consumption of GPUs in servers are constantly rising. Within the same server or rack, there are significant differences in the computational stages, memory access patterns, and communication loads of different GPU tasks, resulting in strong temporal fluctuations and spatial imbalances in GPU chip temperature, local hotspot temperatures, and the thermal load on cooling branches or airflow channels.

[0003] In this context, existing servers typically employ two independent control methods: One type is GPU resource scheduling, which typically involves placing tasks and allocating resources based on GPU utilization, video memory capacity, task priority, and GPU interconnect topology. Another type is server heat dissipation control, which typically adjusts fan speed, water pump speed, valve opening, cooling medium flow rate, or cold source output independently based on the current temperature.

[0004] The two types of control are executed by different modules, and their decision-making bases are independent: the resource scheduling side does not take into account the future thermal load generated by placing, migrating, or increasing resources for candidate tasks, and the thermal control side does not take into account the load variation patterns of GPU tasks. Furthermore, airflow, cooling medium flow, cold plate heat exchange, and the thermal actuators themselves all have response delays. When the GPU load increases rapidly, the time it takes for the cooling effect to reach the hot zone may be later than the time when the chip temperature rises. Summary of the Invention

[0005] This invention provides a method, apparatus, and medium for coordinated scheduling of server GPU resources and heat dissipation. It addresses the problems in the prior art where GPU computing resource scheduling and server heat dissipation control are decided independently and the response timing is mismatched. This leads to the inability to detect future local hotspots, delayed heat dissipation response causing transient overheating, and scheduling oscillations between excessive cooling energy consumption and frequent task migration overhead.

[0006] The core technology of this invention is to map the predicted heat generation sequence of candidate GPU resource actions and the predicted heat dissipation effect sequence of candidate heat dissipation actions including heat dissipation response delay to the same hot zone and the same prediction time domain to form candidate action pairs. The temperature trajectory is inferred using a dynamic thermal model, and hard constraint screening and multi-objective joint optimization are performed based on temperature safety margin and cooling capacity margin. Then, the timing offset is determined according to the heat generation response time and heat dissipation response time to implement peak shift control, and the model coupling parameters are corrected online through measured temperature residuals.

[0007] In a first aspect, the present invention provides a method for coordinated scheduling of server GPU resources and heat dissipation, the method comprising the following steps: An application to a server system comprising multiple GPU units, temperature sensors, and at least one controllable thermal execution unit, executed by a scheduling controller communicatively connected to the GPU units and the thermal execution unit, comprising: The system acquires the GPU resource status, GPU task status, measured temperature of the corresponding hot zone, operating status of the heat dissipation execution unit, and environmental status of each GPU unit, and establishes the topological relationship between the GPU unit, hot zone, temperature sensor, and heat dissipation execution unit. For candidate GPU resource actions, determine the predicted heat generation sequence of the candidate GPU resource actions mapped to each hot zone in the prediction time domain; for candidate heat dissipation actions, determine the predicted heat dissipation effect sequence of the candidate heat dissipation actions mapped to the same hot zone in the same prediction time domain, and the predicted heat dissipation effect sequence includes the heat dissipation response delay of the corresponding heat dissipation execution unit. Candidate GPU resource actions and candidate heat dissipation actions are combined into candidate action pairs. Based on the current measured temperature of each hot zone, the predicted heat generation sequence, and the predicted heat dissipation effect sequence, the predicted temperature trajectory of the hot zone corresponding to each candidate action pair is calculated. The temperature safety margin is determined based on the predicted temperature trajectory of the hot zone and the upper limit of the hot zone temperature, and the cooling capacity margin is determined based on the available heat dissipation capacity of the heat dissipation execution unit and the predicted heat load corresponding to the predicted heat generation sequence. Candidate action pairs that do not meet GPU resource constraints, GPU topology constraints, hot zone temperature constraints, or cooling capacity constraints are excluded. The target action pair is determined from the remaining candidate action pairs based on a joint evaluation index that includes at least GPU task performance cost, GPU task migration cost, and heat dissipation energy consumption cost. Based on the GPU heat response time and heat dissipation response time corresponding to the target action, determine the execution order and time offset of the GPU resource scheduling command and the heat dissipation control command, and send the GPU resource scheduling command and the heat dissipation control command to the GPU resource scheduler and the heat dissipation execution unit respectively. Obtain the measured temperature and heat dissipation feedback after the target action is executed. Based on the temperature prediction residual between the measured temperature and the corresponding predicted temperature, determine whether to update the coupling parameters used to generate the predicted heat generation sequence, the predicted heat dissipation effect sequence, or the predicted temperature trajectory of the hot zone, and trigger the next scheduling cycle. The coupling parameters include at least one of the following: thermal effect matrix, cooling performance matrix, model bias, GPU heat dissipation response time, heat dissipation response time, and temperature prediction error covariance matrix.

[0008] Furthermore, the GPU resource status includes at least one of the following: compute unit utilization, tensor compute unit utilization, memory usage, memory bandwidth, GPU interconnect bandwidth, operating frequency, power, and power limit; the GPU task status includes at least one of the following: task type, running stage, resource requirements, priority, service level requirements, completion deadline, migration status, checkpoint data volume, and parallel communication relationship; the operating status of the thermal execution unit includes at least one of the following: fan speed, airflow, damper opening, inlet air temperature, outlet air temperature, pump speed, branch flow rate, valve opening, supply liquid temperature, return liquid temperature, pressure, and fault flag; hot zones are divided according to GPU units, GPU boards, air duct coverage, cold plate coverage, or cooling branch coverage, and the topology relationship includes the first relationship from GPU units to hot zones and the second relationship from thermal execution units to hot zones. One thermal execution unit can affect one or more hot zones, and a hot zone can be affected by multiple thermal execution units simultaneously; the GPU resource status, GPU task status, measured temperature, and the operating status of the thermal execution unit are aligned according to a unified timestamp.

[0009] Furthermore, the process of determining the predicted heat generation sequence includes: based on the historical load sequence of GPU tasks, the current running stage, batch size, remaining task volume, and service level requirements, using a resource demand prediction model to predict at least one of the following GPU resource requirements in the prediction time domain: computing unit share, memory capacity, memory bandwidth, and GPU interconnect bandwidth, and outputting the corresponding prediction uncertainty; mapping the GPU resource requirements corresponding to candidate GPU resource actions to the predicted load of each GPU unit, and determining the predicted power of each GPU unit based on GPU idle power, computing unit load, memory bandwidth load, input / output load, and GPU operating frequency or power limit; and converting the predicted power into the predicted heat generation sequence of each hot zone based on the topological relationship between the GPU unit and the hot zone.

[0010] Furthermore, a dynamic thermal model incorporating thermal inertia, inter-GPU heat propagation, the interaction between heat dissipation execution units, heat dissipation response delay, and the influence of inlet temperature is used to calculate the predicted temperature trajectory of the hot zone. The dynamic thermal model uses the thermal state vector composed of the chip temperature state nodes and hot spot temperature state nodes of the GPU units in each hot zone as the state variable, the predicted power of each GPU unit as the heat input, and the heat dissipation control quantity considering the heat dissipation response delay and the inlet air temperature or cooling medium inlet temperature considering the inlet temperature response delay as the heat dissipation input. The model outputs the predicted chip temperature and hot spot temperature values ​​at each prediction time. The heat dissipation response delay and the inlet temperature response delay are determined based on the heat dissipation response time, the inlet temperature response time, and the sampling and control cycle, respectively.

[0011] Furthermore, for each hot zone, the difference between the upper limit of chip temperature and the upper limit of chip temperature prediction risk for each GPU unit within that hot zone, as well as the difference between the upper limit of hot spot temperature and the upper limit of hot spot temperature prediction risk, are calculated. The minimum value among these differences is taken as the temperature safety margin for that hot zone. The upper limits of chip temperature prediction risk and hot spot temperature prediction risk are obtained by summing the corresponding temperature prediction value and the prediction error compensation amount determined based on the temperature prediction error, respectively. The cooling capacity margin is the difference between the available cooling capacity of the hot zone corresponding to the candidate cooling action and the predicted heat load and reserved cooling capacity of that hot zone. For liquid-cooled hot zones, the available cooling capacity is determined based on the heat exchange efficiency of the cooling branch, the mass flow rate of the cooling medium, the specific heat capacity of the cooling medium at constant pressure, and the maximum allowable difference between the return temperature and the supply temperature. For air-cooled hot zones, the available cooling capacity is determined based on the air mass flow rate, the specific heat capacity of the air at constant pressure, and the allowable difference between the inlet and outlet air temperatures. When the temperature safety margin corresponding to a candidate action pair is less than the preset lower limit of temperature margin, or the cooling capacity margin is less than the preset lower limit of cooling capacity, the candidate action pair is excluded.

[0012] Furthermore, candidate GPU resource actions include at least one of the following: GPU task placement, GPU task migration, GPU partition adjustment, computing resource share adjustment, GPU operating frequency adjustment, and GPU power limit adjustment; candidate cooling actions include at least one of the following: fan speed adjustment, damper opening adjustment, water pump speed adjustment, cooling branch flow rate adjustment, cooling branch valve opening adjustment, supply air temperature adjustment, and cooling medium inlet temperature adjustment; GPU resource constraints and GPU topology constraints include the following: computing resource capacity constraints, memory capacity constraints, memory bandwidth constraints, GPU interconnect bandwidth constraints, CPU-GPU affinity constraints, GPU task parallel group integrity constraints, and migration interruptible time constraints. At least one; the GPU task migration cost is determined based on at least two of the following: GPU task priority, amount of state data to be migrated, available transmission bandwidth between the source GPU unit and the target GPU unit, task interruption time, cache or model warm-up time, communication relationship reconstruction time, and state replication energy consumption; the joint evaluation index also includes at least one of the following: GPU computing energy consumption, service level default risk, hot zone temperature exceedance risk, and control quantity switching cost, and the performance cost weights corresponding to different GPU tasks are determined based on GPU task priority or service level requirements; hot zone temperature constraints and heat dissipation execution unit operating boundaries are considered hard constraints, and their priority is higher than the energy consumption cost and GPU task migration cost in the joint evaluation index.

[0013] Furthermore, within the rolling prediction time domain, target action pairs are determined, and only the control quantities corresponding to the target action pairs in the current scheduling time slot are executed. The lead time of the cooling control command relative to the GPU resource scheduling command is determined based on the difference between the heat dissipation response time and the GPU heat dissipation response time, ensuring that the cooling effect reaches the corresponding hot zone no later than the predicted temperature rise caused by the GPU resource scheduling command. Upon the expiration of the lead time or after receiving feedback that the cooling execution unit has reached the minimum target output, GPU task startup, GPU task migration, or GPU resource share increase are executed. When the measured temperature or temperature rise rate reaches a critical threshold, the cooling capacity margin is continuously lower than the allowable value, the cooling execution unit malfunctions, the temperature sensor confidence level is lower than the threshold, the temperature prediction residual continuously exceeds the residual threshold, or there are no feasible candidate actions... When appropriate, select one or more safety measures according to the risk level, such as increasing available heat dissipation output, limiting GPU power, reducing GPU operating frequency, reducing the resource share of low-priority GPU tasks, migrating or pausing GPU tasks, and isolating GPU units. After the recovery conditions are continuously met for a preset period, exit the safety measures according to the hysteresis control method. Among them, isolating GPU units includes at least one of software logic isolation and hardware protection isolation. Software logic isolation includes removing risky GPU units from the schedulable resource set, stopping the dispatch of new tasks and migrating them, pausing or saving the existing task state. Hardware protection isolation includes disconnecting the link or power supply of the GPU unit by the baseboard management controller, PCIe hot-swap controller or power supply control circuit when the temperature or fault reaches the hardware protection threshold.

[0014] Secondly, the present invention provides a server GPU resource and heat dissipation coordinated scheduling device, comprising: The data acquisition and topology mapping module is used to acquire the GPU resource status, GPU task status, measured temperature of the corresponding hot zone, operating status of the heat dissipation execution unit, and environmental status of each GPU unit, and to establish the topological relationship between the GPU unit, hot zone, temperature sensor and heat dissipation execution unit. The load and heat prediction module is used to determine the predicted heat sequence of candidate GPU resource actions mapped to each hot zone in the prediction time domain for the candidate GPU resource actions. The thermal state assessment module is used to determine the predicted heat dissipation effect sequence of a candidate heat dissipation action, which is mapped to the same thermal region in the same prediction time domain. The predicted heat dissipation effect sequence includes the heat dissipation response delay of the corresponding heat dissipation execution unit. The candidate action pair generation module is used to combine candidate GPU resource actions and candidate heat dissipation actions into candidate action pairs. Based on the current measured temperature of each hot zone, the predicted heat generation sequence, and the predicted heat dissipation effect sequence, it calculates the predicted temperature trajectory of the hot zone corresponding to each candidate action pair. Based on the predicted temperature trajectory of the hot zone and the upper limit of the hot zone temperature, it determines the temperature safety margin. Based on the available heat dissipation capacity of the heat dissipation execution unit and the predicted heat load corresponding to the predicted heat generation sequence, it determines the cooling capacity margin. The collaborative decision-making module is used to exclude candidate action pairs that do not meet GPU resource constraints, GPU topology constraints, hot zone temperature constraints, or cooling capacity constraints, and to determine the target action pair from the remaining candidate action pairs based on a joint evaluation index that includes at least GPU task performance cost, GPU task migration cost, and heat dissipation energy consumption cost. The timing control module is used to determine the execution order and time offset of the GPU resource scheduling command and the heat dissipation control command based on the GPU heat response time and heat dissipation response time corresponding to the target action, and send the GPU resource scheduling command and the heat dissipation control command to the GPU resource scheduler and the heat dissipation execution unit respectively. The feedback safety module is used to obtain the measured temperature and heat dissipation feedback after the target action is executed. Based on the temperature prediction residual between the measured temperature and the corresponding predicted temperature, it determines whether to update the coupling parameters used to generate the predicted heat generation sequence, the predicted heat dissipation effect sequence, or the predicted temperature trajectory of the hot zone, and triggers the next scheduling cycle. The coupling parameters include at least one of the following: thermal effect matrix, cooling performance matrix, model bias, GPU heat dissipation response time, heat dissipation response time, and temperature prediction error covariance matrix.

[0015] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the above-described server GPU resource and heat dissipation coordinated scheduling method.

[0016] Fourthly, the present invention provides a readable storage medium storing a computer program, which, when executed by a processor, implements the server GPU resource and heat dissipation coordinated scheduling method as described above.

[0017] The main contributions and innovations of this invention are as follows: 1. This invention establishes a many-to-many topology mapping of GPU units, hot zones, sensors, and heat dissipation execution units, and projects discrete GPU computing resource actions and continuous heat dissipation control actions into the same hot zone space and the same prediction time domain. It constructs a dynamic thermal state space equation with joint action pairs as the reference unit, which effectively overcomes the technical defect of traditional solutions that mask severe local hot spots by relying only on the average utilization rate or average temperature of the whole machine. It achieves high-fidelity forward prediction of the future chip junction temperature and local hot spot temperature trajectory.

[0018] 2. Before joint optimization, this invention sets up a hard constraint screening mechanism. On the one hand, it constructs a dynamic temperature safety margin by superimposing the temperature prediction value with the error covariance compensation amount. On the other hand, it constructs a cooling capacity margin based on the physical limit of the hydraulic heat transfer of the cooling branch. This eliminates infeasible combinations that do not meet the requirements of computing resource capacity, network communication topology, chip safety temperature, or branch heat dissipation limit in advance, effectively preventing engineering risks of thermal overload of the cooling circuit or chip overheating damage caused by blindly loading computing power.

[0019] 3. This invention quantifies and extracts the physical time lag difference between the heat dissipation response time of the physical heat dissipation execution unit to generate effective cooling and the chip temperature rise time caused by GPU load changes, and automatically generates a look-ahead control lead amount; it prioritizes issuing heat dissipation control commands before predicting an increase in heat generation, and executes task loading only after ensuring that the effective heat dissipation effect reaches the target hot zone before or simultaneously with the temperature rise, thus completely eliminating the short-term over-temperature problem caused by the physical response lag of the heat dissipation system from the underlying control timing.

[0020] 4. This invention comprehensively incorporates IT computing energy consumption, heat dissipation auxiliary energy consumption, task service level requirement (SLO) default loss, temperature over-limit risk, frequent control switching cost, and high-order migration cost including model loading preheating and communication reconstruction into the objective functional. It balances and optimizes the increase of heat dissipation output, cross-regional migration tasks and limiting chip power, significantly suppressing the frequent oscillations between over-cooling and ineffective migration of the system, and taking into account both the performance guarantee of computing tasks and the optimal overall energy efficiency of the data center.

[0021] 5. This invention utilizes the temperature prediction residual within a sliding window and employs a recursive parameter identification algorithm to dynamically update the thermal influence matrix, cooling efficiency matrix, and model bias online, while adaptively adjusting the error compensation amount. This enables the system to proactively adapt to hardware aging, sudden changes in heat exchange conditions, and task computation mode drift. Simultaneously, it is equipped with multi-level safety handling and hysteresis recovery logic, including soft logic isolation and hardware power failure protection, to ensure that critical computing tasks are not interrupted and to implement tiered reliable protection for hardware when encountering extreme anomalies or when optimization is unsolvable.

[0022] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a server GPU resource and heat dissipation coordinated scheduling method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the system structure of the scheduling controller, GPU unit, sensor and heat dissipation execution unit according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the topological relationship between the GPU unit, the hot zone, and the heat dissipation execution unit according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the candidate action pair filtering and control command timing according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the state transition between normal operation and graded safety handling according to an embodiment of the present invention.

[0024] In the diagram, 100—Schedule Controller; 110—Data Acquisition and Topology Mapping Module; 120—Load and Heat Prediction Module; 130—Thermal Status Assessment Module; 140—Candidate Action Pair Generation Module; 150—Collaborative Decision Module; 160—Timing Control Module; 170—Feedback Safety Module; 200—GPU Unit; 210—GPU Resource Scheduler; 220—GPU Task Queue; 300—Temperature Sensor; 400—Heat Dissipation Execution Unit; 410—Fan; 420—Water Pump; 430—Regulating Valve; 500—Hot Zone. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0026] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the method may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments. In this invention, a GPU unit refers to a GPU chip, GPU board, or logical GPU partition that can independently acquire resource status and temperature status and receive resource scheduling commands; a GPU task includes training jobs, inference instances, high-performance computing jobs, or other computing tasks that can be executed on the GPU; a hot zone is a spatial unit used to uniformly express the effects of heat generation and heat dissipation; a heat dissipation execution unit is a controllable component that can receive control commands and change the heat dissipation effect of the corresponding hot zone, including fans, water pumps, valves, air valves, and adjustable cold sources or combinations thereof. In this invention, "GPU power parameter adjustment" is a higher-level concept, including GPU operating frequency adjustment and GPU power limit adjustment; when describing a change in the power limit value or rated power percentage, it is uniformly referred to as "GPU power limit adjustment". The cold source is used as a heat dissipation unit only when the supply air temperature, supply liquid temperature or output power of the cold source can be directly or indirectly adjusted by the dispatch controller; otherwise, the cold source status is only used as an environmental status input.

[0027] like Figure 2 As shown, the scheduling controller 100 includes a data acquisition and topology mapping module 110, a load and heat prediction module 120, a thermal state assessment module 130, a candidate action pair generation module 140, a collaborative decision-making module 150, a timing control module 160, and a feedback safety module 170. The scheduling controller 100 is communicatively connected to the GPU resource scheduler 210, the GPU task queue 220, the GPU unit 200, the temperature sensor 300, and the heat dissipation execution unit 400. The scheduling controller 100 can be deployed on a server's baseboard management controller, host operating system, container orchestration node, or independent management server; each module can be implemented by software, firmware, programmable logic, or a combination thereof; modules exchange state vectors with unified timestamps, candidate actions, predicted temperature trajectories, constraint flags, target action pairs, and execution feedback.

[0028] The server GPU resource and thermal management coordinated scheduling method executed by the scheduling controller 100 includes the following steps: S1. Operational Data Acquisition and Hot Zone Topology Establishment The system acquires the GPU resource status, GPU task status, measured temperature of the corresponding hot zone, operating status of the heat dissipation execution unit, and environmental status of each GPU unit; aligns the states according to a unified timestamp; and establishes a topological association between the GPU unit, hot zone, temperature sensor, and heat dissipation execution unit.

[0029] The GPU resource status includes at least one of the following: compute unit utilization, tensor compute unit utilization, memory usage, memory bandwidth, GPU interconnect bandwidth, operating frequency, power, and power limit; the GPU task status includes at least one of the following: task type, running stage, resource requirements, priority, service level requirements, completion deadline, migration status, checkpoint data volume, and parallel communication relationship; the thermal execution unit operating status includes at least one of the following: fan speed, airflow, damper opening, inlet air temperature, outlet air temperature, pump speed, branch flow rate, valve opening, supply liquid temperature, return liquid temperature, pressure, and fault indicators; the environmental status includes at least one of the following: external ambient temperature, computer room inlet air temperature, rack main supply liquid pipe temperature, or non-adjustable cold source status.

[0030] Hot zones 500 are divided according to GPU units, GPU boards, airflow coverage, cold plate coverage, or cooling branch coverage. In one implementation, GPU units that simultaneously meet the following conditions are preferentially assigned to the same hot zone: covered by the same dominant airflow, cold plate, or cooling branch; have consistent temperature response directions for the same heat dissipation action and a correlation coefficient not lower than a preset threshold; have higher thermal coupling strength within the zone than across zones; and have at least one effective temperature state node within the zone. When the number of sensors and actuators is sufficient to distinguish single-card responses, fine-grained granularity at the GPU unit or board level is used; when samples are insufficient or parameters are unidentifiable, they are merged into airflow or cooling branch level hot zones. The scheduling controller can periodically re-estimate the hot zone boundaries based on the sliding window temperature response correlation and model residuals.

[0031] A single thermal execution unit can influence one or more hot zones, and a hot zone can be influenced by multiple thermal execution units simultaneously. The topological relationships include the first relationship from the GPU unit to the hot zone and the second relationship from the thermal execution unit to the hot zone, which can be represented as the first association matrix from the GPU to the hot zone and the second association matrix from the thermal execution unit to the hot zone. Besides matrix representation, adjacency lists or thermal resistance network graphs can also be used: adjacency lists record the hot zones connected to each GPU unit, temperature sensor, and thermal execution unit, along with their influence weights; thermal resistance network graphs use temperature state nodes as vertices, thermal resistance or thermal conductance as edge weights, and GPU power and thermal control parameters as node inputs. Both representations can be converted into sparse association matrices for temperature trajectory calculation.

[0032] Taking heat zones Z1 (G1 to G4) and Z2 (G5 to G8) as examples, and considering that regulating valves V1 and V2 and water pump P act on these two heat zones, as follows: Figure 3 As shown, the correlation matrix from the GPU to the hot zone Desirable The correlation matrix between the heat dissipation unit and the hot zone Desirable After normalizing the power, temperature, and control parameters, a set of initial values ​​for describing the matrix structure can be obtained. , , The actual operating values ​​are identified from telemetry data.

[0033] The confidence level of a temperature sensor is determined based on at least two of the following: sensor self-test flag, freshness of sampled data, abnormal range, abnormal temperature change rate, and deviation between redundant temperature sensors.

[0034] Preferably, the sensor confidence level Normalization to Calculate using the following formula: ——(0) in This indicates that the self-test has passed. For data latency, The freshness time constant, Indicates the effective indicated quantity of the range. The deviation between the rate of temperature change and the allowable rate of change. To set an upper limit for the allowable rate of change, This represents the deviation between the fused values ​​of this sensor and redundant sensors in the same thermal zone. The upper limit of the deviation, weight to The sum is 1. Normal use is possible when the value is not lower than 0.8. In During intervals Normalization reduces the fusion weights. Temperatures with a confidence level below 0.6 are marked as invalid. The threshold is determined based on sensor accuracy, sampling period, and the false alarm and false negative rates of historical fault samples; the threshold can be increased for critical hot zones. The scheduling controller reduces the fusion weight or marks temperature data with a confidence level below the threshold as invalid and replaces it with redundant sensor data within the same hot zone, data from adjacent temperature status nodes, or dynamic thermal model estimates. When no usable temperature data meeting the confidence level requirement exists for a critical temperature status node, a safety procedure is triggered.

[0035] S2, GPU resource requirements, predicted power, and generation of predicted heat generation sequences. Predict the resource requirements of GPU tasks in the prediction time domain. For different candidate GPU resource actions such as task placement, task migration, GPU partitioning, resource share, operating frequency or power limit, calculate the predicted power of each GPU unit. Based on the thermal impact relationship between GPU units and hot zones, map the predicted power to the predicted heat generation sequence of each hot zone.

[0036] GPU task At the present moment after The predicted resource demand within each scheduling time slot is represented as follows: ——(1) In the formula, For GPU tasks In length of Load sequence within the historical window, This is the current operational phase. The parameters are batch size, remaining task quantity, priority, and service level. This is a resource demand prediction model; the predicted resource demand includes at least the share of computing units, memory capacity, memory bandwidth, and GPU interconnect bandwidth, and outputs the corresponding prediction uncertainty.

[0037] Resource demand forecasting models can employ autoregressive models, exponentially weighted regression models, or sequential neural networks. When using sequential neural networks, one feasible structure includes a sequentially connected input normalization layer, a first gated recurrent layer with 64 hidden units, a second gated recurrent layer with 32 hidden units, and a fully connected output layer; the input is continuous. The computational unit utilization, memory bandwidth, input / output throughput, task stage encoding, and batch size at each sampling time are output as future values. The computational share, memory bandwidth, interconnect bandwidth, mean predicted power, and logarithmic variance of each scheduling slot are calculated. Training samples are obtained by dividing historical telemetry data from the server into time windows. The sum of the mean square error and the negative log-likelihood of the variance between the predicted and measured values ​​is used as the loss function, and the independent validation period is used to determine the training cycle to stop.

[0038] In one implementation, training data is stratified by GPU model, task type, and load stage. Training, validation, and test sets are divided into non-overlapping time periods, and the test set retains task batches not appearing during the training stage for generalization verification. Each GPU model and task type combination covers at least low, medium, and high load levels and 20 complete task cycles. The number of hidden units in the gated recurrent layer can be selected from 32 to 128 and 16 to 64, respectively, with a history window... It can cover 30 to 300 seconds, and the learning rate can be... to The dropout ratio can be selected within a range of 0 to 0.3, and hyperparameters are determined jointly based on the temperature or power prediction error and negative log-likelihood of the independent validation set. Model embedding, rated power, and memory bandwidth normalization are used for different GPU models. When the uncertainty of a single model exceeds the threshold, a heterogeneous ensemble of 3 to 5 models consisting of autoregressive models, gradient boosting trees, and gated recurrent networks is used. The prediction uncertainty is formed by the difference in the mean and residual variance of the predictions of each model, and the amount of error compensation in the uncertainty and double margin is increased in the early stage of the new model's launch.

[0039] Actions targeting candidate GPU resources Mapping predicted resource requirements to GPU units The predicted load is calculated, and the predicted power is calculated using the following formula: ——(2) In the formula, For GPU unit Idle power; , and These are respectively the computing unit load, the video memory bandwidth load, and the input / output load; The operating frequency or the equivalent frequency corresponding to the power limit. Maximum operating frequency; to This refers to the power coefficient calibrated using historical power data. The predicted power of each GPU unit constitutes the predicted power vector. The thermal influence matrix is ​​used to map the chip temperature status nodes and hot spot temperature status nodes within each hot zone. Simultaneously, the data is aggregated according to the first correlation matrix from the GPU to the hot zone to form a predicted heat generation sequence for each hot zone, used to calculate the predicted heat load and cooling capacity margin for each hot zone. Hot spot temperatures can be obtained through the hot spot temperature sensor built into the GPU chip.

[0040] S3. Predicting heat dissipation effect sequence and generating temperature trajectory. For different candidate heat dissipation actions such as fans, water pumps, valves, cooling medium flow rate, supply air temperature, or cooling medium inlet temperature, a predicted heat dissipation effect sequence mapped to the same hot zone and the same prediction time domain is generated by combining the effect relationship of the heat dissipation execution unit on the hot zone and the heat dissipation response delay. Candidate GPU resource actions and candidate heat dissipation actions are combined into candidate action pairs, and the predicted temperature trajectory of the hot zone corresponding to each candidate action pair is calculated.

[0041] In one implementation, candidate action pairs are generated using a hierarchical approach: "resource action pre-screening—heat dissipation action discretization—intra-hot zone pairing—cross-hot zone conflict resolution." Resource actions are first filtered based on GPU model and instruction set compatibility with the task, available memory and bandwidth of the target GPU, parallel group and CPU affinity, and current temperature safety margin. Heat dissipation actions are sampled at preset levels within the device's allowable range and rate of change constraints. The number of migrated tasks is limited in each scheduling cycle, and only heat dissipation actions that have a direct effect on risky hot zones are retained, thereby deleting obviously infeasible combinations before entering the dynamic thermal model calculation. Only two actions with the same hot zone identifier, prediction start point, sampling interval, and prediction length form a candidate action pair. Each candidate action pair records the target GPU, target setpoint, and effective time of the resource action, as well as the target actuator, target setpoint, and effective time of the heat dissipation action, and serves as a set of joint boundary conditions for the dynamic thermal model.

[0042] A dynamic thermal model incorporating thermal inertia, inter-GPU heat propagation, interaction relationships between thermal execution units, thermal response delay, and inlet temperature effects is used to calculate the predicted thermal zone temperature for candidate action pairs at each prediction time. ——(3) In the formula, It is a thermal state vector composed of the chip temperature state nodes and hot spot temperature state nodes of the GPU units in each hot zone. The thermal inertia matrix; This is the thermal effect matrix. The elements in the table represent the impact of power changes in each GPU unit on its own or adjacent GPU temperature status nodes. Actions for candidate GPU resources The corresponding predicted power vectors for each GPU unit; For cooling efficiency matrix, The elements in the text represent the cooling effect of air volume, flow rate, fan speed, water pump speed, or valve opening on each temperature state node; To account for the response delay of heat dissipation intensity The heat dissipation control vector; To account for inlet temperature response delay Candidate inlet air temperature or cooling medium inlet temperature; The matrix showing the influence of inlet temperature; The corresponding state components output by equation (3) are used as the chip temperature prediction value and hot spot temperature prediction value of each GPU unit, respectively.

[0043] When the sampling and control period is At that time, the heat dissipation response delay and inlet temperature response delay Based on heat dissipation response time Inlet temperature response time With sampling and control cycle Confirmed, specifically , ,in This indicates rounding up to the nearest integer.

[0044] In one implementation, response time can be identified online: within each sliding time window, the GPU power increment... Incremental heat dissipation control Or inlet temperature increment The cross-correlation between the temperature residual after removing the load effect and the value of the absolute correlation coefficient is calculated. The time lag at which the absolute correlation coefficient first reaches the peak value of the temperature residual is taken as the threshold value. or Candidate values; when a safety step exists, the time from command change to temperature response reaching 10% of steady-state change can also be used as a pure time lag estimate. Candidate values ​​are updated exponentially and limited to the minimum and maximum response times calibrated by the equipment; updates are only made when the control excitation amplitude and signal-to-noise ratio meet the identifiable conditions; otherwise, the previous cycle value is used and the uncertainty is increased.

[0045] For cold plate type liquid cooling hot zone Available heat dissipation capacity Calculate using the following formula: ——(4) In the formula, Hot Zone The heat exchange efficiency of the corresponding cooling branch. The mass flow rate of the cooling medium. The specific heat capacity at constant pressure of the cooling medium. The maximum permissible return temperature, The supply temperature is used. For air-cooled hot zones, the cooling medium mass flow rate can be replaced with the air mass flow rate, and the available heat dissipation capacity can be calculated using the allowable temperature difference between the inlet and outlet air. In one embodiment, for each candidate heat dissipation action... First, based on the pump speed-pressure difference curve, valve flow resistance curve, and branch hydraulic model, or based on the flow response model identified by the nearest window, predict the mass flow rate corresponding to this action at each prediction time. Then substitute the candidate flow into equation (4) to obtain the candidate action specific The cooling capacity margin is then calculated by combining the predicted thermal load of candidate GPU resource actions with the calculation of the cooling capacity margin, thereby avoiding circular dependencies.

[0046] S4, Double Margin Assessment and Candidate Action Pair Screening The temperature safety margin is determined based on the predicted temperature trajectory of the hot zone, the upper temperature limit, and the prediction error. The cooling capacity margin is determined based on the available heat dissipation capacity of the heat dissipation execution unit, the predicted heat load, and the reserved heat dissipation capacity. Candidate action pairs that do not meet the GPU resource constraints, GPU topology constraints, hot zone temperature constraints, or cooling capacity constraints are excluded.

[0047] For hot zone Temperature safety margin Perform the calculation: ——(5) In the formula, the minimum value calculation covers the hot zone. All GPU units within; and These are the upper limit of chip temperature and the upper limit of hot spot temperature, respectively, determined based on the GPU's rated junction temperature, device management policies, or preset safe temperatures. and These are the predicted chip temperature and the predicted hot spot temperature, respectively. and The covariance matrix of temperature prediction error The predicted standard deviation is obtained by taking the square root of the corresponding diagonal elements; This is the prediction error compensation coefficient corresponding to the set confidence level. The upper bound of the chip temperature prediction risk is... The upper limit of the risk for hotspot temperature prediction is When the prediction error approximately follows a zero-mean normal distribution, Take one-sided confidence level The corresponding standard normal quantile, for example When the percentages are 90%, 95%, 97.5%, and 99%, The values ​​are approximately 1.282, 1.645, 1.960, and 2.326, respectively. For critical online tasks or when sensor confidence decreases, a higher confidence level should be selected; for deferred batch processing tasks, a lower confidence level, but not below the lower limit of the safety policy, can be selected. When the prediction residuals significantly deviate from a normal distribution, The empirical quantiles or conformal calibration upper bounds of the sliding window residuals are used as substitutes.

[0048] Hot Zone Cooling capacity margin Represented as: ——(6) In the formula, For GPU unit Power to the hot zone heat load conversion coefficient, To reserve heat dissipation capacity, Equations (5) and (6) are calculated for each candidate action pair. When the temperature safety margin corresponding to the candidate action pair is less than the preset lower limit of temperature margin, or the cooling capacity margin is less than the preset lower limit of cooling capacity, the candidate action pair is excluded.

[0049] In one implementation, the quantification of GPU resource constraints and GPU topology constraints includes: after adding the peak memory requirements of the task and safety reserves, the predicted memory utilization of the target GPU must not exceed 90%; the predicted utilization of computing share, memory bandwidth, and interconnect bandwidth must not exceed its allocable capacity; after migration, GPUs within the parallel group should meet the requirements of GPU model consistency in the task configuration and the minimum number of NVLink direct connections. If these requirements cannot be met, they must be connected through the same PCIe switching layer, and the predicted effective bandwidth must not be lower than the minimum communication bandwidth of the task; placement across NUMA nodes should also meet CPU affinity and round-trip latency limits. The above thresholds can be configured according to GPU model and service level. Exceeding any threshold indicates that the action does not meet the corresponding resource or topology constraints.

[0050] S5, Multi-objective collaborative decision-making Based on GPU task priority, service level requirements, GPU computing power consumption, heat dissipation power consumption, GPU task migration cost, temperature over-limit risk, and control quantity switching cost, the remaining candidate action pairs are jointly evaluated to determine the target action pair.

[0051] Candidate GPU resource actions include GPU task placement, task migration, GPU partition rebinding, computing resource share adjustment, operating frequency adjustment, and power limit adjustment; candidate cooling actions include fan speed or damper opening adjustment, water pump speed adjustment, branch valve opening or cooling medium flow rate adjustment, and supply air temperature or cooling medium inlet temperature adjustment. GPU resource constraints and GPU topology constraints include at least one of the following: computing resource capacity constraints, memory capacity constraints, memory bandwidth constraints, GPU interconnect bandwidth constraints, CPU-GPU affinity constraints, GPU task parallel group integrity constraints, and migration interruptible time constraints.

[0052] GPU task From the source GPU unit Migrate to target GPU unit migration cost Represented as: ——(7) In the formula, Weights corresponding to GPU task priorities; The amount of data to be migrated; The available bandwidth between the source GPU unit and the target GPU unit; Allow time for cache or model warm-up; Reconstructing time for parallel communication relationships; Energy consumption for state replication; , Energy consumption and time delay weighting coefficients; This represents the communication latency increment caused by migration. Task priorities can be set sequentially to critical online, high priority, normal, and deferred batch processing, corresponding to... Examples include 8, 4, 2, and 1; if the service level specifies the maximum interruptible time, it can be further normalized by its reciprocal. . It is estimated by the 90th percentile of the most recent model loading, cache filling and kernel warm-up time of the same task on the target GPU. If there is no historical record, it can be estimated by model size and storage or video memory loading bandwidth. Based on the parallel group size, the number of communication topology changes, and the estimated latency of aggregated communication probes, the 90th percentile of the historical reconstruction time for similar topologies can be used. In one implementation, the configurable initial ranges for both are 0.5 to 30 seconds and 0.2 to 10 seconds, respectively; online inference tasks use conservative upper quantile values, while batch processing tasks can use median values.

[0053] Joint evaluation index for candidate action pairs in the prediction time domain Perform the calculation: ——(8) In the formula, For GPU units and server computing power consumption, To cool the energy consumption of the actuator, Service level loss for GPU tasks To mitigate the risks of exceeding temperature limits and the rate of temperature rise, The cost of migrating GPU tasks, The switching cost caused by frequent changes in GPU power parameters and thermal control parameters. To predict the time-domain discount factor, to For evaluation weighting, the joint evaluation metric should include at least GPU task performance cost, GPU task migration cost, and thermal energy consumption cost, and may also include at least one of the following: GPU computing energy consumption, service level breach risk, hotspot temperature exceedance risk, and control variable switching cost. Improvements should be made to high-priority GPU tasks. The corresponding weight is reduced for batch processing tasks that allow for latency; the performance cost weight for different GPU tasks is determined based on the GPU task priority or service level requirements.

[0054] Temperature constraints in hot zones and the operating boundaries of heat dissipation execution units are considered hard constraints, and their priority is higher than the energy consumption cost and GPU task migration cost in the joint evaluation indicators. The weight of the joint evaluation indicators cannot be obtained at the expense of device safety in exchange for energy consumption or task performance gains. In one implementation, in an 8-GPU implementation, only the 8 best-performing GPU resource actions and 6 heat dissipation actions can be retained for each risky hot zone, and the number of tasks allowed to migrate per cycle is limited to 2, so that the initial action pairs for a single hot zone do not exceed 48; after merging across hot zones, no more than 128 candidate action pairs are retained according to the lower bound of the joint evaluation.

[0055] In one implementation, the solution process involves the scheduling controller first using hard constraints to eliminate infeasible action pairs, then establishing a candidate queue based on the lower bound of temperature risk, service level cost, and energy consumption cost. The action pairs in the queue are sorted according to the optimistic lower bound of the joint evaluation index, and a time-bound branch-and-bound or rolling time-domain mixed-integer linear programming search is employed. When a better feasible solution is found, the upper bound is updated, and branches whose lower bound is not better than the current upper bound are pruned. If the number of binary decision variables or candidate action pairs exceeds a preset size, a bundle search is used: each layer retains only the optimally evaluated... Each part of the action is expanded sequentially across hot zones, with a double margin check performed again after each expansion. The solution time limit is set as a preset proportion of the scheduling cycle; for example, at least 1 second is reserved within a 5-second cycle for distribution and feedback. The solver returns the current optimal feasible solution within 4 seconds. If the global optimality proof is not completed by then, the found optimal feasible solution is executed; if no feasible solution is found, a safety preset action is executed. The upper bound of the computational cost for evaluation after pre-screening can be expressed as: ,in , These are the number of resource actions and the number of heat dissipation actions, respectively. To predict the number of time slots, The number of hot zones. When using mixed-integer linear programming to solve the problem, the dynamic thermal model and heat dissipation energy consumption curve are piecewise linearized; when the number of candidate actions is small, the temperature trajectory of each action pair can be calculated one by one, and a heuristic search can be used to determine the action pair with the minimum joint evaluation index.

[0056] S6, Rolling Execution and Response Latency Coordination Within the rolling prediction time domain, the target action pair is determined, and only the control quantity corresponding to the target action pair in the current scheduling time slot is executed. The timing control module 160 determines the execution order and time offset of the GPU resource scheduling command and the thermal control command based on the GPU heat response time and thermal response time corresponding to the target action pair, and sends the corresponding commands to the GPU resource scheduler 210 and the thermal execution unit 400 respectively. For example... Figure 4 As shown, when the target action is executed in a timed manner, the timing of the heat dissipation effect reaching the target hot zone is set by setting an advance amount so that it is no later than the time when the GPU heat rises.

[0057] Lead time of thermal control commands relative to GPU resource scheduling commands Represented as: ——(9) In the formula, The heat dissipation response time of the heat dissipation execution unit from receiving a command to generating effective cooling in the corresponding hot zone. This refers to the thermal response time when a GPU task starts, resources are increased, or power parameters change until a significant temperature rise occurs in the corresponding hot zone. This is a timing protection value set based on prediction errors. When... When the value is greater than zero, first send the heat dissipation control command, and then wait. Alternatively, after receiving feedback that the thermal execution unit has reached the minimum target output, a GPU resource scheduling command can be sent to ensure that the time when the heat dissipation effect reaches the corresponding hot zone is no later than the time when the predicted temperature rise caused by the GPU resource scheduling command occurs; when the GPU task is already running and a thermal risk occurs, the GPU power limit command and the thermal enhancement command can be sent simultaneously.

[0058] S7, Closed-Loop Correction and Safety Handling The feedback safety module 170 collects the measured temperature, GPU power, and heat dissipation feedback after the target action is executed, and the temperature prediction residual. Represented as: ——(10) In the formula, For the first The measured temperature at each predicted time. This corresponds to the predicted temperature.

[0059] The scheduling controller updates the thermal influence matrix using a recursive least squares method with a forgetting factor, based on GPU power, thermal control parameters, inlet temperature, and measured temperature within a sliding time window. Cooling efficiency matrix and model bias Correlation analysis is used to estimate the response time from GPU resource actions or cooling actions to temperature response. Online identification of seasonal regression vectors is also performed. Update the parameter vector matrix according to the following formula. With covariance matrix :

[0060]

[0061]

[0062] Among them, the forgetting factor The value is selected between 0.95 and 0.999, and the updated results are projected onto a set of physical constraints that ensure thermal inertia stability, non-negativity of the power-to-temperature coefficient, and non-negativity of the cooling efficiency coefficient. Temperature prediction error covariance matrix. The temperature prediction residuals are updated based on the temperature state nodes within the sliding time window; as the temperature prediction residuals continuously increase, the accuracy is improved. The diagonal elements corresponding to the temperature state nodes are used to increase the prediction error compensation in equation (5). The above-mentioned thermal effect matrix, cooling efficiency matrix, model bias, GPU heating response time, heat dissipation response time and temperature prediction error covariance matrix constitute the coupling parameters used to generate the predicted heating sequence, predicted heat dissipation effect sequence or hot zone predicted temperature trajectory.

[0063] When the measured chip temperature or hot spot temperature reaches the critical threshold, the temperature rise rate exceeds the critical temperature rise rate, the cooling capacity margin is continuously lower than the allowable value, the heat dissipation execution unit fails, the confidence level of the key temperature sensor is lower than the threshold, the temperature prediction residual continuously exceeds the residual threshold, or there are no feasible candidate action pairs within the specified solution time, the system enters the safety handling state.

[0064] Based on the risk level, select one or more safety measures to implement, including increasing available heat dissipation output, limiting GPU power, reducing GPU operating frequency, reducing the resource share of low-priority GPU tasks, migrating or suspending GPU tasks, and isolating GPU units. Level 1 safety measure involves adjusting the available heat dissipation execution unit in the high-risk hotspot to a safe preset output while limiting the power limit of the corresponding GPU unit. Level 2 safety measure involves reducing or suspending the resource share of low-priority GPU tasks and migrating high-heat-generating GPU tasks that can be migrated to GPU units with both temperature safety margins and cooling capacity margins. Level 3 safety measure involves stopping the dispatch of new tasks to the high-risk hotspot, saving the checkpoints of allowed tasks, and isolating or shutting down GPU units that have reached critical temperatures. A hysteresis is set between the entry and exit thresholds, and after the recovery conditions are continuously met for a preset period, the safety measures are exited step-by-step according to a hysteresis control method.

[0065] Isolating GPU units includes at least one of software logical isolation and hardware protection isolation. Software logical isolation refers to marking the corresponding GPU unit as unschedulable through the GPU resource scheduler, stopping the assignment of new tasks to it, revoking its logical partition or container device allocation, and keeping it in an idle or power-limited state. Hardware protection isolation refers to disabling the corresponding GPU's computing function, disconnecting its PCIe function, or cutting off the corresponding power supply branch through the baseboard management controller, GPU device management interface, PCIe device control interface, or power supply control circuit when the server supports independent device control. For Level 3 security procedures, software logical isolation is performed first after saving the task states that are allowed to be saved; hardware protection isolation is performed when the measured temperature continues to rise, the GPU experiences a hardware failure, or the device's emergency protection threshold is reached.

[0066] Initial thermal effect matrix and cooling efficiency matrix Identified from at least 24 hours of historical telemetry data; when historical data is insufficient, under the condition that the GPU temperature is below the warning threshold and the task allows, a step of no more than 5% of the rated range is applied to the upper limit of the power of a single GPU and the opening of a single regulating valve, respectively, and the temperature response of each hot zone is recorded and the matrix elements and response delay are fitted. In one implementation, the step amplitude starts from 3% of the rated range and gradually increases until the temperature response amplitude is not less than 3 times the standard deviation of the measurement noise of the same hot zone, while ensuring that the predicted temperature does not exceed the warning threshold or that the actuator rate of change exceeds the allowable value; active steps are not applied during tasks that cannot meet the safety excitation conditions. This can be obtained using CFD simulation, thermal resistance network calculation, or migration to a server of the same model. , , , As initial values, the parameter covariance and temperature prediction error covariance are increased; after going online, only natural load fluctuations and normal control actions are used for recursive correction, and the uncertainty is tightened after the residual statistics stabilize.

[0067] Example 1: Scheduling System and Liquid-Cooled GPU Server like Figure 2 As shown, a liquid-cooled GPU server is configured with eight GPU units G1 to G8. G1 to G4 are covered by the first cold plate branch and form a hot zone Z1, while G5 to G8 are covered by the second cold plate branch and form a hot zone Z2. The two branches are respectively equipped with a supply liquid temperature sensor, a return liquid temperature sensor, a flow sensor, and a regulating valve 430, and share an adjustable-speed water pump 420. Each GPU unit is equipped with a chip temperature sensor, a hot spot temperature sensor, and a board-level power sensor.

[0068] like Figure 3As shown, GPU units G1 to G8 are mapped to either heat zone Z1 or Z2 via the first correlation matrix; the two regulating valves primarily affect their respective heat zones, while the water pump 420 affects both Z1 and Z2. Therefore, in the second correlation matrix, the elements between the regulating valve and its corresponding heat zone are relatively large, the elements between the regulating valve and the other heat zone are zero or relatively small, and there are non-zero elements between the water pump and both heat zones. This matrix can express one-to-one, one-to-many, and many-to-many heat dissipation relationships.

[0069] The scheduling controller 100 uses a 1-second data sampling period and a 5-second scheduling period, with 12 scheduling periods for the prediction time domain. The server runs GPU tasks K1 to K3: K1 is a high-priority online inference task, requiring a minimum computational share and prohibiting prolonged interruptions; K2 is a medium-priority training task, which can be migrated at checkpoints; K3 is a low-priority batch processing task, which can be paused or have resources reduced. The collaborative decision-making module 150 sets different service level loss weights based on task priority.

[0070] During a scheduling cycle, the load and heat prediction module 120 determines that K2 will enter a memory bandwidth intensive phase in 20 seconds. If the current resource actions are maintained, the dynamic thermal model predicts that the hotspot temperature of G3 will reach 92.0℃ during this phase, with a prediction standard deviation of 1.2℃; when the compensation coefficient... When the value is 1.645, the upper limit of the hot spot temperature risk is about 94.0℃, and the temperature safety margin relative to the upper limit of the hot spot temperature of 95℃ is about 1.0℃, which is lower than the preset temperature margin lower limit of 2.0℃ set in this embodiment. At the same time, the cooling capacity margin of Z1 is -0.18kW.

[0071] The candidate action pair generation module 140 generates the following candidate action pairs: The first action pair maintains the GPU resource state and cooling state; the second action pair maintains the GPU resource state and increases the opening of the Z1 branch valve and the pump speed; the third action pair relocates a training copy of K2 from G3 to G7 while maintaining cooling state; the fourth action pair simultaneously increases the opening of the Z1 branch valve and the pump speed, relocates a training copy of K2 from G3 to G7, and briefly limits the power limit of G3. The first action pair is excluded because it does not meet the constraints of temperature safety margin and cooling capacity margin; the second action pair meets the temperature constraints, but has high cooling energy consumption; the third action pair still has a short-term hot spot risk when the cooling response is insufficient; the fourth action pair has a higher migration cost than the second action pair, but the sum of temperature risk, cooling energy consumption, and task performance cost is the lowest, therefore it is determined as the target action pair.

[0072] The timing control module 160 adjusts the opening of the Z1 branch valve from 52% to 70%, adjusts the water pump speed from 60% to 68%, relocates one training copy of K2 from G3 to G7, and temporarily adjusts the power limit of G3 from 350W to 315W. Since both task migration and power limiting are used to reduce the thermal load of Z1, the timing control module 160 simultaneously sends GPU resource scheduling commands and thermal control commands, and ensures that the relocation of the training copy is completed before K2 is expected to enter the memory bandwidth intensive stage; for resource actions that will increase heat generation in the hot zone, the advance amount of the thermal control command is determined according to equation (9), and the specific timing is shown in Example 2.

[0073] After the target action is executed, the predicted hotspot temperature of Z1 is 88.9℃, and the corresponding measured hotspot temperature is 89.4℃. The temperature prediction residual is 0.5℃, which is below the residual update threshold of 1.5℃, so the scheduling controller retains the current model parameters. If the residual exceeds 1.5℃ for three consecutive scheduling cycles, the thermal influence matrix is ​​updated using the most recent 60 sampling points. Cooling efficiency matrix And the prediction error variance. After the hotspot risk subsides, the G3 power limit and heat dissipation output gradually recover according to the preset rate of change. The sampling period, prediction time domain, temperature, power, and valve opening mentioned above are all example parameters used to illustrate the control process. Actual parameters are configured according to the GPU model, server architecture, cooling system rated capacity, and task service level requirements.

[0074] Example 2: Air-cooled GPU Server A single air-cooled GPU server is configured with four GPU units (G1 to G4), a front fan assembly (410), a rear fan assembly, two adjustable dampers, and inlet / outlet air temperature sensors. G1 and G2 are designated as hot zone Z1 based on their airflow coverage, while G3 and G4 are designated as hot zone Z2. The front fan assembly affects both Z1 and Z2, the two dampers alter the airflow distribution into Z1 and Z2 respectively, and the rear fan assembly affects the exhaust resistance of both hot zones. The scheduling controller uses fan PWM duty cycle, damper opening, and supply air temperature as candidate cooling actions, calculating available cooling capacity based on air mass flow rate, air specific heat capacity at constant pressure, and allowable inlet / outlet temperature difference. When an online inference task is expected to add a replica in 15 seconds, the scheduling controller combines the candidate GPU resource addition actions with multiple fan and damper actions, predicting the temperature trajectory of each action pair. If placing the replica in G2 causes the upper bound of the Z1 hotspot temperature risk to exceed the limit, while placing it in G4 and increasing the Z2 air valve opening from 45% to 60% and the front fan PWM from 55% to 65% can satisfy the double margin constraint, then the latter action is determined to be correct. The duct's calibrated heat dissipation response time is 6 seconds, the time from task replica startup to significant temperature rise is 2 seconds, and the timing protection amount is 1 second. According to formula (9), the heat dissipation command advance amount can be obtained as follows: Therefore, the heat dissipation command is sent 5 seconds before the copy startup command. This embodiment illustrates that the present invention does not rely on a specific liquid cooling structure.

[0075] Example 3: Safety Handling During the operation of Example 1, if the flow feedback of the first cold plate branch is lower than the control target, and the temperature prediction residual of Z1 exceeds the residual threshold for three consecutive scheduling cycles, the feedback safety module 170 determines that the cooling efficiency of the first cold plate branch has decreased. The feedback safety module 170 first enters a level one safety procedure, adjusting the water pump and available regulating valves to a safe preset output and limiting the power limits of G1 to G4; if the hot spot temperature continues to rise, it enters a level two safety procedure, reducing or suspending K3, migrating K2 from the most recent checkpoint to available GPU units in G5 to G8, while retaining the minimum resource share of K1; if the hot spot temperature reaches a critical threshold, it enters a level three safety procedure, stopping the dispatch of new tasks to G1 to G4, saving the allowed GPU task states, and isolating the risky GPU units. Figure 5 As shown, after the flow rate, hot spot temperature, cooling capacity margin, and temperature prediction residual of the first cold plate branch return to normal for several consecutive scheduling cycles, the feedback safety module 170 exits step by step in the order of Level 3 safety handling, Level 2 safety handling, Level 1 safety handling, and normal operation. The exit threshold at each level is lower than the corresponding entry threshold, forming a hysteresis interval to avoid frequent state switching. Among them, hysteresis and minimum hold time are used for entry and exit to suppress state oscillation.

[0076] Example 4: Multi-server rack-level collaborative scheduling In a multi-server rack embodiment, the rack includes four GPU servers S1 to S4, each configured with eight GPU units and a local cold plate branch. The four servers share a rack-wide supply and return manifold and an adjustable-speed water pump. Each server's GPU units are first mapped to its local cold plate hot zone, and each local hot zone is then mapped to the shared manifold hot zone, thus forming a two-level topology of "GPU—local hot zone—shared cooling loop." The rack-level scheduling controller aggregates the predicted heat generation sequence, double margin, and migrated tasks for each server, and combines cross-server task migration, branch valve opening, and main pump speed into rack-level candidate action pairs. During screening, in addition to single-machine resource and temperature constraints, the maximum flow rate of the main pipe, main pump power, supply temperature, rack uplink network bandwidth, and number of cross-server migrations are also constrained. During execution, a pump speed or supply temperature command that applies to multiple servers is first issued, followed by valve commands and cross-server migration commands based on the response time of each branch. Each server's local controller retains temperature criticality protection permissions, and reverts to local safety preset action pairs when rack-level communication is interrupted.

[0077] Example 5: Systems, Electronic Devices, and Storage Media The scheduling controller 100 can be deployed on a server's baseboard management controller, host operating system, container orchestration node, or standalone management server. Each module can be implemented by software, firmware, programmable logic, or a combination thereof; modules exchange state vectors with uniform timestamps, candidate actions, predicted temperature trajectories, constraint flags, target action pairs, and execution feedback.

[0078] The data acquisition and topology mapping module 110 is used to acquire the GPU resource status, GPU task status, measured temperature of the corresponding hot zone, operating status of the heat dissipation execution unit and environmental status of each GPU unit, and to establish the topological association between the GPU unit, hot zone, temperature sensor and heat dissipation execution unit.

[0079] The load and heat prediction module 120 is used to determine the predicted heat generation sequence of a candidate GPU resource action mapped to each hot zone in the prediction time domain. The thermal state evaluation module 130 is used to determine the predicted heat dissipation effect sequence of a candidate heat dissipation action mapped to the same hot zone in the same prediction time domain, wherein the predicted heat dissipation effect sequence includes the heat dissipation response delay of the corresponding heat dissipation execution unit.

[0080] The candidate action pair generation module 140 is used to combine candidate GPU resource actions and candidate heat dissipation actions into candidate action pairs, and calculate the predicted temperature trajectory of the hot zone corresponding to each candidate action pair based on the current measured temperature of each hot zone, the predicted heat generation sequence, and the predicted heat dissipation effect sequence. It determines the temperature safety margin based on the predicted temperature trajectory of the hot zone and the upper limit of the hot zone temperature, and determines the cooling capacity margin based on the available heat dissipation capacity of the heat dissipation execution unit and the predicted heat load corresponding to the predicted heat generation sequence.

[0081] The collaborative decision-making module 150 is used to exclude candidate action pairs that do not meet GPU resource constraints, GPU topology constraints, hot zone temperature constraints, or cooling capacity constraints, and to determine the target action pair from the remaining candidate action pairs based on a joint evaluation index that includes at least GPU task performance cost, GPU task migration cost, and heat dissipation energy consumption cost.

[0082] The timing control module 160 is used to determine the execution order and time offset of the GPU resource scheduling command and the heat dissipation control command based on the GPU heat response time and heat dissipation response time corresponding to the target action, and sends the GPU resource scheduling command and the heat dissipation control command to the GPU resource scheduler and the heat dissipation execution unit, respectively. The feedback safety module 170 is used to obtain the measured temperature and heat dissipation feedback after the target action is executed, and determines whether to update the coupling parameters used to generate the predicted heat generation sequence, the predicted heat dissipation effect sequence, or the predicted temperature trajectory of the hot zone based on the temperature prediction residual between the measured temperature and the corresponding predicted temperature, and triggers the next scheduling cycle; the coupling parameters include at least one of the following: thermal influence matrix, cooling performance matrix, model bias, GPU heat response time, heat dissipation response time, and temperature prediction error covariance matrix.

[0083] An electronic device includes a processor, a memory, a communication interface, and a computer program stored in the memory. When the processor executes the computer program, it performs state acquisition, sequence mapping, action pair generation, temperature trajectory prediction, double margin calculation, constraint screening, joint evaluation, timing control, model updating, and security handling, i.e., it executes the aforementioned server GPU resource and heat dissipation coordinated scheduling method. A computer-readable storage medium stores a computer program thereon, which, when executed by the processor, implements the aforementioned server GPU resource and heat dissipation coordinated scheduling method; the computer program can also be stored in a disk, solid-state memory, read-only memory, random access memory, optical media, or other non-transitory computer-readable storage media. This invention can also be applied to immersion liquid-cooled servers, cold plate liquid-cooled servers, single servers, GPU clusters composed of multiple servers, and racks sharing a cooling loop. Any implementation of this invention's concept is possible as long as the heat generation effect of GPU resource actions and the cooling effect of heat dissipation actions are mapped to the same thermal zone and the same prediction time domain, and constraint screening, joint evaluation, and timing execution are performed in the form of action pairs.

[0084] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for coordinated scheduling of server GPU resources and heat dissipation, characterized in that, A method applied to a server system comprising multiple GPU units, temperature sensors, and at least one controllable thermal execution unit, and executed by a scheduling controller communicatively connected to the GPU units and the thermal execution unit, the method comprising: The GPU resource status, GPU task status, measured temperature of the corresponding hot zone, operating status of the heat dissipation execution unit, and environmental status of each GPU unit are obtained, and a topological relationship is established between the GPU unit, the hot zone, the temperature sensor and the heat dissipation execution unit. For candidate GPU resource actions, a predicted heat generation sequence is determined for each hot zone within the prediction time domain; for candidate heat dissipation actions, a predicted heat dissipation effect sequence is determined for each hot zone within the same prediction time domain, wherein the predicted heat dissipation effect sequence includes the heat dissipation response delay of the corresponding heat dissipation execution unit. Candidate GPU resource actions and candidate heat dissipation actions are combined into candidate action pairs, and the predicted temperature trajectory of the hot zone corresponding to each candidate action pair is calculated based on the current measured temperature of each hot zone, the predicted heat generation sequence, and the predicted heat dissipation effect sequence. The temperature safety margin is determined based on the predicted temperature trajectory of the hot zone and the upper limit of the hot zone temperature, and the cooling capacity margin is determined based on the available heat dissipation capacity of the heat dissipation execution unit and the predicted heat load corresponding to the predicted heat generation sequence. Candidate action pairs that do not meet GPU resource constraints, GPU topology constraints, hot zone temperature constraints, or cooling capacity constraints are excluded. The target action pair is determined from the remaining candidate action pairs based on a joint evaluation index that includes at least GPU task performance cost, GPU task migration cost, and heat dissipation energy consumption cost. Based on the GPU heat response time and heat dissipation response time corresponding to the target action, the execution order and time offset of the GPU resource scheduling command and the heat dissipation control command are determined, and the GPU resource scheduling command and the heat dissipation control command are sent to the GPU resource scheduler and the heat dissipation execution unit, respectively. The measured temperature and heat dissipation feedback after the target action is executed are obtained. Based on the temperature prediction residual between the measured temperature and the corresponding predicted temperature, it is determined whether to update the coupling parameters used to generate the predicted heat generation sequence, the predicted heat dissipation effect sequence, or the predicted temperature trajectory of the hot zone, and the next scheduling cycle is triggered. The coupling parameters include at least one of the following: thermal effect matrix, cooling performance matrix, model bias, GPU heat dissipation response time, heat dissipation response time, and temperature prediction error covariance matrix.

2. The server GPU resource and heat dissipation coordinated scheduling method according to claim 1, characterized in that, The GPU resource status includes at least one of the following: compute unit utilization, tensor compute unit utilization, memory usage, memory bandwidth, GPU interconnect bandwidth, operating frequency, power, and power limit. The GPU task status includes at least one of the following: task type, running stage, resource requirements, priority, service level requirements, completion deadline, migration status, checkpoint data volume, and parallel communication relationship. The operating status of the cooling execution unit includes at least one of the following: fan speed, airflow, damper opening, inlet air temperature, outlet air temperature, water pump speed, branch flow rate, valve opening, supply liquid temperature, return liquid temperature, pressure, and fault flag. The hot zones are divided according to GPU units, GPU boards, air duct coverage, cold plate coverage, or cooling branch coverage. The topology association includes a first association from GPU units to hot zones and a second association from cooling execution units to hot zones. One cooling execution unit can affect one or more hot zones, and a hot zone can be affected by multiple cooling execution units simultaneously. The GPU resource status, the GPU task status, the measured temperature, and the operating status of the cooling execution unit are aligned using a unified timestamp.

3. The server GPU resource and heat dissipation coordinated scheduling method according to claim 1, characterized in that, The process of determining the predicted heat generation sequence includes: based on the historical load sequence of GPU tasks, the current running stage, batch size, remaining task volume, and service level requirements, using a resource demand prediction model to predict at least one of the following GPU resource requirements in the prediction time domain: computing unit share, memory capacity, memory bandwidth, and GPU interconnect bandwidth, and outputting the corresponding prediction uncertainty; mapping the GPU resource requirements corresponding to candidate GPU resource actions to the predicted load of each GPU unit, and determining the predicted power of each GPU unit based on GPU idle power, computing unit load, memory bandwidth load, input / output load, and GPU operating frequency or power limit; and converting the predicted power into the predicted heat generation sequence of each hot zone based on the topological association between the GPU unit and the hot zone.

4. The server GPU resource and heat dissipation coordinated scheduling method according to claim 1, characterized in that, A dynamic thermal model, incorporating thermal inertia, inter-GPU heat propagation, the interaction of heat dissipation execution units, heat dissipation response delay, and the influence of inlet temperature, is used to calculate the predicted temperature trajectory of the hot zone. The dynamic thermal model uses a thermal state vector composed of chip temperature state nodes and hot spot temperature state nodes of GPU units within each hot zone as state variables, the predicted power of each GPU unit as heat dissipation input, and heat dissipation control variables considering the heat dissipation response delay and inlet air temperature or cooling medium inlet temperature considering the inlet temperature response delay as heat dissipation input. The model outputs the predicted chip temperature and hot spot temperature values ​​at each prediction time. The heat dissipation response delay and the inlet temperature response delay are determined based on the heat dissipation response time, the inlet temperature response time, and the sampling and control cycle, respectively.

5. The server GPU resource and heat dissipation coordinated scheduling method according to claim 1, characterized in that, For each hot zone, the difference between the upper limit of chip temperature and the upper limit of chip temperature prediction risk for each GPU unit within that hot zone, as well as the difference between the upper limit of hot spot temperature and the upper limit of hot spot temperature prediction risk, are calculated. The minimum value among these differences is taken as the temperature safety margin for that hot zone. The upper limit of chip temperature prediction risk and the upper limit of hot spot temperature prediction risk are obtained by summing the corresponding temperature prediction value and the prediction error compensation amount determined based on the temperature prediction error, respectively. The cooling capacity margin is the difference between the available cooling capacity of the hot zone corresponding to the candidate cooling action and the predicted heat load and reserved cooling capacity of that hot zone. For liquid-cooled hot zones, the available cooling capacity is determined based on the heat exchange efficiency of the cooling branch, the mass flow rate of the cooling medium, the specific heat capacity of the cooling medium at constant pressure, and the maximum allowable difference between the return temperature and the supply temperature. For air-cooled hot zones, the available cooling capacity is determined based on the air mass flow rate, the specific heat capacity of the air at constant pressure, and the allowable inlet and outlet air temperature difference. When the temperature safety margin corresponding to a candidate action pair is less than the preset temperature margin lower limit, or the cooling capacity margin is less than the preset cooling capacity lower limit, the candidate action pair is excluded.

6. The server GPU resource and heat dissipation coordinated scheduling method according to claim 1, characterized in that, The candidate GPU resource actions include at least one of the following: GPU task placement, GPU task migration, GPU partition adjustment, computing resource share adjustment, GPU operating frequency adjustment, and GPU power limit adjustment; the candidate cooling actions include at least one of the following: fan speed adjustment, damper opening adjustment, water pump speed adjustment, cooling branch flow rate adjustment, cooling branch valve opening adjustment, supply air temperature adjustment, and cooling medium inlet temperature adjustment; the GPU resource constraints and GPU topology constraints include at least one of the following: computing resource capacity constraints, memory capacity constraints, memory bandwidth constraints, GPU interconnect bandwidth constraints, CPU-GPU affinity constraints, GPU task parallel group integrity constraints, and migration interruptible time constraints. The GPU task migration cost is determined based on at least two of the following: GPU task priority, amount of state data to be migrated, available transmission bandwidth between the source GPU unit and the target GPU unit, task interruption time, cache or model warm-up time, communication relationship reconstruction time, and state replication energy consumption. The joint evaluation index also includes at least one of GPU computing energy consumption, service level breach risk, hot zone temperature exceedance risk, and control quantity switching cost. The performance cost weights corresponding to different GPU tasks are determined based on GPU task priority or service level requirements. The hot zone temperature constraint and the operating boundary of the heat dissipation execution unit are considered hard constraints, and their priority is higher than the energy consumption cost and GPU task migration cost in the joint evaluation index.

7. The server GPU resource and heat dissipation coordinated scheduling method according to claim 1, characterized in that, Within the rolling prediction time domain, the target action pair is determined, and only the control quantity corresponding to the target action pair in the current scheduling time slot is executed. The advance of the cooling control command relative to the GPU resource scheduling command is determined based on the difference between the cooling response time and the GPU heat response time, ensuring that the cooling effect reaches the corresponding hot zone no later than the predicted temperature rise time caused by the GPU resource scheduling command. Upon the expiration of the advance or after receiving feedback that the cooling execution unit has reached the minimum target output, GPU task startup, GPU task migration, or GPU resource share increase are executed. When the measured temperature or temperature rise rate reaches a critical threshold, the cooling capacity margin is continuously lower than the allowable value, the cooling execution unit malfunctions, the temperature sensor confidence level is lower than the threshold, or the temperature prediction residual continuously exceeds the residual threshold or does not exist... When candidate actions are executed, one or more safety measures are selected according to the risk level, including increasing available heat dissipation output, limiting GPU power, reducing GPU operating frequency, reducing the resource share of low-priority GPU tasks, migrating or pausing GPU tasks, and isolating GPU units. After the recovery conditions are continuously met for a preset period, the safety measures are exited according to the hysteresis control method. The isolated GPU unit includes at least one of software logic isolation and hardware protection isolation. The software logic isolation includes removing the risky GPU unit from the schedulable resource set, stopping the dispatch of new tasks and migrating it, and pausing or saving the existing task state. The hardware protection isolation includes disconnecting the link or power supply of the GPU unit by the baseboard management controller, PCIe hot-swap controller or power supply control circuit when the temperature or fault reaches the hardware protection threshold.

8. A server GPU resource and heat dissipation coordinated scheduling device, characterized in that, include: The data acquisition and topology mapping module is used to acquire the GPU resource status, GPU task status, measured temperature of the corresponding hot zone, operating status of the heat dissipation execution unit, and environmental status of each GPU unit, and to establish the topological relationship between the GPU unit, hot zone, temperature sensor and heat dissipation execution unit. The load and heat prediction module is used to determine the predicted heat sequence of candidate GPU resource actions mapped to each hot zone in the prediction time domain for the candidate GPU resource actions. The thermal state assessment module is used to determine the predicted heat dissipation effect sequence of a candidate heat dissipation action in the same predicted time domain, which is mapped to the same thermal region. The predicted heat dissipation effect sequence includes the heat dissipation response delay of the corresponding heat dissipation execution unit. The candidate action pair generation module is used to combine candidate GPU resource actions and candidate heat dissipation actions into candidate action pairs, and calculate the predicted temperature trajectory of the hot zone corresponding to each candidate action pair based on the current measured temperature of each hot zone, the predicted heat generation sequence, and the predicted heat dissipation effect sequence. The module determines the temperature safety margin based on the predicted temperature trajectory of the hot zone and the upper limit of the hot zone temperature, and determines the cooling capacity margin based on the available heat dissipation capacity of the heat dissipation execution unit and the predicted heat load corresponding to the predicted heat generation sequence. The collaborative decision-making module is used to exclude candidate action pairs that do not meet GPU resource constraints, GPU topology constraints, hot zone temperature constraints, or cooling capacity constraints, and to determine the target action pair from the remaining candidate action pairs based on a joint evaluation index that includes at least GPU task performance cost, GPU task migration cost, and heat dissipation energy consumption cost. The timing control module is used to determine the execution order and time offset of the GPU resource scheduling command and the heat dissipation control command based on the GPU heat response time and heat dissipation response time corresponding to the target action, and send the GPU resource scheduling command and the heat dissipation control command to the GPU resource scheduler and the heat dissipation execution unit respectively. The feedback safety module is used to obtain the measured temperature and heat dissipation feedback after the target action is executed, and to determine whether to update the coupling parameters used to generate the predicted heat generation sequence, the predicted heat dissipation effect sequence or the predicted temperature trajectory of the hot zone based on the temperature prediction residual between the measured temperature and the corresponding predicted temperature, and to trigger the next scheduling cycle. The coupling parameters include at least one of the following: thermal effect matrix, cooling performance matrix, model bias, GPU heat dissipation response time, heat dissipation response time, and temperature prediction error covariance matrix.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the server GPU resource and heat dissipation coordinated scheduling method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the server GPU resource and heat dissipation coordinated scheduling method as described in any one of claims 1 to 7.

Images