Reinforcement learning-based data center high-density computing resource adaptive scheduling system
By constructing an action network and a heat conduction model, combined with a microchannel cooling system, the problem of scheduling strategy divergence caused by heat conduction lag in traditional reinforcement learning in high-density computing systems was solved, thereby improving the stability and security of high-density computing systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING SHIYUN INFORMATION TECH CO LTD
- Filing Date
- 2026-05-25
- Publication Date
- 2026-07-31
AI Technical Summary
Traditional reinforcement learning in high-density computing systems suffers from the lag in heat conduction across time and space, which causes the optimization direction of scheduling strategies to diverge. It is impossible to accurately trace the source of thermal runaway, resulting in thermoelectric positive feedback and subthreshold leakage, making it difficult to meet the stability and security requirements of high-density computing systems.
By constructing an action network, the absolute temperature field and dynamic command heating power density field in three-dimensional space are obtained. Combined with the partial differential analytical model of heat conduction, a hysteresis temperature increment field and a local entropy yield prediction field are generated. The update step of the action network is limited by the expected value of the instantaneous reward and the dynamic discount factor, and precise scheduling is achieved by combining it with a microchannel cooling system.
It effectively overcomes the attribution misalignment problem caused by spatiotemporal thermal delay, significantly reduces the risk of chip thermal runaway, improves system operation stability and computing power efficiency, and reduces the temperature fluctuation range.
Smart Images
Figure CN122240283B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically, to an adaptive scheduling system for high-density computing resources in data centers based on reinforcement learning. Background Technology
[0002] As the computing power density of data centers continues to rise, computing units and cache units in modern ultra-high-density computing systems (such as 3D chip stacking packaging) are extremely close in space. When the scheduling system allocates high-intensity computing tasks to the underlying logic units, the instantaneously generated localized high heat will be conducted to the surrounding three-dimensional space. Due to the thermal inertia of the silicon substrate and other media, the transfer of heat energy is accompanied by an inevitable time delay. When this heat wave is conducted to the neighboring cache unit, the passively increased temperature will trigger a severe subthreshold leakage current phenomenon, which will then generate secondary waste heat and form a local thermoelectric positive feedback.
[0003] Current conventional approaches to adaptive resource scheduling using reinforcement learning generally rely on Markov decision process theory, which assumes that the system state at the next moment is determined solely by the current state and actions. However, in the aforementioned ultra-high-density computing systems, severe leakage and thermal runaway occurring in a certain region at the current moment are often caused by scheduling actions taken against other regions at a previous moment. This spatiotemporal thermal conduction lag and associative memory effect leads to severe attribution misalignment in traditional reinforcement learning agents: the agent is prone to incorrectly punishing routine scheduling actions at the current moment, failing to accurately trace the source action that caused the disaster. These shortcomings cause the optimization direction of traditional strategies to easily diverge, with the agent oscillating violently between maximizing computing power and avoiding thermal failure, making it difficult to meet the stringent requirements of ultra-stability and safe operation in high-density computing systems. Summary of the Invention
[0004] This invention provides an adaptive scheduling system for high-density computing resources in data centers based on reinforcement learning, which solves the technical problems mentioned in the background art.
[0005] This invention provides a high-density computing resource adaptive scheduling system for data centers based on reinforcement learning, applicable to high-density computing architectures including logic processing units, cache units, and microchannel cooling systems. The system includes a pre-defined action network, comprising: Obtain the absolute temperature field and dynamic command heating power density field of the computational system in three-dimensional space, and construct the volume entropy yield field; The absolute concurrent total computing power requirement is obtained, and the dimensionless preference continuous field is output by the action network to generate a three-dimensional continuous computing power density field. Substitute the three-dimensional continuous computational density field as a heat source term into the partial differential analytical model of heat conduction, and obtain the hysteresis temperature increment field by feedforward solution. The absolute temperature field and the hysteresis temperature increment field are superimposed to calculate the total local entropy productivity prediction field, and the instantaneous reward expectation value is obtained by combining the obtained ambient air intake absolute cold source temperature. Obtain the total convective heat transfer admittance of the system and the total heat capacity of the cluster, and generate a dynamic discount factor; The cumulative advantage function is derived using the expected value of the instant reward and the dynamic discount factor. Combined with the obtained total true thermodynamic potential energy of the system, the update step size of the action network is limited to update the weights of the action network. The local flow velocity command field is calculated inversely based on the three-dimensional continuous computing power density field and sent to the microchannel cooling system for execution.
[0006] The beneficial effects of this invention are as follows: by mapping the multidimensional computing power distribution into a continuous field of spatial heat dissipation and leakage coupling, and combining the heat conduction attenuation law of the time scale for feedforward evolution, the scheduling action is strictly limited within the boundary of energy conservation and irreversible loss of system work capacity. This effectively overcomes the attribution misalignment problem caused by cross-temporal thermal delay in traditional reinforcement learning under extremely high-density computing power systems, realizes smooth scheduling of high-density computing arrays under critical thermal limit state, significantly reduces the risk of chip thermal runaway and improves overall computing power efficiency and system operation stability. Attached Figure Description
[0007] Figure 1 This is a flowchart of the adaptive scheduling system for high-density computing resources in data centers based on reinforcement learning, as described in this invention. Detailed Implementation
[0008] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.
[0009] like Figure 1 As shown, a reinforcement learning-based adaptive scheduling system for high-density computing resources in data centers is applied to a high-density computing architecture that includes logic processing units, cache units, and microchannel cooling systems. The system includes a pre-defined action network, comprising: Obtain the absolute temperature field and dynamic command heating power density field of the computational system in three-dimensional space, and construct the volume entropy yield field; The absolute concurrent total computing power requirement is obtained, and the dimensionless preference continuous field is output by the action network to generate a three-dimensional continuous computing power density field. Substitute the three-dimensional continuous computational density field as a heat source term into the partial differential analytical model of heat conduction, and obtain the hysteresis temperature increment field by feedforward solution. The absolute temperature field and the hysteresis temperature increment field are superimposed to calculate the total local entropy productivity prediction field, and the instantaneous reward expectation value is obtained by combining the obtained ambient air intake absolute cold source temperature. Obtain the total convective heat transfer admittance of the system and the total heat capacity of the cluster, and generate a dynamic discount factor; The cumulative advantage function is derived using the expected value of the instant reward and the dynamic discount factor. Combined with the obtained total true thermodynamic potential energy of the system, the update step size of the action network is limited to update the weights of the action network. The local flow velocity command field is calculated inversely based on the three-dimensional continuous computing power density field and sent to the microchannel cooling system for execution.
[0010] This implementation is applied to a high-density computing system that includes logic processing units, cache units, and a microchannel cooling system. The system includes a preset action network and a value network, and performs the following steps: The absolute temperature field and dynamic command heating power density field of the computational system in three-dimensional space are obtained to construct the volume entropy yield field.
[0011] The preset activation bandgap energy, Boltzmann constant, reference leakage current density constant, and thermal conductivity of silicon material are obtained, and the spatial temperature gradient modulus of the absolute temperature field is extracted. The activation bandgap energy of silicon material is an intrinsic physical parameter of silicon material, and the correction formula for its variation with temperature is as follows: In the formula, This represents the temperature-corrected activation bandgap energy of silicon, in units of... ; The absolute temperature of silicon is expressed in units of 1000°C. The absolute temperature field value is taken from the corresponding spatial location; to convert it to Joules, it needs to be multiplied by . The Boltzmann constant is a physical constant with a value of [value missing]. The reference leakage current density constant is 300K, and the intrinsic leakage current heating density of silicon material at standard temperature is expressed in units of... The initial values were obtained from the semiconductor device datasheet and calibrated online during operation; the thermal conductivity of the material is a parameter representing the thermal conductivity of silicon, and the correction formula for its variation with temperature is as follows. The unit is The absolute temperature field was acquired using an array of temperature sensors distributed across the surface and interior of the silicon wafer. The sensors were arranged in a grid pattern with a spacing of no more than 1 mm and a sampling frequency of 100 Hz. Discrete sensor data were reconstructed into a continuous three-dimensional spatial temperature distribution using Kriging interpolation. The units are... The spatial temperature gradient modulus is the result obtained by taking the modulus after performing spatial gradient calculations on the absolute temperature field, and the unit is 1. This characterizes the rate of temperature change in space. All sensors are clocked using a precise time protocol with a synchronization error of no more than 1ms, ensuring time consistency of multi-source data.
[0012] The leakage current heating density is obtained by dividing the activation bandgap energy of the silicon material by the product of the Boltzmann constant and the absolute temperature field, taking the negative number, performing a natural exponential operation, and then multiplying it by the reference leakage current density constant. The calculation formula is as follows: ; In the formula, Representing three-dimensional spatial position place, time The leakage current heating density, in units of ; Indicates the reference leakage current density constant; This represents the temperature-corrected activation bandgap energy of silicon. Represents the Boltzmann constant; Representing three-dimensional spatial position place, time The absolute temperature.
[0013] The first entropy production component is obtained by adding the dynamic command heating power density field to the leakage current heating density and then dividing by the absolute temperature field. The dynamic command heating power density field is the instantaneous heating power density generated by the logic processing unit executing the command at the current moment, and the unit is... The execution rates of various instructions are directly read from the performance counters of each computing core through the out-of-band management interface or eBPF technology at the hardware level to meet the requirements of low latency and low CPU overhead within the scheduling time window. The collected execution rates are multiplied by the single-instruction energy consumption constant of the corresponding instruction and then summed. The energy consumption constants of different instruction types are pre-calibrated through chip power consumption testing. The first entropy production component characterizes the entropy production rate generated by the combined current heating and leakage current heating, with units of [unit missing]. .
[0014] Dividing the square of the spatial temperature gradient modulus by the square of the absolute temperature field, and then multiplying by the thermal conductivity of the material, yields the second entropy production component. This second entropy production component characterizes the entropy production rate resulting from the irreversibility of heat conduction, and its unit is... .
[0015] The volumetric entropy yield field is obtained by adding the first entropy yield component and the second entropy yield component. The calculation formula is as follows: ; In the formula, Represents the volume entropy yield field, in units of ; This represents the dynamic command-driven heating power density field; This represents the temperature-corrected thermal conductivity of the material. Represents the spatial temperature gradient of the absolute temperature field; Represents a three-dimensional spatial position vector; Represents the time variable.
[0016] The absolute concurrent total computing power requirement is obtained, and the dimensionless preference continuous field is output by the action network to generate a three-dimensional continuous computing power density field.
[0017] Obtain the preset silicon core space indicator function and define the usable silicon wafer volume range as the boundary of the three-dimensional spatial integration. The silicon core space indicator function is a binary function, and when the spatial position... When located within the silicon core region, the function value is Otherwise This is used to define the effective spatial range for computing power allocation. The silicon core space indicator function is extracted from the chip's GDSII design document and converts the physical layout into a voxelized mesh with a voxel resolution of [missing information]. The grid coordinates correspond one-to-one with the physical space of the silicon wafer, with the origin located at the bottom left corner of the wafer. The usable silicon wafer volume range is the three-dimensional space occupied by all silicon cores in the high-density computing system.
[0018] The dimensionless preference continuous field is subjected to natural exponential operation and multiplied by the silicon core space indicator function to obtain the local preference weight field. The dimensionless preference continuous field is a three-dimensional voxel tensor output by the action network, with the same dimension as the silicon core space indicator function, and its value range is [value range missing]. To avoid numerical overflow during natural exponent arithmetic, a maximum value subtraction method is used for numerical stabilization; that is, the maximum value is calculated first. Then, further calculations are performed. Exponentialization ensures that local preference weights are non-negative.
[0019] The local preference weight field is integrated in three dimensions over the available silicon wafer volume to obtain a globally normalized weight benchmark. The three-dimensional volume integration uses the trapezoidal rule, with the integration step size consistent with the voxel resolution, and the unit is [unit missing]. The globally normalized weight benchmark is used to normalize local preference weights to... Interval.
[0020] The local preference weight field is divided by the global normalized weight benchmark, and then multiplied by the absolute total concurrent computing power requirement to generate the three-dimensional continuous computing power density field. The calculation formula is as follows: ; In the formula, Represents a three-dimensional continuous computational density field, with units of ; This represents a dimensionless, biased continuous field. Indicates the silicon core space indicator function; Indicates the range of available silicon wafer volume; Represents a three-dimensional volume integral element; This represents the total absolute concurrent computing power requirement, in units of... This represents the total floating-point operation rate that the system needs to process at the current moment, obtained from the upper-level task scheduler.
[0021] The action network employs a 3D convolutional neural network architecture. The input state vector contains four tensors: absolute temperature field, dynamic command heating power density field, ambient intake absolute cold source temperature, and current local flow velocity command field. These input tensors are normalized to... The action network consists of three 3D convolutional layers with kernel sizes of [sizes to be filled in]. , , The number of channels is 32, 16, and 1 respectively. The activation function is ReLU, and the output layer has no activation function, resulting in a dimensionless, biased continuous field. To ensure the spatial smoothness of the output field, a spatial gradient penalty term is added to the loss function, with a penalty coefficient of [value missing]. The value network shares the first two convolutional layers with the action network, followed by two fully connected layers. The output dimension is 1, which is used to estimate the state value function.
[0022] The three-dimensional continuous computational density field is substituted as a heat source term into the partial differential analytical model of heat conduction, and the hysteresis temperature increment field is obtained by feedforward solution.
[0023] Obtain the preset intrinsic energy consumption constant, volumetric specific heat capacity, and thermal diffusivity of silicon dielectric for a single floating-point operation, and set the integration time variable and scheduling time window. The intrinsic energy consumption constant for a single floating-point operation is the average energy consumed by the silicon core to perform one floating-point operation, in units of... The initial value was obtained through chip power consumption testing and calibrated online during operation; the volumetric specific heat capacity is the temperature rise per unit volume of silicon material. The required heat is taken as a value. The thermal diffusivity of silicon dielectric characterizes the rate of heat conduction in silicon materials, and is taken as a value of The integral time variable is the time variable of the heat conduction process, and its unit is _____. The scheduling time window is the time interval during which the system makes a scheduling decision, measured in units of... The value is To balance scheduling accuracy and computational overhead, for heterogeneous dielectric regions, the thermal conductivity of logic processing units is taken as the thermal conductivity of silicon material, the thermal conductivity of cache units is taken as 0.85 times the thermal conductivity of silicon material, and the thermal conductivity of through-silicon vias is taken as... The interlayer thermal resistance of the three-dimensional stacked chip is taken as: In heat conduction calculations, the interfacial heat flow continuity condition is used for processing.
[0024] Multiplying the three-dimensional continuous computing power density field by the intrinsic energy consumption constant of a single floating-point operation yields the dynamically injected heating field. The unit of the dynamically injected heating field is... , which represents the instantaneous heat power density generated by the allocated computing power.
[0025] The spatial diffusion attenuation term is derived by taking the negative of the square of the distance between two points in space, dividing it by four times the product of the thermal diffusivity of the silicon dielectric and the integration time variable, and then using this product as the exponent for natural exponential calculation. The distance between the two points represents their three-dimensional spatial positions. and The Euclidean distance between them, in units of The spatial diffusion attenuation term characterizes the rate attenuation of heat from location during heat conduction. propagation to location The degree of spatial decay over time. To avoid numerical singularities when the integration time variable approaches 0, the minimum integration time step is set to... When the integral time variable is less than the threshold, the Green's function is approximated by Taylor expansion.
[0026] Dividing the dynamically injected heating field by the volumetric specific heat capacity, multiplying by the spatial diffusion attenuation term, and then dividing by a cube-2 term comprising the product of four times pi, the thermal diffusivity of the silicon medium, and the integral time variable, yields the feedforward solution core formula. The cube-2 term, comprising the product of four times pi, the thermal diffusivity of the silicon medium, and the integral time variable, is the normalization factor for the three-dimensional heat conduction Green's function. The feedforward solution core formula characterizes the time... ,Location The heat generated at that location over time propagation to location The density of temperature increment caused by time.
[0027] The feedforward solution core is subjected to three-dimensional volume integration within the available silicon wafer volume, and time integration is performed within the scheduling time window to obtain the hysteresis temperature increment field. Both the three-dimensional volume integration and time integration employ the trapezoidal rule, with the spatial integration step size consistent with the voxel resolution, and the time integration step size being... The integration operation is accelerated using GPU parallel computing, with spatial block processing to ensure that the computation time is less than the scheduling time window. The calculation formula is as follows: ; In the formula, Represents the hysteresis temperature increment field, in units of ; Indicates the scheduling time window, in units of ; This represents the thermal diffusivity of silicon dielectric, in units of... ; Represents the integration time variable, in units of ; Represents the time integral infinitesimal element, with units of . ; This represents the intrinsic energy consumption constant for a single floating-point operation, in units of 1. ; Specific heat capacity is expressed in units of volume. ; Represents pi; Represents the natural constant; Represents the three-dimensional spatial position vector of the temperature increment to be calculated, in units of ; A three-dimensional spatial integral variable vector representing the location of the heat source, in units of ; This represents the three-dimensional continuous computing power density field at the location of the heat source. The value at the location, in units of ; Represents a three-dimensional volume integral element, with units of . The hysteresis temperature increment field characterizes the temperature increment caused by the heat generated by the current computing power allocation propagating to various spatial locations through heat conduction within the scheduling time window, reflecting the time hysteresis effect of heat conduction. Considering that the actual chip has a finite volume boundary, in order to ensure the applicability of the feedforward solution core formula obtained from the aforementioned infinite medium analytical model in the physical system, the system introduces a first-order mirror heat source near the chip periphery and interlayer adiabatic boundaries to perform boundary folding compensation for the spatial diffusion attenuation term; while for the heat dissipation surface boundary in contact with the microchannel, its boundary heat flux has been independently extracted and processed by the local flow velocity command field and convective heat transfer steps subsequently solved by the system, so that the hysteresis temperature increment field mainly characterizes the internal solid-solid conduction hysteresis.
[0028] The absolute temperature field and the hysteresis temperature increment field are superimposed to calculate the total local entropy yield prediction field. This prediction is then combined with the obtained ambient intake absolute cold source temperature to derive the immediate reward expectation. Finally, the absolute temperature field and the hysteresis temperature increment field are added together to obtain the predicted evolution absolute temperature field. The unit of the predicted evolution absolute temperature field is... This characterizes the predicted temperature distribution of the system after the scheduling time window.
[0029] The active bandgap energy of the silicon material is divided by the product of the Boltzmann constant and the predicted evolution absolute temperature field. The negative number is then used for natural exponential calculation, multiplied by the baseline leakage current density constant, and finally divided by the predicted evolution absolute temperature field to obtain the predicted leakage current heating entropy production term. The predicted leakage current heating entropy production term characterizes the entropy production rate generated by leakage current heating at the predicted temperature, and its unit is... .
[0030] The spatial temperature gradient modulus of the predicted evolving absolute temperature field is extracted, its square is divided by the square of the predicted evolving absolute temperature field, and then multiplied by the thermal conductivity of the material to obtain the predicted heat transfer dissipation entropy term. The unit of the spatial temperature gradient modulus of the predicted evolving absolute temperature field is... The predicted heat dissipation entropy production term characterizes the entropy production rate caused by the irreversibility of heat conduction at the predicted temperature, with units of . .
[0031] The predicted entropy product of leakage current heating is added to the predicted entropy product of heat transfer dissipation to obtain the total local entropy yield prediction field. The unit of the total local entropy yield prediction field is... .
[0032] Multiplying the predicted total local entropy yield by the ambient inlet absolute cold source temperature and performing a three-dimensional volume integral within the available silicon wafer volume range yields the irreversible damage rate of workability. The ambient inlet absolute cold source temperature is the liquid inlet temperature of the microchannel cooling system, in units of... The temperature is collected by a temperature sensor at the inlet of the cooling system. The unit for the irreversible damage rate of work capacity is... This characterizes the rate at which the system loses its work capacity due to entropy production within the scheduling time window.
[0033] The expected value of the immediate reward is obtained by inverting the irreversible damage rate of the work capacity and adding it to the computing power satisfaction rate penalty term. The computing power satisfaction rate penalty term is the penalty when the actual total computing power is less than the absolute concurrent total computing power requirement, and the penalty coefficient is set to a value of [value missing]. The calculation formula is: ; In the formula, This represents the expected value of the immediate reward, in units of... ; This indicates the absolute cold source temperature of the ambient intake air, in units of... ; Represents the dynamic command heating power density field, with units of ; Represents the reference leakage current density constant, in units of ; This represents the Boltzmann constant, with units of 1000 ppm. ; Representing three-dimensional spatial position place, time absolute temperature, in units of ; Represents the hysteresis temperature increment field, in units of ; Represents the temperature-corrected three-dimensional spatial position place, time Temperature-corrected thermal conductivity of silicon, in units of Its correction formula is Independent variable Substitute the absolute temperature at the current location ; Represents the spatial gradient operator; Represents a three-dimensional volume integral element, with units of . ; This represents the penalty coefficient for computing power satisfaction rate, in units of... ; This represents the total absolute concurrent computing power requirement, in units of... Obtained from the upper-level task scheduler; Represents a three-dimensional continuous computational density field, with units of .
[0034] The expected value of immediate rewards guides reinforcement learning agents to optimize in a direction that reduces irreversible system losses while meeting computing power requirements.
[0035] Obtain the total convective heat transfer admittance and total heat capacity of the cluster, and generate a dynamic discount factor.
[0036] Obtain the preset surface convection heat transfer coefficient and the defined heat dissipation surface area boundary. The surface convection heat transfer coefficient is the convective heat transfer coefficient between the cooling medium and the silicon wafer surface in the microchannel cooling system, and its unit is 1. The correction formula for flow velocity variation, obtained in advance through computational fluid dynamics simulation, is as follows: In the formula, The reference convective heat transfer coefficient is expressed in units of 1000 ppm. ; The flow rate of the microchannel cooling medium is expressed in units of... ; The velocity correction exponent for the convective heat transfer coefficient is dimensionless and represents an empirical fitting constant for the laminar-turbulent transition region of the microchannel. The heat dissipation surface area boundary encompasses all surface areas of the silicon wafer in contact with the cooling medium, determined through the chip packaging design documentation.
[0037] The total convective heat transfer admittance of the system is obtained by integrating the surface convective heat transfer coefficient over the boundary of the heat dissipation surface area. The unit of the heat dissipation surface area is... The total convective heat transfer admittance of the system is in units of It represents the heat dissipation capacity of the entire cooling system.
[0038] The total heat capacity of the cluster is obtained by performing a three-dimensional volume integral over the available silicon wafer volume using the volumetric specific heat capacity. The unit of total heat capacity of the cluster is [missing information - likely a unit of measurement]. It characterizes the ability of the entire high-density computing system to store heat.
[0039] Multiplying the scheduling time window by the total convective heat transfer admittance of the system, and then dividing by the total heat capacity of the cluster, yields the heat dissipation relaxation ratio. The heat dissipation relaxation ratio is a dimensionless parameter that characterizes the ratio of system heat dissipation to total heat storage within the scheduling time window, reflecting the system's thermal response speed.
[0040] The dynamic discount factor is derived by taking the negative of the heat dissipation relaxation ratio and performing a natural exponential operation, then incorporating a lower bound constraint. The lower bound of the dynamic discount factor is set at 0.1 to prevent it from approaching 1 and causing an unlimited expansion of the credit allocation range. The calculation formula is as follows: ; In the formula, This represents the dynamic discount factor, a dimensionless parameter with a value range of [value range missing]. ; Indicates the boundary of the heat dissipation surface area; Indicates the surface convective heat transfer coefficient; This represents a micro-element of the heat dissipation surface area; This represents the position vector of the heat dissipation surface. The dynamic discount factor can adapt to the thermal state of the system, solving the problem in traditional reinforcement learning where the fixed discount factor cannot adapt to thermal conduction hysteresis.
[0041] The cumulative advantage function is derived using the expected value of the instant reward and the dynamic discount factor. Combined with the obtained total true thermodynamic potential energy of the system, the update step size of the action network is limited to update the action network weights.
[0042] A time series of the expected instant reward is established and obtained. The action network weights of the previous time step are extracted, and a preset Fisher information matrix and the policy log probability gradient output by the action network are obtained. The time series length of the expected instant reward is set to 100 scheduling time windows; historical data exceeding this length are truncated. The action network weights of the previous time step are the action network parameters obtained after the last update. The Fisher information matrix is approximated using the KFAC method, divided into block diagonal matrices according to network layers, and the damping coefficient is set to a value of... The update frequency is once every 10 scheduling iterations. The policy distribution adopts a multivariate Gaussian distribution, with a diagonal covariance matrix, and the initial value is [value missing]. It is adaptively updated during the training process. The policy log probability gradient is the gradient of the policy log probability output by the action network with respect to the network weights, where the current system state includes information such as the absolute temperature field and the dynamic command heating power density field, and the current action is a dimensionless preference continuous field.
[0043] Based on the dynamic discount factor, the expected value of the immediate reward over the time series is cumulatively calculated, and the cumulative advantage function is derived using generalized advantage estimation. The parameters of the generalized advantage estimation are... The value is set to 0.95, and the time series cutoff length is 100. After the cumulative dominance function is calculated, it is normalized by moving average with a sliding window size of 1000 to ensure that the numerical range of the dominance value is stable. Between. The cumulative advantage function unit is This represents the advantage of the current action over the average action, that is, the incremental reward that the current action can bring in the future.
[0044] Multiplying the volumetric specific heat capacity by the absolute temperature field and performing a three-dimensional volume integral over the usable silicon wafer volume yields the total true thermodynamic potential energy of the system. It represents the total thermal energy stored in the entire high-density computing system and is a global measure of the system's thermal state.
[0045] Dividing the cumulative advantage function by the total true thermodynamic potential energy of the system yields the restricted update step size coefficient. This restricted update step size coefficient is a dimensionless parameter used to limit the update step size of the action network, ensuring that policy updates do not cause drastic changes in the system's thermal state and avoiding the risk of thermal runaway. An upper limit of 0.01 is set for the restricted update step size coefficient to prevent excessively large step sizes.
[0046] Multiplying the logarithmic probability gradient of the policy by the inverse of the Fisher information matrix yields the natural evolution gradient vector. This natural evolution gradient vector, corrected by the inverse of the Fisher information matrix, allows for the steepest descent on the Riemannian manifold of the policy distribution, improving the efficiency and stability of policy optimization.
[0047] The updated action network weights are obtained by multiplying the naturally evolved gradient vector by the restricted update step coefficient and adding it to the action network weights from the previous time step. The calculation formula is as follows: ; In the formula, This represents the updated action network weights; Indicates the network weights representing the actions at the previous moment; This represents the cumulative advantage function obtained through generalized advantage estimation; The KFAC approximation of the inverse of the Fisher information matrix; This represents the logarithmic probability gradient of the policy; Indicates the current state; Indicates the current action.
[0048] Reinforcement learning training is divided into two stages: offline pre-training and online fine-tuning. The offline pre-training stage uses a physical simulation environment with historical running data as initial conditions. This physical simulation environment strictly adheres to the partial differential physical models disclosed in the aforementioned specification, such as the absolute temperature field, hysteresis temperature increment field, and volume entropy yield field, to construct the state transition logic, and employs the finite difference method to perform numerical integration evolution over the time window. The experience replay pool size is... The batch size is 64, and the learning rate is... The weight decays to The number of training iterations is Pre-training terminates when the average reward shows no improvement after 1000 consecutive iterations. The online fine-tuning phase uses real-time running data for training, with an experience replay pool size of [size missing]. Batch size is 32, learning rate is Parameters are updated every 10 scheduling iterations. The loss function of the value network uses mean squared error loss, and the update frequency is the same as that of the action network.
[0049] The local flow velocity command field is calculated inversely based on the three-dimensional continuous computing power density field and sent to the microchannel cooling system for execution.
[0050] A one-dimensional integral path along the longitudinal direction of the silicon wafer thickness is defined, and the absolute inlet temperature, the absolute surface temperature of the cold plate, and the preset volumetric specific heat capacity of the cooling liquid medium are obtained. The one-dimensional integral path along the longitudinal direction of the silicon wafer thickness is the vertical path from the bottom to the top of the silicon wafer, and the unit is... The absolute inlet temperature refers to the inlet coolant temperature of the microchannel cooling system, in units of... The temperature is collected by a temperature sensor at the inlet of the cooling system. The absolute temperature of the cold plate surface is the temperature of the contact surface between the silicon wafer and the cold plate, and the unit is 100°C. The temperature data is collected by temperature sensors distributed on the surface of the cold plate. The volumetric specific heat capacity of the cooling liquid medium is the temperature rise per unit volume of the cooling medium. The corrected formula for the required heat as a function of temperature is: The unit is .
[0051] Multiplying the three-dimensional continuous computing power density field by the intrinsic energy consumption constant of a single floating-point operation yields the dynamically calculated heat density field. The unit of the dynamically calculated heat density field is... .
[0052] The leakage current heating density is added to the dynamically calculated heating density field, and then integrated along a one-dimensional integral path along the longitudinal direction of the silicon wafer thickness to obtain the surface equivalent absolute heat flux distribution. The one-dimensional integral element unit of thickness is... The unit of surface equivalent absolute heat flux distribution is , which represents the total heat flux that needs to be dissipated per unit area of the silicon wafer surface.
[0053] The heat exchange temperature difference is obtained by subtracting the absolute liquid inlet temperature from the absolute surface temperature of the cold plate. The unit of heat exchange temperature difference is... It is the driving force of convective heat transfer.
[0054] The heat absorption capacity per unit volume is obtained by multiplying the specific heat capacity of the cooling liquid medium by the heat transfer temperature difference. The unit of heat absorption capacity per unit volume is... It represents the amount of heat that a unit volume of coolant can absorb.
[0055] The local velocity command field is calculated by dividing the surface equivalent absolute heat flux distribution by the heat absorption capacity per unit volume. After incorporating upper and lower velocity limits, this command is sent to the microchannel cooling system for execution. The lower velocity limit is set as follows: To avoid insufficient laminar flow heat dissipation; the upper limit of the flow rate is set to To avoid cavitation and pressure shocks, when the calculated flow rate exceeds the upper and lower limits, the corresponding boundary value is used. The response time of the microchannel flow control valve is 100ms. Feedforward compensation is added to the scheduling algorithm to issue the flow rate command one scheduling time window in advance. The flow-velocity conversion relationship of each microchannel is obtained through factory calibration and is calibrated online during operation through feedback from the flow sensor. The calculation formula is: ; In the formula, Represents the local flow velocity command field, in units of ; This indicates that the input value is restricted to... Amplitude limiting function within the interval; This represents the one-dimensional integral path along the longitudinal direction of the silicon wafer thickness, in units of... ; This represents a one-dimensional integral infinitesimal element of thickness, with units of 1. ; Represents a three-dimensional continuous computational density field, with units of ; This represents the intrinsic energy consumption constant for a single floating-point operation, in units of 1. ; Represents the reference leakage current density constant, in units of ; This represents the Boltzmann constant, with units of 1000 ppm. ; Representing three-dimensional spatial position place, time absolute temperature, in units of ; This represents the volumetric specific heat capacity of the cooling liquid medium after temperature correction, in units of... ; This indicates the absolute inlet temperature, in units of... ; This indicates the absolute surface temperature of the cold plate, in units of... ; Represents the position vector of the heat dissipation surface, in units of The system sends local flow rate commands to the flow control valves of the microchannel cooling system via Ethernet protocol, with a communication delay of no more than 1ms. By adjusting the coolant flow rate of each microchannel, precise heat dissipation matching the computing power distribution is achieved.
[0056] During system operation, online parameter calibration and anomaly handling are performed. The reference leakage current density constant and the intrinsic energy consumption constant of a single floating-point operation are calibrated online every hour. The calibration process uses the Levenberg-Marquardt algorithm to solve the inverse heat transfer problem, using the mean square error between the measured temperature collected by the sensor and the model predicted temperature as the objective function. The parameter matrix is iteratively updated until the mean square error is less than a preset convergence threshold, yielding the calibrated parameters. The temperature sensors employ a redundant design. When the data from a single sensor is abnormal, it is supplemented by interpolation from adjacent sensors. When more than 20% of the sensors fail, the system's frequency reduction protection is triggered. When the microchannel flow rate is below 80% of the lower limit, it is determined to be a microchannel blockage, automatically reducing the computing power allocation to the corresponding area and issuing an alarm. When the maximum value of the predicted evolution absolute temperature field exceeds 373K, the thermal runaway emergency mechanism is triggered, immediately reducing the total system computing power to 50% of the rated value and increasing the flow rate of all microchannels to the upper limit.
[0057] The system is deployed on a dedicated management server and connects to compute nodes and the cooling control system via a PCIe interface. The scheduling time window is dynamically adjusted based on system load. When the load exceeds 80%, the scheduling time window is shortened to 0.05 seconds to improve response speed; when the load is below 30%, the scheduling time window is extended to 0.2 seconds to reduce computational overhead. The three-dimensional continuous computing power density field is converted into specific task allocation instructions through a task mapping module and distributed to each compute unit. Specifically, the task mapping module first performs a three-dimensional volume integral of the three-dimensional continuous computing power density field using the actual spatial geometric boundaries of each physical computing core as the integration domain, calculating the discrete expected computing power value for each physical core. Subsequently, the task to be processed is bound to the corresponding core through the operating system's thread affinity mechanism, and the dynamic voltage and frequency adjustment interface is called to set the core's operating frequency to the physical level closest to the expected computing power value. The task mapping process incorporates the principle of minimum migration cost, prioritizing the allocation of tasks to currently idle computing cores.
[0058] Experimental verification shows that, in a 3D stacked high-density computing system, compared with traditional reinforcement learning scheduling algorithms, this system reduces the probability of thermal runaway by more than 95%, improves the energy efficiency ratio by 22%, maintains the computing power satisfaction rate at more than 99.9%, and controls the temperature fluctuation range within ±3K, effectively solving the problem of spatiotemporal thermal hysteresis attribution misalignment in ultra-high-density computing systems.
[0059] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.
Claims
1. A reinforcement learning based adaptive scheduling system for high-density computing resources in data centers, applied to a high-density computing system comprising logical processing units, cache units, and micro-channel cooling systems, the system comprising a pre-set action network, characterized in that, Configured for execution: Obtain the absolute temperature field and dynamic command heating power density field of the computational system in three-dimensional space, and construct the volume entropy yield field, including: The preset activation bandgap energy, Boltzmann constant, reference leakage current density constant, and thermal conductivity of silicon material are obtained, and the spatial temperature gradient modulus of the absolute temperature field is extracted. The leakage current heating density is obtained by dividing the activation band gap energy of the silicon material by the product of the Boltzmann constant and the absolute temperature field, taking the opposite number, performing natural exponentiation, and then multiplying it by the reference leakage current density constant. Add the dynamic command heating power density field to the leakage current heating density, and then divide by the absolute temperature field to obtain the first entropy production component. Divide the square of the spatial temperature gradient modulus by the square of the absolute temperature field, and then multiply by the thermal conductivity of the material to obtain the second entropy production component. Add the first entropy production component to the second entropy production component to obtain the volume entropy production field; Obtain the absolute concurrent total computing power requirement, and generate a three-dimensional continuous computing power density field based on the dimensionless preference continuous field output by the action network, including: Obtain the preset silicon core space indicator function and define the available silicon wafer volume range as the three-dimensional space integration boundary; The dimensionless preference continuous field is subjected to natural exponential operation and multiplied by the silicon core space indicator function to obtain the local preference weight field; The local preference weight field is integrated in three dimensions over the available silicon wafer volume to obtain a global normalized weight benchmark. The local preference weight field is divided by the global normalized weight benchmark, and then multiplied by the absolute concurrent total computing power requirement to generate the three-dimensional continuous computing power density field. Substituting the three-dimensional continuous computational density field as a heat source term into the partial differential analytical model of heat conduction, the hysteresis temperature increment field is obtained by feedforward solution, including: Obtain the preset intrinsic energy consumption constant, volumetric specific heat capacity, and thermal diffusivity of silicon dielectric for a single floating-point operation, and set the integration time variable and scheduling time window; Multiply the three-dimensional continuous computing power density field by the intrinsic energy consumption constant of a single floating-point operation to obtain the dynamic injection heating field; The square of the distance between two points in space is taken as the opposite, divided by four times the product of the thermal diffusivity of the silicon medium and the integral time variable, and then used as the exponent for natural exponential calculation to obtain the spatial diffusion attenuation term. Divide the dynamically injected heating field by the volumetric specific heat capacity, multiply by the spatial diffusion attenuation term, and then divide by the cube of two terms which include the product of four times pi, the thermal diffusivity of the silicon dielectric, and the integral time variable to obtain the feedforward solution core formula. The feedforward solution core is integrated in three dimensions within the available silicon wafer volume range, and integrated in time within the scheduling time window to obtain the hysteresis temperature increment field. The absolute temperature field and the hysteresis temperature increment field are superimposed to calculate the total local entropy yield prediction field. Combined with the obtained ambient intake absolute cold source temperature, the immediate reward expectation value is derived, including: The absolute temperature field is added to the hysteresis temperature increment field to obtain the predicted evolution absolute temperature field. Divide the aforementioned activated bandgap energy of the silicon material by the product of the aforementioned Boltzmann constant and the predicted evolution absolute temperature field, take the negative number and perform natural exponentiation, multiply by the aforementioned benchmark leakage density constant, and then divide by the predicted evolution absolute temperature field to obtain the predicted leakage heating entropy term. Extract the spatial temperature gradient modulus of the predicted evolution absolute temperature field, divide its square by the square of the predicted evolution absolute temperature field, and then multiply it by the aforementioned obtained thermal conductivity of the material to obtain the predicted heat transfer dissipation entropy product. The predicted leakage current heating entropy term is added to the predicted heat transfer dissipation entropy term to obtain the total local entropy production rate prediction field. Multiply the total local entropy production rate prediction field by the absolute cold source temperature of the ambient air intake, and perform three-dimensional volume integration within the available silicon wafer volume range to obtain the irreversible damage rate of workability. The expected value of the immediate reward is obtained by taking the opposite of the irreversible damage rate of the functional ability. Obtain the total convective heat transfer admittance of the system and the total heat capacity of the cluster, and generate a dynamic discount factor, including: Obtain the preset surface convection heat transfer coefficient and the defined heat dissipation surface area boundary; The total convective heat transfer admittance of the system is obtained by integrating the surface convective heat transfer coefficient over the boundary of the heat dissipation surface area. The total heat capacity of the cluster is obtained by performing a three-dimensional volume integral on the aforementioned volumetric specific heat capacity within the available silicon wafer volume range; Multiply the aforementioned scheduling time window by the total convective heat transfer admittance of the system, and then divide by the total heat capacity of the cluster to obtain the heat dissipation relaxation ratio. The dynamic discount factor is obtained by taking the opposite of the heat dissipation relaxation ratio and performing natural exponentiation. Using the expected value of the immediate reward and the dynamic discount factor, a cumulative advantage function is derived. Combined with the obtained total true thermodynamic potential energy of the system, the update stride of the action network is limited to update the action network weights, including: Establish and obtain the time series of the expected value of the instant reward, extract the action network weights of the previous moment, and obtain the preset Fisher information matrix and the policy log probability gradient output by the action network. Based on the dynamic discount factor, the expected value of the instantaneous reward on the time series is cumulatively calculated to obtain the cumulative advantage function; Multiply the aforementioned volumetric specific heat capacity by the absolute temperature field, and perform a three-dimensional volume integral within the available silicon wafer volume range to obtain the total true thermodynamic potential energy of the system. Divide the cumulative advantage function by the total true thermodynamic potential energy of the system to obtain the constrained update step coefficient; Multiply the logarithmic probability gradient of the policy by the inverse of the Fisher information matrix to obtain the natural evolution gradient vector; Multiply the natural evolution gradient vector by the restricted update step coefficient and add it to the action network weights of the previous time step to obtain the updated action network weights. The local flow velocity command field is calculated inversely based on the three-dimensional continuous computing power density field and then sent to the microchannel cooling system for execution, including: Define a one-dimensional integral path along the longitudinal direction of the silicon wafer thickness, and obtain the absolute liquid inlet temperature, the absolute temperature of the cold plate surface, and the preset volumetric specific heat capacity of the cooling liquid medium; Multiply the three-dimensional continuous computing power density field by the aforementioned single floating-point operation intrinsic energy consumption constant to obtain the dynamically calculated heat density field; The leakage current heating density calculated above is added to the dynamically calculated heating density field, and then integrated along a one-dimensional integration path along the longitudinal direction of the silicon wafer thickness to obtain the surface equivalent absolute heat flux distribution. The heat exchange temperature difference is obtained by subtracting the absolute liquid inlet temperature from the absolute surface temperature of the cold plate. The heat absorption capacity per unit volume is obtained by multiplying the specific heat capacity of the cooling liquid medium by the heat exchange temperature difference. The local flow velocity command field is calculated by dividing the surface equivalent absolute heat flux distribution by the heat absorption capacity per unit volume and then sent to the microchannel cooling system for execution.