A processor peak temperature optimization method and system based on deep reinforcement learning
By using a hard real-time constraint model based on deep reinforcement learning and a hybrid reward function, the problem of insufficient suppression of processor temperature peaks in existing technologies is solved, achieving effective reduction of processor temperature and improvement of system reliability, adapting to the real-time requirements of different architectures and complex task flows.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WENZHOU UNIV
- Filing Date
- 2026-04-02
- Publication Date
- 2026-07-07
AI Technical Summary
Existing thermal management technologies rely on precise thermodynamic models, which are insufficient to cope with dynamic changes in the processor environment and sudden changes in task flow, resulting in insufficient suppression of temperature peaks. Furthermore, traditional algorithm optimization strategies are conservative and cannot achieve optimal results in terms of real-time performance and temperature control.
We adopt a deep reinforcement learning-based approach, construct a deep reinforcement learning agent by establishing a hard real-time constraint model based on real-time calculus, and use a hybrid reward function to guide the agent's iterative learning to generate the optimal cooling strategy that satisfies the hard real-time constraints, thus abandoning the traditional conservative approximation method.
It effectively suppresses local hot spots in multi-core processors, avoids degradation of system reliability, reduces processor peak temperature, adapts to processors with different architectures, handles uncertainties in complex task flows, and meets hard real-time constraints.
Smart Images
Figure CN122346237A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multi-core processor technology, and more specifically, to a method and system for optimizing processor peak temperature based on deep reinforcement learning. Background Technology
[0002] In recent years, multi-core processors, with their powerful parallel computing capabilities, have become core components of high-performance computing systems. However, with the increasing integration of chips, power consumption density has grown exponentially, making processor temperature issues increasingly serious. According to chantem et al., a temperature difference of 10-15°C above the operating limit can significantly reduce device lifespan. At the same time, high temperatures can also degrade the performance of portable devices equipped with multi-core processor chips, affecting the user experience.
[0003] Existing thermal management technologies are mainly divided into Dynamic Voltage Frequency Scaling (DVFS) and Dynamic Power Management (DPM). DVFS reduces processor temperature by lowering clock frequency and power supply voltage, while DPM adjusts CPU power consumption to reduce temperature by switching between different power modes. While both control processor temperature, they also have their drawbacks. DVFS struggles to address static leakage power consumption caused by current leakage, and its voltage limitation is hardware-dependent, resulting in insufficient temperature peak smoothing. DPM often employs fixed threshold strategies or greedy algorithms, making local decisions based solely on the current system state, lacking global planning for future heat accumulation effects, and prone to short-sighted operations leading to subsequent temperature runaway. Furthermore, both technologies typically use simple mathematical or thermodynamic models. However, due to the complexity of the processor's operating environment and the heterogeneity of manufacturing processes, these pre-set models often fail to accurately match the actual physical system and struggle to cope with dynamic changes in the environment and task flow, especially the suddenness and uncertainty of task arrival. This leads to lag and bias in the control strategies' management of transient thermal behavior, making it impossible to minimize peak temperature while ensuring real-time performance. In summary, existing technologies have the following problems: 1. They rely on accurate thermodynamic models in hardware and lack adaptive capabilities, making it difficult to cope with dynamic changes in the environment and task flow, especially the suddenness and uncertainty of task arrival, and insufficient suppression of temperature peaks; 2. Traditional algorithms use bounded delay functions to approximate the service curve to optimize the PTM strategy, making its optimization strategy biased towards conservatism. Summary of the Invention
[0004] The purpose of this invention is to overcome the problem that existing technologies rely too much on accurate modeling and that hard real-time constraint modeling is too conservative, resulting in poor cooling effects. This invention provides a processor peak temperature optimization method and system based on deep reinforcement learning, which effectively reduces the processor peak temperature.
[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for optimizing processor peak temperature based on deep reinforcement learning is provided, comprising the following steps: S1. Establish a hard real-time constraint model based on real-time calculus; S2. Construct a deep reinforcement learning agent based on a hard real-time constraint model, mapping the core parameters of the processor to the state space; and output the control strategy for periodic thermal management as an action through the deep reinforcement learning agent. S3. Design a hybrid reward function to guide the deep reinforcement learning agent to perform iterative learning, and finally generate the optimal cooling strategy that satisfies hard real-time constraints.
[0006] This invention presents a processor peak temperature optimization method based on deep reinforcement learning. It models the real-time performance of the system using real-time calculus theory, abandoning conservative approximation methods and enabling the handling of complex task flows with uncertainties while strictly meeting hard real-time constraints. By introducing a hybrid reward function, it effectively suppresses local hotspots in multi-core processors, avoiding system reliability degradation due to overheating of a single core. The deep reinforcement learning agent built based on the hard real-time constraint model can adapt to processors with different architectures without requiring a precise physical thermal model. In summary, the method of this invention can effectively reduce the peak temperature of the processor.
[0007] Furthermore, the hard real-time constraint model based on real-time calculus in step S1 is represented as follows:
[0008]
[0009]
[0010]
[0011] In the formula, and This represents the periodic thermal management scheme vector for each core of the processor, and its internal elements represent the duration of each core's active and sleep states, respectively. It is the time overhead required for the processor core to switch from "sleep state" to "active state". It is the time overhead required for a processor core to switch from an "active state" to a "sleep state". This represents the peak temperature of each core within one sampling period. This represents the aggregated service curve of the processor system based on event modeling. Indicates the arrival status of the application stream. This indicates the relative deadline for the application flow, set according to the designer or the characteristics of the task.
[0012] Furthermore, in step S1, an upper arrival curve is introduced. To abstractly describe the task flow at any time interval Maximum possible load; utilizing the lower service curve To characterize the minimum computing resources that a processor core can provide within a corresponding time interval; then for a given upper arrival curve Constraints and relative deadlines are Regarding the event flow, if the system's service curve If the following conditions are met, the system can guarantee that the delay of any event is no greater than D:
[0013] Then, the mathematical expression of hard real-time constraints is defined as follows:
[0014] in It is an aggregated service curve that aggregates all cores and is modeled based on events by the processor system. This aggregated service curve consists of a set of periodic thermal management strategy vectors for each core of the processor. and Sure; Finally, based on the mathematical expression of the above hard real-time constraints, a hard real-time constraint model for real-time calculus is established.
[0015] Furthermore, step S2 specifically includes the following steps: S21. Define the state space of the deep reinforcement learning agent. Configure core parameters of the processor; core parameters include processor core peak temperature. Processor core temperature variation Number of processor mode switches Delay factor The specific meaning is as follows:
[0016] S22. Define the feasible domain of the action space for the deep reinforcement learning agent; Define action space Represented as: ; Lower limit of action space Set as: ; Indicates the state transition time; Upper limit of motion space Set as: ; S23. Post-process the policy output of the deep reinforcement agent to ensure... If the integer is used, the post-processing procedure is as follows: .
[0017] Furthermore, in step S21, regarding the peak temperature of the processor core... Defined as:
[0018] In the formula, This indicates that within [0,t], the first... i The peak temperature of each core; for Defined as:
[0019] In the formula, This represents the average temperature of the processor core during the first t' time period of the sampling period; This represents the average temperature of the processor core during the time period t', where the value of t' is determined by the designer or the conditions of device use. Number of processor mode switches Select the number of times the processor mode is switched within a sampling period. As a state feature of the processor's current state, it is represented as:
[0020] For delay factor Defined as: .
[0021] Furthermore, the hybrid reward function includes a temperature optimization reward item. , Rewards for Violating Constraints Core Balance Rewards Limit-approaching bonus items ;in: Temperature optimization reward items Represented as: ; In the formula, and Indicates weight, Indicates the maximum peak temperature The reward; Indicates the average peak temperature The reward; Rewards for Violating Constraints Represented as:
[0022] In the formula, The constraint out-of-bounds degree is represented by the difference in the ordinate of the first intersection point between the lower service curve and the upper arrival curve; Core Balance Rewards Represented as:
[0023] In the formula, and These represent the core peak temperature. The variance and range; Limit approaching the reward item Represented as:
[0024] Among them, according to the relative deadline Design the integration interval and calculate using the following formula. :
[0025] By designing a baseline service curve to achieve The normalization of the baseline service curve is as follows: Sure, This represents the interval integral value of the reference service curve.
[0026] Furthermore, regarding the maximum peak temperature Rewards Calculated in the following way:
[0027] For average peak temperature Rewards Calculated in the following way: .
[0028] This invention also provides a processor peak temperature optimization system based on deep reinforcement learning, comprising: Hard real-time constraint model building module: used to build hard real-time constraint models based on real-time calculus; The Deep Reinforcement Learning Agent Construction Module is used to construct deep reinforcement learning agents based on hard real-time constraint models, mapping the core parameters of the processor to the state space; and outputting periodic thermal management control strategies as actions through the deep reinforcement learning agent. Hybrid reward function module: Used to design hybrid reward functions to guide deep reinforcement learning agents in iterative learning, and finally generate the optimal cooling strategy that satisfies hard real-time constraints.
[0029] Furthermore, the hard real-time constraint model based on real-time calculus established by the real-time constraint model construction module is represented as follows:
[0030]
[0031]
[0032]
[0033] In the formula, and This represents the periodic thermal management scheme vector for each core of the processor, and its internal elements represent the duration of each core's active and sleep states, respectively. It is the time overhead required for the processor core to switch from "sleep state" to "active state". It is the time overhead required for a processor core to switch from an "active state" to a "sleep state". This represents the peak temperature of each core within one sampling period. This represents the aggregated service curve of the processor system based on event modeling. Indicates the arrival status of the application stream. This indicates the relative deadline for the application flow, set according to the designer or the characteristics of the task.
[0034] Furthermore, the deep reinforcement learning agent construction module performs the following steps during the agent construction process: The state space of a deep reinforcement learning agent Configure core parameters of the processor; core parameters include processor core peak temperature. Processor core temperature variation Number of processor mode switches Delay factor The specific meaning is as follows:
[0035] Define the feasible domain of the action space for a deep reinforcement learning agent; Define action space Represented as: ; Lower limit of action space Set as: ; Indicates the state transition time; Upper limit of motion space Set as: ; Post-processing is performed on the strategy output of the deep reinforcement agent to ensure... If the integer is used, the post-processing procedure is as follows: .
[0036] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the processor peak temperature optimization method based on deep reinforcement learning described above.
[0037] Compared with the prior art, the beneficial effects of the present invention are: This invention presents a processor peak temperature optimization method based on deep reinforcement learning. It abandons conservative approximation methods, can handle complex task flows with uncertainties, and strictly meets hard real-time constraints. By introducing a hybrid reward function, it effectively suppresses local hotspots in multi-core processors, avoiding system reliability degradation due to overheating of a single core. This invention can adapt to processors with different architectures without requiring a precise physical thermal model. The method of this invention maximizes processor sleep time, effectively reducing the processor's peak temperature. Attached Figure Description
[0038] Figure 1 This is a flowchart illustrating a processor peak temperature optimization method based on deep reinforcement learning in one embodiment. Figure 2 This is a schematic diagram of the service curve and arrival curve when the real-time constraints are met in one embodiment. Figure 3 This is a schematic diagram illustrating the periodic change of processor temperature over time in one embodiment; Figure 4 This is a schematic diagram of the curve spacing integral value in the limit approximation reward in another embodiment; Figure 5 This is an example of agent reward curves under different real-time constraints in one embodiment. Figure 6 This represents the real-time constraint completion status at the maximum delay in one embodiment. Figure 7 This is a comparison of peak temperature values under different methods in one embodiment. Detailed Implementation
[0039] The present invention will be further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only, representing schematic diagrams rather than actual physical objects, and should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0040] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0041] Example 1 This embodiment is a first embodiment of a processor peak temperature optimization method based on deep reinforcement learning. In this embodiment, the traditional bounded delay function is abandoned, and linear approximation is no longer used. Instead, modeling is performed using the TDMA service curve. Then, a deep reinforcement learning agent is constructed to map the processor's temperature state, mode switching frequency, etc., into the state space. The deep reinforcement learning agent outputs a periodic thermal management control strategy (activity time). and hibernation time The action is defined as follows: Finally, a hybrid reward function is designed, incorporating temperature optimization, constraint out-of-bounds penalties, and limit approximation, to guide the agent's iterative learning and ultimately generate the optimal cooling strategy that satisfies hard real-time constraints. For example... Figure 1 As shown, the specific steps include: Step S1. Establish a hard real-time constraint model based on real-time calculus.
[0042] This embodiment models the system's real-time performance based on real-time calculus theory. To quantify the matching relationship between task load and processing capacity, this embodiment introduces an upper arrival curve. To abstractly describe the task flow at any time interval The maximum load that can be reached within the range; at the same time, utilizing the lower service curve. This characterizes the minimum computing resources (i.e., service capacity) that a processor core can provide within a corresponding time interval.
[0043] Then for a curve that is affected by the upper arrival curve Constraints and relative deadlines are Regarding the event flow, if the system's service curve If the following conditions are met, the system can guarantee that the delay of any event is no greater than D:
[0044] When optimizing peak temperature within a deep reinforcement learning framework, this embodiment employs a more stringent strategy compared to the bounded delay function used in traditional methods, further reducing the processor's peak temperature. Therefore, in this embodiment, the mathematical representation of the defined real-time constraints (Real-Time Calculus (RTC) framework) is as follows:
[0045] in It is an aggregated service curve that combines all cores and is modeled based on events by the processor system. This curve consists of a set of periodic thermal management strategy vectors for each core of the processor. and Determined. It characterizes the time interval at any given time interval. Internally, the processor system can provide the minimum cumulative computing resources to the task flow when executing specific periodic thermal management strategies. For example... Figure 2 As shown, it gives and Explanation of the relationship.
[0046] In summary, this embodiment proposes a model for optimizing processor peak temperature under hard real-time constraints, and its mathematical expression is as follows:
[0047] in and This represents the periodic thermal management scheme vector for each core of the processor. It is the time overhead required for a processor core to switch from a "sleep" state to an "active" state. This refers to the time overhead required for the processor core to switch from an "active state" to a "sleep state". This represents the peak temperature of each core within one sampling period. This represents the aggregated service curve of the processor system based on event modeling. Indicates the arrival status of the application stream. This indicates the relative deadline for the application flow, set according to the designer or the characteristics of the task.
[0048] Step S2. Construction of the Deep Reinforcement Learning (DRL) agent framework.
[0049] S21. Setting up the state space The state space of a deep reinforcement learning agent Configure core parameters of the processor; core parameters include processor core peak temperature. Processor core temperature variation Number of processor mode switches Delay factor The specific meaning is as follows:
[0050] The meanings of its various characteristics are as follows: (1) Peak temperature of processor core : Considering that the processor model has n cores, therefore... The definition is as follows:
[0051] in This indicates that within [0,t], the first... The peak temperature of each core.
[0052] (2) : To ensure To ensure the authenticity of the data and effectively utilize the experimental data, The definition is as follows:
[0053] in This represents the average temperature of the processor core during the first t' time period of the sampling period. This represents the average temperature over the time interval t', while the value of t' can be customized according to the designer or the equipment's operating conditions. The final form is as follows: Figure 3 As shown.
[0054] (3) Number of processor mode switches : The duration of the strategy is closely related to temperature. To avoid directly applying the strategy, we choose to measure the number of processor mode switches within a sampling period. This serves as a state characteristic of the processor's current state.
[0055]
[0056] (4) Delay factor : Considering the overhead of processor state switching The processor is unable to process tasks for a total of [time]. Therefore, the delay factor Defined as follows:
[0057] S22. Motion Space Setting (1) Feasible domain of action space Defined action space for:
[0058] Its lower limit Set as:
[0059] Its upper limit Set as:
[0060] S33. Strategy post-processing; To meet the requirements of the RTC model in generating service curves, the strategy output by the network is post-processed to ensure... Since it is an integer, we perform the following processing:
[0061] After processing The value is larger than the original value, meaning the delay factor of the processed strategy is larger. A larger value translates to a more aggressive service curve after the processed strategy. Therefore, the service curve generated by the processed strategy... When formula (2) is satisfied, the curve generated by the original strategy It can also be satisfied, and its mathematical form is as follows:
[0062] Step S3. Establishing a hybrid reward function guides the deep reinforcement learning agent to perform iterative learning, ultimately generating the optimal cooling strategy that satisfies hard real-time constraints.
[0063] Design a hybrid reward function Each reward can be weighted according to the designer's requirements or usage conditions to enhance the convergence speed of the agent. The following is a detailed design and mathematical expression of each reward item: (1) Temperature optimization reward items
[0064] A continuous reward design based on an exponential function is employed to design a system targeting the maximum peak temperature. Rewards and for average peak temperature Rewards The specific design is as follows:
[0065] Among them, targeting Rewards Specifically as follows:
[0066] For average peak temperature Rewards Specifically as follows:
[0067] in, , This indicates the highest temperature that the processor can theoretically reach, i.e., the maximum peak temperature. This represents the average peak temperature of the processor core.
[0068] (2) Rewards for violating constraints
[0069] When the processor fails to meet the constraints, the service curve and the arrival curve may intersect. Therefore, the difference in the ordinate of the first intersection point of the two curves is defined as the constraint violation degree. Obviously, The larger the value, the more severe the violation of the constraint. The design is as follows:
[0070] (3) Core Balance Rewards
[0071] This incentive aims to eliminate thermal imbalances between cores and prevent system reliability degradation caused by overheating of a single core. The design is as follows:
[0072] in, and Core peak temperature The variance and range. Variance reflects the degree of dispersion of temperature values; the larger the variance, the greater the difference between the peak temperatures at each core and the average peak temperature. The greater the deviation, the more uneven the temperature distribution; the range can reflect the maximum difference between the core peak temperatures. The larger the range, the more likely a core is in an overloaded or extremely idle state.
[0073] (4) Limit Approach Bonus
[0074] To maximize sleep time while satisfying constraints, the service curve should be as close as possible to the arrival curve. The smaller the area between them, the closer the provided resources are to the lower limit of task requirements, meaning the strategy is more "extreme." Figure 4 As shown, an integral value for the interval between index curves is proposed. Used to describe how closely the service curve and the arrival curve are close, it represents the area between the service curve and the arrival curve.
[0075] Furthermore, due to the constraint of exceeding the limit... At this point, the service curve will always be above the curve. If an integration interval is not defined, then... This would be an infinitely large number, unsuitable for designing a reward function. Therefore, based on the relative deadline... The design integration interval is as follows: ,then The calculation formula is as follows:
[0076] Considering This involves a very large amount of data. This embodiment designs a baseline service curve to achieve this. The normalization. This baseline service curve is formed by... It is determined that the integral value of its curve spacing is determined by This indicates that the limit is approaching the reward item. It can be expressed by the following formula:
[0077] This embodiment provides a processor peak temperature optimization method based on deep reinforcement learning. First, addressing the model dependency and burst load issues in existing technologies, it utilizes an agent-environment interaction adaptive learning approach. This allows it to adapt to the suddenness and uncertainty of task arrival without requiring a precise physical thermal model, enabling it to formulate effective optimization strategies in complex environments. Second, addressing the conservative nature of real-time analysis in existing technologies, it abandons the traditional "bounded delay function (BDF)" and directly models constraints by constructing a TDMA aggregated service curve. This curve accurately describes resource supply under periodic thermal management, achieving a limit approximation of the arrival curve by the service curve. This allows for more aggressive strategies without violating hard real-time constraints, thus better reducing processor peak temperature. In summary, the method in this embodiment achieves the following significant advancements: 1. Significant cooling effect: Compared with traditional methods based on bounded delay function (BDF) (such as the fast greedy bounded descent algorithm), the method provided in this embodiment maximizes the processor sleep time by approximating the arrival curve by the limit. Experiments have shown that the peak temperature can be reduced by 1-6℃.
[0078] 2. Real-time performance is guaranteed: By directly modeling using aggregated service curves, the conservative approximation method is abandoned, which can handle complex task flows with uncertainties (such as H.263 decoding and MP3 decoding) and strictly meet hard real-time constraints.
[0079] 3. Achieves core thermal balance: The method in this embodiment effectively suppresses local hot spots in multi-core processors by introducing a core balance reward, thus avoiding a decrease in system reliability caused by overheating of a single core.
[0080] 4. Model independence and generalization: The method in this embodiment is based on model-free deep reinforcement learning, which can adapt to different processor architectures (such as ARM and Intel) without the need for an accurate physical thermal model, and the online fine-tuning mechanism enables it to quickly adapt to environmental changes.
[0081] Example 2 This embodiment is a second embodiment of a processor peak temperature optimization method based on deep reinforcement learning. This embodiment is similar to the first embodiment, except that a comparative embodiment is given for the method proposed in the first embodiment to further illustrate the advantages of the method provided in the first embodiment.
[0082] In this embodiment, the simulation model used is a quad-core processor based on the ARM architecture, which is used as the experimental object in this embodiment to highlight the effectiveness of the framework established by the present invention.
[0083] First, Table 1 lists the relevant parameters of the deep reinforcement learning algorithm in this embodiment.
[0084] Table 1. Relevant parameters of the real-time constraint experiment.
[0085] In this embodiment, the classic PJD timing model is used to generate the application stream, and the processor peak temperature optimization method based on deep reinforcement learning (hereinafter referred to as ALPTM (Auto-Learning Periodic Thermal Management)) of this invention is used to optimize the temperature for three different tasks: (1) H.263 decoder, (2) MP3 decoder, and (3) MAD player. Their worst-case execution time Listed in Table 2. In addition, a jitter factor was added based on the characteristics of the timing model. This is used to increase the uncertainty of task arrival, for application streams. In terms of its vibration 。
[0086] Table 2 Worst-case execution time and cycle of application flow
[0087] like Figure 5The diagram illustrates the relationship between agent reward and round number for temperature optimization under real-time constraints in three application flows. It's clear that the agent meets the real-time constraints for all tasks. While there are some fluctuations, the overall learning performance and convergence are satisfactory. This demonstrates that the proposed ALPTM method remains applicable under real-time constraints and adapts to different application flows, significantly expanding its scope. However, inconsistent convergence across tasks is observed. Data analysis suggests this is due to an unbalanced reward function weight distribution. Different application flows employ different optimal strategies, leading to inconsistent convergence values and consequently, varying convergence outcomes.
[0088] For the trained agent, validation experiments were conducted according to the validation algorithms in Table 3. Several algorithms in the PTM method for solving real-time constraint problems were compared, including Fast Bounded Greedy Descending (FBGD), Accurate Neighbor Simulated Annealing (ANSA), and brute-force searching (BS). The advantages of the proposed ALPTM method were verified through analysis of various aspects, including processor real-time constraint completion and processor peak temperature optimization. The experimental results and related analysis will be presented next.
[0089] Table 3 PTM Strategy Verification Algorithm
[0090] like Figure 6 The figures illustrate the completion of real-time constraints when processor latency is at its maximum. These figures show that the policy output by the ALPTM method satisfies the requirements of the processor temperature optimization problem under real-time constraints. This indicates that the policy output by the agent does not violate real-time constraints during thermal management and can provide real-time guarantees. (Bounded delay function) The curve is directly tangent to the arrival curve, which means that its actual TDMA service curve is more conservative compared to the ALPTM method, resulting in a higher peak temperature in the processor. This is because a more conservative strategy reduces the time the processor spends without being able to process tasks, leading to a larger proportion of time the processor is effectively processing tasks, thus causing a rise in peak temperature.
[0091] like Figure 7The figure illustrates how the processor searches for the optimal strategy using different methods during thermal management. From the figure, we can see that when... At that time, the strategies obtained by all algorithms were better than those obtained by all algorithms. At that time, the temperature should be low. This is because of the shaking. This indicates the burstiness and nondeterminism of tasks in the application stream. As jitter increases, task arrival times vary more significantly, exhibiting higher burstiness. This forces the processor to operate under high load for extended periods under real-time constraints, leading to an increase in peak temperature. Furthermore, it can be seen that the ALPTM method proposed in this invention is superior to existing methods, reducing processor peak temperature by 1-6°C. This is attributed to abandoning the traditional finite delay function and adopting a more extreme TMDA service curve.
[0092] Example 3 This embodiment is an example of a processor peak temperature optimization system based on deep reinforcement learning. This embodiment is similar to Embodiment 1, except that it provides a processor peak temperature optimization system based on deep reinforcement learning, comprising: Hard real-time constraint model building module: used to build hard real-time constraint models based on real-time calculus; The Deep Reinforcement Learning Agent Construction Module is used to construct deep reinforcement learning agents based on hard real-time constraint models, mapping the core parameters of the processor to the state space; and outputting periodic thermal management control strategies as actions through the deep reinforcement learning agent. Hybrid reward function module: Used to design hybrid reward functions to guide deep reinforcement learning agents in iterative learning, and finally generate the optimal cooling strategy that satisfies hard real-time constraints.
[0093] In this embodiment, the hard real-time constraint model based on real-time calculus established by the hard real-time constraint model construction module is represented as follows:
[0094] In the formula, and This represents the periodic thermal management scheme vector for each core of the processor. It is the time overhead required for the processor core to switch from "sleep state" to "active state". It is the time overhead required for a processor core to switch from an "active state" to a "sleep state". This represents the peak temperature of each core within one sampling period. This represents the aggregated service curve of the processor system based on event modeling. Indicates the arrival status of the application stream. This indicates the relative deadline for the application flow, set according to the designer or the characteristics of the task.
[0095] In this embodiment, the deep reinforcement learning agent construction module performs the following steps during the agent construction process: The state space of a deep reinforcement learning agent Configure core parameters of the processor; core parameters include processor core peak temperature. Processor core temperature variation Number of processor mode switches Delay factor The specific meaning is as follows:
[0096] Define the feasible domain of the action space for a deep reinforcement learning agent; Define action space Represented as: ; Lower limit of action space Set as: ; Upper limit of motion space Set as: ; Post-processing is performed on the strategy output of the deep reinforcement agent to ensure... If the integer is used, the post-processing procedure is as follows: .
[0097] In this embodiment, the hybrid reward function includes a temperature optimization reward term. , Rewards for Violating Constraints Core Balance Rewards Limit-approaching bonus items ;in: Temperature optimization reward items Represented as: ; In the formula, and Indicates weight, Indicates the maximum peak temperature The reward; Indicates the average peak temperature The reward; Rewards for Violating Constraints Represented as:
[0098] In the formula, The constraint out-of-bounds degree is represented by the difference in the ordinate of the first intersection point between the lower service curve and the upper arrival curve; Core Balance Rewards Represented as:
[0099] In the formula, and These represent the core peak temperature. The variance and range; Limit approaching the reward item Represented as:
[0100] Among them, according to the relative deadline Design the integration interval and calculate using the following formula. :
[0101] By designing a baseline service curve to achieve The normalization of the baseline service curve is as follows: Sure, This represents the interval integral value of the reference service curve.
[0102] Furthermore, regarding the maximum peak temperature Rewards Calculated in the following way:
[0103] For average peak temperature Rewards Calculated in the following way: .
[0104] Example 4 This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the processor peak temperature optimization method based on deep reinforcement learning described in Embodiment 1.
[0105] In the specific implementation of the above embodiments, the technical features can be combined in any non-contradictory way. For the sake of brevity, not all possible combinations of the above technical features are described. However, as long as the combination of these technical features is not contradictory, it should be considered to be within the scope of this specification.
[0106] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A processor peak temperature optimization method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Establish a hard real-time constraint model based on real-time calculus; S2. Construct a deep reinforcement learning agent based on a hard real-time constraint model, mapping the core parameters of the processor to the state space; and output the control strategy for periodic thermal management as an action through the deep reinforcement learning agent. S3. Design a hybrid reward function to guide the deep reinforcement learning agent to perform iterative learning, and finally generate the optimal cooling strategy that satisfies hard real-time constraints.
2. The processor peak temperature optimization method based on deep reinforcement learning according to claim 1, characterized in that, The hard real-time constraint model based on real-time calculus in step S1 is represented as follows: In the formula, and This represents the periodic thermal management scheme vector for each core of the processor. It is the time overhead required for the processor core to switch from "sleep state" to "active state". It is the time overhead required for the processor core to switch from "active state" to "sleep state". This represents the peak temperature of each core within one sampling period. This represents the aggregated service curve of the processor system based on event modeling. Indicates the arrival status of the application stream. This indicates the relative deadline for the application flow, set according to the designer or the characteristics of the task.
3. The processor peak temperature optimization method based on deep reinforcement learning according to claim 2, characterized in that, In step S1, the upper arrival curve is introduced. To abstractly describe the task flow at any time interval Maximum possible load; utilizing the lower service curve To characterize the minimum computing resources that a processor core can provide within a corresponding time interval; then for a given upper arrival curve Constraints and relative deadlines are Regarding the event flow, if the system's service curve If the following conditions are met, the system can guarantee that the delay of any event is no greater than D: Then, the mathematical expression of hard real-time constraints is defined as follows: in It is an aggregated service curve that aggregates all cores and is modeled based on events by the processor system. This aggregated service curve consists of a set of periodic thermal management strategy vectors for each core of the processor. and Sure; Finally, based on the mathematical expression of the above hard real-time constraints, a hard real-time constraint model for real-time calculus is established.
4. The processor peak temperature optimization method based on deep reinforcement learning according to claim 3, characterized in that, Step S2 specifically includes the following steps: S21. Define the state space of the deep reinforcement learning agent. Configure core parameters of the processor; core parameters include processor core peak temperature. Processor core temperature changes Number of processor mode switches Delay factor The specific meaning is as follows: S22. Define the feasible domain of the action space for the deep reinforcement learning agent; Define action space Represented as: ; Lower limit of action space Set as: ; Indicates the state transition time; Upper limit of motion space Set as: ; S23. Post-process the policy output of the deep reinforcement agent to ensure... If the integer is used, the post-processing procedure is as follows: .
5. The processor peak temperature optimization method based on deep reinforcement learning according to claim 4, characterized in that, In step S21, regarding the peak temperature of the processor core... Defined as: In the formula, This indicates that within [0,t], the first... i The peak temperature of each core; for Defined as: In the formula, This represents the average temperature of the processor core during the first t' time period of the sampling period; This represents the average temperature of the processor core during the time period t', where the value of t' is determined by the designer or the conditions of device use. Number of processor mode switches Select the number of times the processor mode is switched within a sampling period. As a state feature of the processor's current state, it is represented as: For delay factor Defined as: 。 6. The processor peak temperature optimization method based on deep reinforcement learning according to claim 3, characterized in that, In step S3, the hybrid reward function includes a temperature optimization reward term. , Rewards for Violating Constraints Core Balance Rewards Limit-approaching bonus items ;in: Temperature optimization reward items Represented as: ； In the formula, and Indicates weight, Indicates the maximum peak temperature The reward; Indicates the average peak temperature The reward; Rewards for Violating Constraints Represented as: In the formula, The constraint out-of-bounds degree is represented by the difference in the ordinate of the first intersection point between the lower service curve and the upper arrival curve; Core Balance Rewards Represented as: In the formula, and These represent the core peak temperature. The variance and range; Limit approaching the reward item Represented as: Among them, according to the relative deadline Design the integration interval and calculate using the following formula. : By designing a baseline service curve to achieve The normalization of the baseline service curve is as follows: Sure, This represents the interval integral value of the reference service curve.
7. The processor peak temperature optimization method based on deep reinforcement learning according to claim 6, characterized in that, For the maximum peak temperature Rewards Calculated in the following way: For average peak temperature Rewards Calculated in the following way: 。 8. A processor peak temperature optimization system based on deep reinforcement learning, characterized in that, include: Hard real-time constraint model building module: used to build hard real-time constraint models based on real-time calculus; The Deep Reinforcement Learning Agent Construction Module is used to construct deep reinforcement learning agents based on hard real-time constraint models, mapping the core parameters of the processor to the state space; and outputting periodic thermal management control strategies as actions through the deep reinforcement learning agent. Hybrid reward function module: Used to design hybrid reward functions to guide deep reinforcement learning agents in iterative learning, and finally generate the optimal cooling strategy that satisfies hard real-time constraints.
9. The processor peak temperature optimization system based on deep reinforcement learning according to claim 8, characterized in that, The hard real-time constraint model based on real-time calculus established by the hard real-time constraint model construction module is represented as follows: In the formula, and This represents the periodic thermal management scheme vector for each core of the processor. It is the time overhead required for the processor core to switch from "sleep state" to "active state". It is the time overhead required for the processor core to switch from "active state" to "sleep state". This represents the peak temperature of each core within one sampling period. This represents the aggregated service curve of the processor system based on event modeling. Indicates the arrival status of the application stream. This indicates the relative deadline for the application flow, set according to the designer or the characteristics of the task.
10. The processor peak temperature optimization system based on deep reinforcement learning according to claim 8, characterized in that, The deep reinforcement learning agent construction module performs the following steps during the agent construction process: The state space of a deep reinforcement learning agent Configure core parameters of the processor; core parameters include processor core peak temperature. Processor core temperature changes Number of processor mode switches Delay factor The specific meaning is as follows: Define the feasible domain of the action space for a deep reinforcement learning agent; Define action space Represented as: ; Lower limit of action space Set as: ; Indicates the state transition time; Upper limit of motion space Set as: ; Post-processing is performed on the strategy output of the deep reinforcement agent to ensure... If the integer is used, the post-processing procedure is as follows: .