Multi-agent thermal management method for isomorphic dark silicon multi-core chip

Abstract

The invention belongs to the field of integrated circuit design and thermal management, and provides a multi-core chip voltage frequency regulation and core gating optimization method based on reinforcement learning. According to the method, under a COMET framework, the voltage, the frequency and the core gating state of a chip core are dynamically adjusted, and the temperature and the system performance are optimized under the condition that the power consumption does not exceed the budget. A core state is collected by a multi-core chip simulation platform (SNIPER), a state data set is generated through energy consumption analysis (MCPAT) and heat distribution calculation (hotspot), an Actor-Critic reinforcement learning network is combined, and an optimization strategy is dynamically output. Actor is responsible for generating an adjusting strategy, Critic evaluates the long-term value of the adjusting strategy, and a decision model is optimized step by step. After the COMET framework is embedded, the method realizes integration of data collection, model training and performance optimization, remarkably improves the performance, power consumption and temperature control effect of the multi-core chip, and has wide adaptability and high efficiency.

Description

Technical Field

[0001] The present invention relates to the field of integrated circuit design and thermal management, and more particularly to temperature, power consumption and performance optimization of dark silicon multi-core processors. More specifically, the present invention relates to a method for implementing dynamic power consumption adjustment and hotspot management using a COMET framework in a homogeneous multi-core chip architecture using multi-agent reinforcement learning technology. Background Art

[0002] With the advancement of integrated circuit technology, multi-core processors have been widely used in computationally intensive tasks. Although multi-core processors improve computing efficiency, they also bring challenges in power consumption and thermal management.

[0003] In today's most widely used homogeneous chip architecture, how to ensure that the overall power consumption of the chip does not exceed the power budget while ensuring that the chip's performance is optimized and that the temperature of each core is not too high, causing the risk of chip overheating, has become a design challenge.

[0004] Existing technologies such as dynamic voltage frequency scaling (DVFS), although the best solution for single-core chips, are often not the best solution for power optimization for today's complex homogeneous multi-core chips. Another technology, core gating, can alleviate the power consumption problem of multi-core chips to a certain extent, but once the frequency and voltage between cores are not consistent, the power consumption after core gating cannot guarantee the optimal solution. In addition, there are some other dynamic thermal management methods (DTM), but most methods often ignore the mutual influence between cores and the overall power consumption limit, lack of global optimization and local optimization, resulting in the inability to achieve the best balance between performance, power and thermal optimization at the system level. Summary of the invention

[0005] The purpose of the present invention is to provide a new technical solution to optimize both global performance power consumption and local chip temperature.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] (I) Complete data set collection, model training, and model embedding based on the COMET framework. The three tasks can be completed in an integrated manner without the need for additional simulation tools, making the process simple, efficient, integrated, and automated.

[0008] (2) Multi-agent reinforcement learning method using the MADDPG algorithm.

[0009] First, model the problem:

[0010] 1. Status: (s t )

[0011] For each core's status information:

[0012] (1) Current frequency (f)

[0013] (2) Current temperature (temp)

[0014] (3) Current voltage (V)

[0015] (4) Current power consumption (power)

[0016] (5) Current nuclear gating (G)

[0017] For global status information:

[0018] (1) Total power consumption of the chip now )

[0019] 2. Action: (a t )

[0020] (1) Voltage and frequency adjustment

[0021] (2) Nuclear gating

[0022] 3. Reward (r t )

[0023] (1) Performance reward (r perf )

[0024]

[0025] Among them, power now Indicates the power consumption of the current chip, power budget Represents the power budget of the chip, and the reward indicates the state after the action: the power consumption of the chip should be as large as possible within the power budget to increase the performance of the chip.

[0026] (2) Temperature threshold reward (r temp )

[0027]

[0028] Among them, temp now Indicates the temperature of the core in the current state, temp threshold Represents the threshold temperature of the core allowed. The reward indicates that the temperature of the state after the action should be closer to the temperature threshold (tempthreshol d ) the better.

[0029] (3) Nuclear gating optimization (r gating )

[0030]

[0031] Where N represents the number of all cores, G i Represents the core gating switch (0 means off, 1 means on). This reward indicates that the core should be turned on or off as little as possible to reduce the increase in switching power consumption.

[0032] r t =w1*r perf +w2*r temp +w3*r gating

[0033] Among them, w1, w2, and w3 refer to weights, which control the priority of different targets.

[0034] 4. Objectives:

[0035] When the number of core gates and the current state of each core (core frequency, voltage) are obtained, the temperature of each core is made as close to the temperature threshold as possible, and the highest performance is achieved as much as possible without the power consumption of the chip exceeding the power budget.

[0036] Secondly, the multi-agent reinforcement learning algorithm framework design:

[0037] Using MADDPG (multi-agent reinforcement learning algorithm), a centralized training and distributed execution framework is adopted, allowing each agent to use independent strategies while optimizing the strategy by sharing global information.

[0038] 1. Actor-Critic Framework:

[0039] (1) Each agent, or each core, has an independent Actor, which is used to generate actions based on local states.

[0040] (2) The centralized critic shares the global state and actions of all agents to optimize the global value function.

[0041] 2. Centralized training, distributed execution

[0042] (1) Training phase: Critic optimizes the strategy using the global state and all actions.

[0043] (2) Execution phase: Actors make independent decisions without accessing global information. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 This is a specific implementation flow chart in an embodiment of the present invention.

[0046] Figure 2 It is a set of chip temperature distribution calculation process framework COMET referenced in the embodiments of the present invention;

[0047] Figure 3 It is a flowchart of the specific implementation of COMET in an embodiment of the present invention.

[0048] Figure 4 It is a flowchart of the specific steps when training the reinforcement learning model in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] The overall implementation process of the embodiment of the present invention is as follows: Figure 1 As shown, a set of chip temperature distribution process framework COMET is referenced, such as Figure 2 As shown, it includes: the SNIPER module is used to give the module to obtain the parameter information corresponding to the chip in the current state, including the frequency, voltage, core gating, etc. of the chip in the current state; the MCPAT module outputs the power consumption information of the current state according to the chip performance in the current state; the hotspot module outputs the power consumption information as the chip temperature distribution information; the DTM (dynamic temperature thermal management) module gives the performance indicators required for the current state chip to reach the temperature threshold according to the thermal temperature distribution and power consumption information of the current chip. The operation process of COMET is as follows Figure 3As shown, within a state cycle, the SNIPER module gives the performance indicators of the current state (step one), the MCPAT module calculates the power consumption information according to the performance indicators of the current state (step two), the hotspot module calculates the temperature distribution of the chip according to the power consumption information (step three), and the DTM gives the optimized performance indicators according to the power consumption and temperature distribution, and returns the performance indicators to the SNlPER module for updating, so as to obtain the performance indicators of the next state cycle (step four).

[0051] The present invention mainly innovates step 4 in the process, and the specific steps are as follows:

[0052] (I) First, the data set is collected, such as Figure 4 As shown:

[0053] 1. Take a longer total time (T total ) is a benchmark. In each time period (T one ) (step 4 is not considered for the time being), change the core voltage, frequency and core gating in each cycle, extend the total time as much as possible, and perform n calibration experiments

[0054]

[0055] The specific steps are as follows.

[0056] 2. Use module 1 SNIPER to conduct a round of data calibration experiments to collect the voltage (V), frequency (f), and core gating information (G) of each core of the chip at this moment as the current state s{V, f, G}.

[0057] 3. The data set obtained in 2 is used to calculate the power consumption (power) and temperature distribution (temp) of the current state through module 2 MCPAT and module 3 hotspot, and then added into the current state s t-1 {V, f, G, power, temp}.

[0058] 4. At this time, a time period (T one ) ends and enters the next time period.

[0059] 5. Control module SN1PER takes action (a t-1 ), including changes in voltage (V), frequency (f), and nuclear gating information (G).

[0060] 6. Then make the action made in 5 (a t ), through step 2 and step 3, the power consumption (power) and temperature distribution (temp) are obtained, and the state s at this time is obtained t{V, f, G, power, temp}.

[0061] 7. At this time, for state s t-1 {V, f, G, power, temp} makes the action (a t-1 ) Calculate the reward reward (r t-1 )

[0062] 8. After completing all n calibration experiments, n sets of complete data sets can be obtained and stored in the buffer (Replay Buffer).

[0063] (II) Training the Actor-Critic Network:

[0064] 1. Design network:

[0065] (1) Actor Network:

[0066] Input: The current state of each core (s t-1 ).

[0067] Hidden layer: fully connected layer

[0068] Output: The action of each core at this state (a t-1 ).

[0069] (2) Critic Network:

[0070] Global state input: current global state (s t-1 ): That is, the global power consumption and the status of each core.

[0071] Global action input: accepts actions from the actor network as input.

[0072] Outputting global value: Distributed Q-value decomposition

[0073]

[0074] Network structure: multi-layer fully connected network.

[0075] 2. Initialization of the Actor-Critic network.

[0076] 3. Randomly sample a batch of data from the buffer (Replay Buffer).

[0077] 4. Update the Actor-Critic network based on the data sampled in 3.

[0078] 5. Repeat the training data set until the pre-set accuracy value is reached.

[0079] (3) The trained reinforcement learning model is embedded into module 4 (DTM), and the entire COMET framework is constructed.

Claims

1. A dark silicon multi-core chip voltage frequency adjustment and core gating optimization method based on reinforcement learning, characterized in that: The method optimizes system performance and temperature distribution under the COMET framework by dynamically adjusting the voltage, frequency and core gating state of the chip core under the condition that the power consumption does not exceed the power budget. The method includes the following steps: (1) With a longer total time Tt otal As a benchmark, divide multiple time periods into To ne Calibrate the data set, each time period includes the following sub-steps: a) Collect the voltage (V), frequency (f) and core gating information (G) of the chip core through the multi-core chip simulation platform (SNlPER) to form the current state (s t-1 ); b) Use the energy consumption analysis module (MCPAT) and the heat distribution calculation module (hotspot) to calculate the power consumption and temperature distribution of the current state, respectively, and expand the state information (s t-1 ); c)According to the current state (s) through the Actor network t-1 ) Generates regulation strategies, including adjusting voltage (V), frequency (f), and nuclear gating information (G); d) According to the adjustment strategy (a t-1 ), generate the next state (s) through the simulation module and the analysis module t ) and reward value (r t-1 ), and store the result in the buffer (Replay Buffer); (2) Based on the data in the replay buffer, the Actor-Critic network is used for training, where: a) Actor network input current state (s t-1 ), output regulation strategy (a t-1 ); b) Critic network input state (s) t-1 ) and action (a t-1 ), evaluate its long-term value Q(s t-1 , a t-1 ); c) By optimizing the reward function r t-1 =w1*r perf +w2*r temp +w3*r gating Dynamically adjust Actor network output to improve system performance and energy efficiency.

2. The method according to claim 1, characterized in that The reward function includes the following objectives: (1) Performance reward r perf , represents the improvement in system throughput or performance after adjusting the strategy; (2) Core gated reward r gating , which means shutting down unnecessary cores as much as possible; (3) Temperature reward r temp , indicating the optimization effect of the maximum temperature or temperature distribution uniformity.

3. The method according to claim 1, characterized in that The method is applicable to dark silicon multi-core processor systems, including voltage frequency regulation and core gating optimization of high-performance computing chips and embedded low-power chips.

4. According to the method described in claim 1, a dynamic thermal management method (DTM) based on reinforcement learning method is obtained.

5. According to the method described in claim 4, it is embedded in the COMET framework to obtain an integrated intelligent thermal simulation tool that combines data acquisition, model training and optimization execution.

Images