A multi-agent thermal management method for an isomorphic dark silicon multi-core chip

Abstract

The application belongs to the field of integrated circuit design and thermal management, and proposes a multi-core chip voltage frequency adjustment and core gating optimization method based on reinforcement learning. The method realizes the optimization of temperature and system performance under the condition that the power consumption does not exceed the budget by dynamically adjusting the voltage, frequency and core gating state of the chip core under the COMET framework. The core state is collected by the multi-core chip simulation platform (SNIPER), and the state dataset is generated by energy consumption analysis (MCPAT) and thermal distribution calculation (hotspot). Combined with the Actor-Critic reinforcement learning network, the optimization strategy is dynamically output. Actor is responsible for generating adjustment strategy, and Critic evaluates its long-term value, and gradually optimizes the decision model. After embedding in the COMET framework, the method realizes the integration of data collection, model training and performance optimization, significantly improves the performance, power consumption and temperature control effect of the multi-core chip, and has wide adaptability and high efficiency.

Description

Technical Field

[0001] This invention relates to the fields of integrated circuit design and thermal management, and particularly to the optimization of temperature, power consumption, and performance of dark silicon multi-core processors. More specifically, this invention pertains to a method for dynamic power consumption adjustment and hotspot management using multi-agent reinforcement learning techniques within a homogeneous multi-core chip architecture, employing the COMET framework. Background Technology

[0002] With the advancement of integrated circuit technology, multi-core processors have been widely used in computationally intensive tasks. While multi-core processors improve computing efficiency, they also bring challenges in power consumption and thermal management.

[0003] In today's most widely used homogeneous chip architecture, the challenge lies in ensuring that the overall power consumption of the chip does not exceed the power budget, while also ensuring that the chip's performance reaches its optimal level and that the temperature of each core does not become too high, thus preventing the chip from overheating.

[0004] Existing technologies such as Dynamic Voltage and Frequency Scaling (DVFS), while optimal for single-core chips, are often not the best power optimization solution for today's complex homogeneous multi-core chips. Another technology, core gating, can alleviate power consumption issues in multi-core chips to some extent, but if the frequency and voltage between cores are not consistent, the power consumption after core gating cannot be guaranteed to be optimal. In addition, there are other dynamic thermal management (DTM) methods, but most of these methods often ignore the interaction between cores and the overall power consumption limit, lacking a balance between global and local optimization, thus failing to achieve the best balance between performance, power, and thermal optimization at the system level. Summary of the Invention

[0005] The purpose of this invention is to provide a new technical solution for optimizing both global performance and power consumption as well as local chip temperature.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] (i) The dataset collection, model training, and model embedding are completed based on the COMET framework. These three tasks can be completed in one integrated manner without the need for additional simulation tools, making the process simple, efficient, integrated, and automated.

[0008] (ii) Multi-agent reinforcement learning method using MADDPG algorithm.

[0009] First, let's model the problem:

[0010] 1. State: (s) t )

[0011] For the status information of each core:

[0012] (1) Current frequency (f)

[0013] (2) Current temperature (temp)

[0014] (3) Current voltage (V)

[0015] (4) Current power consumption

[0016] (5) Current nuclear gating (G)

[0017] For global state information:

[0018] (1) Total power consumption of the chip now )

[0019] 2. Action: (a) t )

[0020] (1) Voltage and frequency adjustment

[0021] (2) Nuclear gating

[0022] 3. Rewards (r) t )

[0023] (1) Performance Bonus (r) perf )

[0024]

[0025] Among them, power now This indicates the current power consumption of the chip. budget This represents the chip's power budget, and the reward indicates the state after the action: the chip's power consumption should be as high as possible within the power budget constraints in order to increase the chip's performance.

[0026] (2) Temperature threshold reward (r) temp )

[0027]

[0028] Among them, temp now The temperature of the core in the current state, temp threshold This represents the allowed threshold temperature of the nucleus; the reward indicates that the temperature of the state after the action should be as close as possible to the temperature threshold (tempthreshold). d The better.

[0029] (3) Core gating optimization (r gating )

[0030]

[0031] Where N represents the total number of cores, G i This indicates the switch of the core gating (0 for off, 1 for on). This reward suggests that the core should be turned on or off as little as possible to reduce the increase in power consumption during switching.

[0032] r t =w1*r perf +w2*r temp +w3*r gating

[0033] Here, w1, w2, and w3 refer to weights, which control the priority of different objectives.

[0034] 4. Objective:

[0035] Given the number of core gates and the current state of each core (core frequency, voltage), the goal is to make the temperature of each core as close as possible to the temperature threshold, and to achieve the highest possible performance while keeping the chip's power consumption within the power budget.

[0036] Secondly, the design of the multi-agent reinforcement learning algorithm framework:

[0037] Using MADDPG (Multi-Agent Reinforcement Learning Algorithm), a centralized training and distributed execution framework is employed, allowing each agent to use an independent policy while optimizing the policy by sharing global information.

[0038] 1. Actor-Critic framework:

[0039] (1) Each agent, that is, each kernel, has an independent Actor, which is used to generate actions based on the local state.

[0040] (2) Centralized Critic shares the global state and the actions of all agents to optimize the global value function.

[0041] 2. Centralized training, distributed execution

[0042] (1) Training phase: Critic optimizes the strategy using global state and all actions.

[0043] (2) Execution phase: Actors make independent decisions without needing to access global information. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the specific implementation of this invention.

[0046] Figure 2 This is a set of computational chip temperature distribution process frameworks referenced in the embodiments of the present invention, COMET.

[0047] Figure 3 This is a flowchart illustrating the specific implementation of COMET in this embodiment of the invention.

[0048] Figure 4 This is a flowchart illustrating the specific steps involved in training a reinforcement learning model in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The overall implementation process of this invention embodiment is as follows: Figure 1 As shown, it references a computational chip temperature distribution process framework called COMET, such as... Figure 2 As shown, it includes: the SNIPER module, which provides the chip with parameter information corresponding to the current state, including the chip's current frequency, voltage, core gating, etc.; the MCPAT module, which outputs the power consumption information of the current state based on the chip's current performance; the hotspot module, which outputs the power consumption information as chip temperature distribution information; and the DTM (Dynamic Temperature Thermal Management) module, which provides the performance indicators required for the chip to reach the temperature threshold in the current state based on the current chip's thermal temperature distribution and power consumption information. The COMET operation flow is as follows: Figure 3As shown, within a state cycle, the SNIPER module provides the performance index of the current state (step one), the MCPAT module calculates the power consumption information based on the performance index of the current state (step two), the hotspot module calculates the temperature distribution of the chip based on the power consumption information (step three), and the DTM provides the optimized performance index based on the power consumption and temperature distribution, and returns the performance index to the SNIPER module for updating, to obtain the performance index of the next state cycle (step four).

[0051] This invention mainly innovates on step four of the process, and the specific steps are as follows:

[0052] (a) The first step is the collection of the dataset, such as Figure 4 As shown:

[0053] 1. With a relatively long total time (T) total Using 1 as a benchmark, in each time period (T) consisting of steps 1, 2, and 3... one Within a given period (ignoring step four for now), change the core voltage, frequency, and core gating within each cycle to extend the total time as much as possible, and perform n calibration experiments.

[0054]

[0055] The specific steps are as follows.

[0056] 2. Use module one SNIPER to conduct a round of data calibration experiments, collect the voltage (V), frequency (f), and core gating information (G) of each core of the chip at this moment, as the current state s{V, f, G}.

[0057] 3. Calculate the power consumption and temperature distribution (temp) of the current state from the dataset obtained in step 2 using Module 2 (MCPAT) and Module 3 (hotspot), and add it to the current state s. t-1 {V, f, G, power, temp}.

[0058] 4. At this point, a time period (T) one (End of cycle) and begin the next time period.

[0059] 5. The control module SNlPER performs an action (a) t-1 This includes changing voltage (V), frequency (f), and nuclear gating information (G).

[0060] 6. Then perform the action made in step 5 (a) t Through steps two and three, the power consumption (power) and temperature distribution (temp) are obtained, and the state s at this time is obtained. t{V, f, G, power, temp}.

[0061] 7. At this point, regarding state s t-1 The action performed by {V, f, G, power, temp} (a t-1 ) Calculate the reward (r) t-1 )

[0062] 8. After completing all n calibration experiments, n complete datasets will be obtained. Store the datasets in the buffer (Replay Buffer).

[0063] (II) Training the Actor-Critic Network:

[0064] 1. Design the network:

[0065] (1) Actor Network:

[0066] Input: Current state of each core (s) t-1 ).

[0067] Hidden layer: Fully connected layer

[0068] Output: The action of each core in this current state (a t-1 ).

[0069] (2) Critic network:

[0070] Global state input: Current global state (s) t-1 ): This refers to the global power consumption and the state of each core.

[0071] Global action input: Accepts actions from the actor network as input.

[0072] Output Global Value: Distributed Q-value Decomposition

[0073]

[0074] Network structure: multi-layer fully connected network.

[0075] 2. Initialization of the Actor-Critic network.

[0076] 3. Randomly sample a batch of data from the Replay Buffer.

[0077] 4. Update the Actor-Critic network based on the sampled data from step 3.

[0078] 5. Repeat the training on the dataset until the preset accuracy value is reached.

[0079] (III) The trained reinforcement learning model is embedded into module four (DTM), and the entire COMET framework is completed.

Claims

1. A reinforcement learning-based method for voltage frequency regulation and core gating optimization of dark silicon multi-core chips, characterized in that, The method, within the COMET framework, optimizes system performance and temperature distribution by dynamically adjusting the chip core's voltage, frequency, and core gating state, while ensuring power consumption does not exceed the power budget. The method includes the following steps: (1) Over a relatively long total time Based on this, multiple time periods are divided. The dataset is calibrated, and each time period includes the following sub-steps: a) Collect the voltage of the chip core using the SNIPER multi-core chip simulation platform. ,frequency and nuclear gate information To form the current state ; b) Using the energy consumption analysis module MCPAT and the heat distribution calculation module hotspot, calculate the power consumption and temperature distribution of the current state, respectively, to expand the state information. ; c) Based on the current state via the Actor network Generate regulation strategies, including voltage adjustment ,frequency and nuclear gating information ; d) Based on the adjustment strategy The next state is generated through the simulation module and the analysis module. and reward value The result is then stored in the Replay Buffer. (2) Based on the Replay Buffer data in the buffer, the Actor-Critic network is trained, where: a) Actor network input current state Output adjustment strategy ; b) Critic network input status and actions Assess its long-term value ; c) By optimizing the reward function: Dynamically adjusting the Actor network output to improve system performance and energy efficiency: The reward function includes the following objectives: a) Performance Bonus This represents the improvement in system throughput or performance after adjusting the strategy; b) Nuclear gating rewards This means shutting down unnecessary cores as much as possible; c) Temperature reward This indicates the optimization effect on the highest temperature or temperature distribution uniformity.

2. A system suitable for dark silicon multi-core processors, characterized in that, The method for voltage and frequency regulation and core gating optimization of dark silicon multi-core chips based on reinforcement learning as described in claim 1 includes voltage and frequency regulation and core gating optimization of high-performance computing chips and embedded low-power chips.

3. A dynamic thermal management method (DTM) based on reinforcement learning, characterized in that, The method for voltage and frequency regulation and core gating optimization of dark silicon multi-core chips based on reinforcement learning as described in claim 1 is implemented.

4. A unified intelligent simulation tool that combines data acquisition, model training, and optimized execution, characterized in that: Within the COMET framework, the Dynamic Thermal Management Method (DTM) based on reinforcement learning as described in claim 3 is executed.

Images