Chip macro component placement method based on reinforcement learning for layout adjustment

Abstract

The application discloses a chip macro component placement method based on reinforcement learning for layout adjustment, uses reinforcement learning to learn how to adjust the existing layout instead of placing components from the beginning, thereby providing sufficient information for the strategy and obtaining relatively dense and accurate rewards. In addition, the method introduces the concept of regularity in the training process, which is considered as an important indicator in the chip design industry, but is often ignored in current reinforcement learning layout methods. The method is evaluated in the ICCAD 2015 benchmark test, and compared with several competitive methods, the half perimeter wire length and regularity can be improved. The power consumption, performance and area performance of the chip are tested by using commercial software, and the results show that the proposed layout adjustment method can realize significant PPA improvement. In addition, the method can fine-tune the layout of any method to improve its quality.

Description

Technical Field

[0001] This invention relates to a method for placing chip macro-components, and more particularly to a method for placing chip macro-components based on reinforcement learning. Background Technology

[0002] Electronic design automation (EDA) technology has been widely used to assist engineers in designing chips. However, the rapid development of very large-scale integration (VLSI) technology has led to an exponential increase in chip size, posing significant challenges to existing EDA techniques. This is particularly true for a crucial stage in the design flow: chip floorplanning. This stage aims to place a series of components on a chip canvas while adhering to constraints such as congestion and density, optimizing metrics like power consumption, performance, and area. The rapidly increasing number of components on the chip canvas and the increasingly complex routing relationships necessitate novel and efficient methods. In the placement phase of floorplanning, we need to determine the location of each component on the chip canvas. Modern chips typically contain thousands of macro-components (such as memory modules) and millions of standard cells (such as logic gates). In the chip design flow, upstream steps usually provide a netlist to the chip placement task, outlining the placement requirements. A netlist can be represented as a large-scale hypergraph containing many hyperedges representing the routing relationships between macro-components and standard components. Each hyperedge contains multiple components, indicating that these components need to be connected by a set of wires during placement. Component placement is typically performed in two phases sequentially: macro-component placement, usually solved using heuristic or learning-based methods; and standard component placement, usually solved using a solver. After all components (including macro-components and standard components) have been placed, the routing phase begins. The general flow of chip placement planning and routing is as follows: Figure 1 As shown.

[0003] Researching the placement of macro-components on chips can improve the technological level and competitiveness of China's chip industry, and promote industrial development. This is of great significance for alleviating the supply and demand imbalance in the domestic chip industry, reducing dependence on foreign countries, and achieving industrial self-reliance and control.

[0004] The macro-component placement problem is typically modeled as an optimization problem, which involves iterative evaluation to obtain better layout results. This optimization problem requires satisfying the non-overlapping constraint, meaning that there can be no overlapping parts between macro-components; otherwise, they cannot be used for actual manufacturing.

[0005] Reinforcement learning has recently emerged as a promising technique for improving the quality of macro-component placement. By learning strategies through interaction with complex environments, it offers a new approach to addressing various challenges in macro-component placement. Specifically, reinforcement learning methods model the problem as a Markov decision process in a discrete action space by dividing the chip layout into a discrete grid and progressively placing macro-components into the grid. At each step of the Markov process, the reinforcement-learned agent determines the placement position of the current macro-component. After placing the current component, the system provides a reward, and the state transitions. This process is repeated until every macro-component is placed on the chip layout. Reinforcement learning-based chip macro-component placement methods can achieve placement results with lower line lengths, and the trained placement strategies can generalize to other chips, with faster placement speeds compared to traditional methods. However, current reinforcement learning-based methods suffer from problems such as difficulty in satisfying non-overlapping constraints, sparse rewards, limited available information during placement, and potential congestion in the placement results, making them difficult to handle complex modern chips.

[0006] Therefore, for the chip macro-component placement problem based on reinforcement learning, a method that satisfies constraints, is efficient, and reliable needs to be designed. Summary of the Invention

[0007] Purpose of the invention: To address the problems of existing reinforcement learning-based chip macro-element placement methods, such as difficulty in satisfying non-overlapping constraints, sparse rewards, limited available information during placement, and easy congestion of placement results, this invention proposes an adjustment-based reinforcement learning placement method.

[0008] Technical Solution: A chip macro-component placement method based on reinforcement learning for layout adjustment. Reinforcement learning is used to progressively adjust each macro-component on the existing layout, rather than placing them uniformly. The optimization problem for macro-component placement is formulated as follows: under the constraint of non-overlapping between components, determine the position of each component to minimize the wiring length of these components. The half-perimeter wirelength (HPWL) is used to estimate the macro-component wiring length.

[0009] The optimization problem of macro-component placement (chip component layout problem) can be modeled as follows: The input includes a netlist H = (V, E), where V represents the information of all components (i.e., height and width), which need to be placed on the chip board; E is a netlist consisting of hyperedges e iA hypergraph consisting of elements ∈E, where each hyperedge contains two or more elements, indicates that these elements need to be connected by a set of wires during the placement process. The element placement solution s includes all elements. The position, where k represents the number of elements, and each element v i The coordinates of ∈V on the chip surface are represented as (x i ,y i For a component layout solution s, its HPWL index is calculated as follows: for each hyperedge e j ∈E corresponds to a rectangular region, which is defined by coordinates The lower left endpoint and coordinates The upper right endpoint is marked, indicating that the rectangle encloses e. j The smallest rectangle containing all elements. Let... and Let the width and height of the rectangle represent the width and height, respectively. The method for calculating the half-perimeter wiring length is as follows:

[0010]

[0011] During the optimization process, we aim to minimize the HPWL objective function while satisfying the non-overlapping constraint. Therefore, the chip component placement problem can be formalized as:

[0012] min HPWL(s,H),stD(s)≤∈

[0013] Where D represents chip density, and ∈ represents the constraint value that the density needs to satisfy. Ultimately, the goal is to improve chip power consumption, performance, and area (PPA) by minimizing HPWL.

[0014] The application of reinforcement learning in chip placement currently faces several limitations, including long training times, inability to guarantee performance improvement (PPA), and a lack of generalization ability across different chip layouts. In this study, we emphasize these problems primarily because of the problem statement: traditional RL methods perform macroscopic placement from scratch, often resulting in limited information state and inaccurate reward signals during the learning process. Current reinforcement learning-based placement methods suffer from long training times, poor generalization ability, and inability to guarantee PPA performance. A key issue lies in the problem statement itself: using reinforcement learning for placement from scratch leads to a lack of useful information and inaccurate rewards during training. To address these challenges, we propose a novel reinforcement learning method called MaskRegulate, which shifts the focus from the initial placement to optimizing existing placements. The reinforcement learning policy acts as a regulator rather than a placer, acting on the pre-existing placement to acquire comprehensive state information and obtain more accurate rewards. This adjustment improves the efficiency of the learning process, ultimately improving the placement results.

[0015] The order of macro-element placement is determined based on predefined rules, such as the number of networks, the size of macro-elements, and the number of connections to already placed modules. An episode ends after all macro-elements have been placed. Typically, the state representation includes information about the chip layout, the macro-elements already placed, and the macro-element currently being placed. In early reinforcement learning methods, the reward was determined only after all macro-elements had been placed, resulting in a sparse reward signal and increasing the complexity of the training process. Recent research has introduced various methods to enhance the density of the reward signal. Unlike reinforcement learning placers, our reinforcement learning adjuster focuses on optimizing the existing layout by adjusting the position of one macro-element at each step. Unlike placers that start from scratch, the adjuster can utilize additional information when adjusting each macro-element. Specifically, the adjuster considers not only the macro-elements already placed but also the positions of all other macro-elements. Furthermore, although using a similar reward function as before, it still improves accuracy by considering all macro-elements. Due to these advantages in the formulation of the MDP problem, even without considering additional factors (such as regularity), the reinforcement learning adjuster achieves better results than the reinforcement learning placer in experiments. Furthermore, compared to starting the layout from scratch, the adjuster is easier to adjust when no chip is seen, because the incomplete state information of the layouter is worse when no chip is seen, resulting in worse performance.

[0016] The placement of macro-components on a chip has a significant impact on subsequent chip design processes, including standard cell placement and chip routing. If only minimizing line length is the focus (as is the case with most current reinforcement learning placers), some macro-components may end up in the center of the chip layout, causing congestion. This results in the available placement area being divided into separate, unconnected sub-regions. Consequently, standard cells connected by the same network may be scattered across different placement sub-regions, increasing the overall line length and ultimately degrading timing performance. Therefore, experienced engineers typically place macro-components on the periphery of the chip during placement to prevent this. We integrate regularity into our learning-based placement method to achieve similar placement preferences as experienced engineers. Intuitively, macro-components near the edges tend to have lower regularity. Therefore, we propose a simple and effective way to measure regularity. On the chip layout, the regularity of a grid located at (x,y) is calculated as min{x,X} max -x}+min{y,Y max -y}, where X max and Y max These represent the actual lengths of the horizontal and vertical axes, respectively.

[0017] During training, the reward for MaskRegulate consists of two parts: r wire and r reg , representing the reduction in HPWL and the improvement in regularity after optimizing the current macro-element, respectively. To mitigate the impact of line length and regularity on the scale difference of training, r wire and r reg All values ​​are normalized. The final reward is r = α·r. wire +(1-α)·r reg α is a tradeoff coefficient, typically taken as 0.5. We use the Proximity Policy Optimization (PPO) algorithm to learn the policy.

[0018] The detailed algorithm flow is as follows:

[0019] Algorithm Flow 1: Chip Macro-Component Placement Method Based on Reinforcement Learning for Layout Adjustment

[0020] Input: A valid initial layout result s, a netlist H, the number of adjustments N, and an initial adjustment strategy π.

[0021] Output: The trained adjuster policy π′

[0022] Based on the netlist information H, the macro element adjustment sequence O is determined, and the current adjustment count i = 1.

[0023] 1. When the number of training iterations i ≤ N:

[0024] 2. When there are still macro components that have not been adjusted:

[0025] 3. Select a component according to the macro component adjustment sequence O.

[0026] 4. Observe the current state s and adjust it using strategy π to obtain action a = π(s).

[0027] 5. Update the layout state s', obtain reward r, and update the dataset D←(s,a,s′,r).

[0028] 6. Based on dataset D, use PPO for policy training.

[0029] 7. Return to the trained adjuster policy π′

[0030] Beneficial effects: The method proposed in this invention, which uses reinforcement learning to learn how to adjust the existing layout, can provide sufficient information for the policy and obtain relatively dense and accurate rewards, thereby improving the training efficiency of reinforcement learning algorithms for chip macro-component layout and obtaining better layout results.

[0031] Compared with existing chip macro-component placement methods, this method has high optimization efficiency, good performance, and is easy to implement. It can be conveniently applied to adjust various existing placement results and has great potential for practical application. Attached Figure Description

[0032] Figure 1 This is the general process for chip layout;

[0033] Figure 2 This is a flowchart of a chip macro-component placement method based on reinforcement learning for layout adjustment according to an embodiment of the present invention;

[0034] Figure 3 This shows the layout and congestion of different methods in eight ICCAD 2015 benchmarks. Congestion results were obtained from Cadence Innovus, with red dots indicating critical congestion areas, including (a) DREAMPlace, (b) AutoDMP, (c) WireMask-EA, (d) MaskPlace, and (e) MaskRegulate. Detailed Implementation

[0035] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0036] MaskRegulate is a chip macro-component placement method based on reinforcement learning for layout adjustment. This method uses reinforcement learning to adjust existing chip layouts. Compared to traditional methods that place components from scratch, MaskRegulate obtains more information and receives denser and more accurate rewards during the adjustment process, thus improving training efficiency and resulting in better-performing chips. The input to MaskRegulate is a valid initial layout result *s*, a netlist *H*, the number of adjustments *N*, and a randomly initialized policy *π*. The output is the trained adjuster policy *π′*. MaskRegulate adjusts chip components one by one based on the initial layout result. The adjustment order *O* is determined by the netlist information *H*, generally prioritizing macro-components with larger areas. After determining the adjustment order, completing one adjustment for all components is called an episode. Within each episode, at each step, a component is selected sequentially, its current state *s* is observed, and it is adjusted using policy *π*, resulting in the action *a = π(s)*. Next, the layout state *s′* is updated, a reward *r* is obtained, and the dataset *D←(s,a,s′,r)* is updated. After each round, policy training is performed using PPO based on dataset D. After a total of N rounds, the trained adjuster policy π′ is returned.

[0037] For a component layout solution s, the HPWL index is calculated as follows: for each hyperedge e j ∈E corresponds to a rectangular region, which is defined by coordinates The lower left endpoint and coordinates The upper right endpoint is marked, indicating that the rectangle encloses e. j The smallest rectangle containing all elements. Let... and Let the width and height of the rectangle represent the width and height, respectively. The method for calculating the half-perimeter wiring length is as follows:

[0038]

[0039] During the optimization process, we aim to minimize the HPWL objective function while satisfying the non-overlapping constraint. Therefore, the chip component placement problem can be formalized as:

[0040] min HPWL(s,H),stD(s)≤∈

[0041] Where D represents chip density, and ∈ represents the constraint value that the density needs to satisfy. Ultimately, the goal is to improve chip power consumption, performance, and area performance by minimizing HPWL.

[0042] The effectiveness of the MaskRegulat algorithm was validated by selecting eight chips from the ICCAD2015 chip design dataset.

[0043] Our comparison algorithm is as follows:

[0044] • DREAMPlace (DMP): An advanced analysis-based layout method

[0045] WireMask-EA: A state-of-the-art black-box macro component placement method.

[0046] AutoDMP: Improves DREAMPlace's macro-component placement method by iteratively exploring its configuration space.

[0047] MaskPlace: A representative reinforcement learning method whose policy architecture, state, and HPWL reward are similar to our method.

[0048] • MaskRegulate: Our proposed method

[0049] The overall evaluation results are shown in Table 1. MaskRegulate achieved the best average ranking in both proxy metrics and PPA metrics. DREAMPlace had the worst average ranking in terms of line length, congestion, and timing. However, after adjustments to MaskRegulate, the resulting layout achieved the best average ranking. Compared to MaskPlace, MaskRegulate significantly improved in several PPA metrics: route line length was increased by 17.08%, horizontal and vertical congestion overflow were improved by 73.08% and 38.81% respectively, worst negative margin was improved by 18.35%, total negative margin was improved by 37.89%, and the number of violation points was reduced by 46.17%. By integrating regularity, MaskRegulate achieved the highest regularity across all eight chips.

[0050] Table 1: Results of surrogate and PPA metrics in the ICCAD 2015 benchmark. Global HPWL (1e8) and regularity (1e6) are two surrogate metrics. PPA metrics were evaluated using Cadence Innovus. Layout was performed using different methods, with subsequent stages completed by Cadence Innovus. The following are the definitions of each metric: rWL (m): route length; rO-H (%) and rO-V (%): representing horizontal and vertical congestion, respectively; WNS (ns): worst-case latency; TNS (1e5μs): total latency; NVP (1e4): number of violations. For WNS and TNS, higher values ​​are better, while for other metrics, lower values ​​are better.

[0051]

[0052]

[0053] In addition, we provide detailed visualizations of all method placement results for all eight chips in ICCAD 2015. For example... Figure 3 As shown, MaskRegulate demonstrates a significant improvement in congestion metrics. Furthermore, MaskRegulate's layout results are more regular than all other methods.

[0054] The generalization ability of reinforcement learning strategies is an important research question. We pre-trained MaskRegulate and MaskPlace on the first four ICCAD chips (superblue1, superblue3, superblue4, and superblue5) and tested them on the remaining four chips. To further verify the ability of MaskRegulate to adjust the results of different initial layouts, we used it to adjust the results obtained on the test chips with different initial layouts.

[0055] Table 2 shows the generalization results of the proxy metrics for the four chips in the ICCAD 2015 benchmark.

[0056]

[0057]

Claims

1. A chip macro-component placement method based on reinforcement learning for layout adjustment, characterized in that, The optimization problem of macro component placement is formulated as follows: under the constraint that components do not overlap, determine the position of each component such that the wiring length of these components is minimized. Using half-perimeter wiring length to estimate macro-component wiring length, chip power consumption, performance, and area performance are improved by minimizing HPWL; The reinforcement learning adjuster focuses on optimizing the existing layout by adjusting the position of a macroelement at each step; unlike a layouter that starts from scratch, the adjuster can utilize additional information when adjusting each macroelement; specifically, the adjuster considers not only the macroelement that has been placed, but also the positions of all other macroelements. The method input is a valid initial layout result. Netlist Number of adjustments and the strategy of random initialization The output is the trained adjuster policy. ; Based on the initial layout, the chip components are adjusted one by one, in the following order: From netlist information First, prioritize adjusting macro-components with larger areas. After determining the adjustment order, completing one adjustment cycle for all components constitutes one round. Within each round, at each step, a component is selected sequentially, and its current state is observed. Use strategy Adjust it to obtain the action. ; Next, update the page layout status. Receive rewards Update the dataset After the round ends, based on the dataset Policy training was performed using PPO; in total After the round, return to the trained adjuster policy. ; Regularity is integrated into a learning-based layout approach to achieve layout preferences similar to those of experienced engineers; a method is proposed to measure regularity as follows: on the chip board, located at... The regularity of the mesh is calculated as follows: ,in and These represent the actual lengths of the horizontal and vertical axes, respectively. The rewards during training consist of two parts: and , representing the reduction in HPWL and the improvement in regularity after optimizing the current macro-element, respectively; to mitigate the impact of line length and regularity on the scale difference of training, and All were normalized; the final reward was ,in It is a tradeoff coefficient; the policy is learned using a proximal policy optimization algorithm.

2. The chip macro-component placement method based on reinforcement learning for layout adjustment according to claim 1, characterized in that, The optimization problem of macro-component placement is modeled as follows: the input includes a netlist. ,in This indicates information about all components, which need to be laid out on the chip board. It is a super-edge The resulting hypergraph, where each hyperedge contains two or more components, indicates that these components need to be connected by a set of wires during the placement process; component placement solution Includes all components The location, among which Indicates the number of components, each component The coordinates on the chip surface are represented as follows: For a component layout solution The HPWL index is calculated as follows: for each hyperedge This corresponds to a rectangular region, which is defined by coordinates ( The lower left endpoint and coordinates of ) The upper right endpoint of the rectangle is marked, indicating that the rectangle is enclosing the area. The smallest rectangle containing all elements; let and Let represent the width and height of the rectangle, respectively. During the optimization process, the HPWL objective function is minimized while satisfying the non-overlapping constraint; the chip component placement problem is formalized as follows: in Indicates chip density, This represents the constraint value that the density needs to satisfy.

Images