A reinforcement learning-based legalization approach for chip instance placement

Abstract

The present invention discloses a reinforcement learning-based legalization method for chip instance placement, which can dynamically adapt to various placement rules and manufacturing constraints, including processing units of different heights and sizes. This makes the algorithm not only suitable for the placement of standard units, but also effectively handles the placement of macroblocks. The method of the present invention can learn from past placement practices and continuously optimize its placement strategy. This learning ability enables the algorithm to continue to improve with the development of technology and changes in design requirements, and better cope with the rapid iteration of integrated circuit design. Moreover, when faced with large-scale chips, the reinforcement learning algorithm can also complete the legalization of chip instances within a limited time. In short, the method of the present invention not only improves the flexibility and accuracy of chip placement, but also reduces the dependence on complex algorithm modifications, providing a more efficient and reliable solution for modern integrated circuit design.

Description

Technical Field

[0001] The present invention relates to the legalization of chip instance placement, and in particular to a chip instance placement legalization method based on reinforcement learning. Background Art

[0002] Legalized placement in chip design is the process of ensuring that all cells in the chip layout do not overlap and meet manufacturing specifications. Although traditional placement algorithms such as Abacus and Tetris perform well in some scenarios, they begin to show some limitations as the complexity of integrated circuit design increases.

[0003] Taking Abacus as an example, the algorithm is mainly designed to process standard cells with a single row height, but it does not work well for cells with multiple rows of height and different sizes that are common in modern chip design. In addition, as a predefined deterministic algorithm, Abacus has a fixed response to input, which is particularly insufficient in an environment of rapid iteration of integrated circuit design and continuous technological advancement. In order to improve its ability to handle new and complex scenarios, the algorithm may need to be significantly modified or even redesigned. This not only involves high costs, but also requires in-depth algorithmic knowledge and experiments to ensure the effectiveness of the modified algorithm. Therefore, it is challenging and time-consuming to update and maintain these algorithms to adapt to new design requirements. Summary of the invention

[0004] The purpose of the present invention is to provide a reinforcement learning-based legalization method for chip instance placement that is more efficient, flexible and self-optimizing to meet the growing complexity and diversity requirements of modern integrated circuit design.

[0005] The technical solution of the present invention is:

[0006] A reinforcement learning-based legitimization method for chip instance placement includes the following steps:

[0007] S1. Data preprocessing, including:

[0008] S1-1. Select the legalization direction: Legalization is completed independently in the horizontal and vertical directions, or it is completed alternately in the two directions;

[0009] S1-2, data acquisition: obtain the data of each instance, including the coordinates of the lower left corner, length and width values;

[0010] S1-3, intercepting the legalization processing area: for each instance to be legalized, intercepting an area near it as the legalization processing area of ​​the instance;

[0011] S1-4, instance sorting: sort the instances in each legalization processing area according to the size of the ordinate or abscissa of the lower left corner of the instance;

[0012] S1-5. Construct instance directed acyclic graph DAG: Based on the sorted instances, check the overlapping relationship between instances and construct a DAG graph, where nodes represent line segments and edges represent parent-child relationships;

[0013] S1-6, find connected branches: start from the instance to be processed in the DAG graph and find the connected branch where it is located; the movement of the instance is only restricted by the instances inside the connected branch, and the instances outside the connected branch are not considered to reduce the amount of calculation;

[0014] S1-7, establish topological sorting: topologically sort the instances inside the connected branches to determine the order in which the instances are moved, ensuring that any instance is moved only after all its parent nodes have been processed;

[0015] S2. Legalization solution based on reinforcement learning; specifically including:

[0016] S2-1. Define the state space: Define the state space of the reinforcement learning model to describe the characteristics of each instance, including the distance relationship with other line segments and the topological structure of the parent-child relationship;

[0017] S2-2, define discrete action space: select {-K,...,-1,0,1,...,K} as the discrete action space for movement in the selected direction, where the unit distance of movement and the maximum distance represented by K are set;

[0018] S2-3. Design reward function: impose penalties on behaviors that violate minimum distance requirements or destroy topological order;

[0019] S2-4, Model Optimization: The MaskablePPO strategy optimization algorithm is adopted, and the action mask is used to ensure that in each step, the model only considers legal actions that meet the constraints, thereby accelerating convergence and avoiding unnecessary exploration;

[0020] S2-5. Perform reinforcement learning training: In a complete layout cycle, all instances are adjusted through a series of actions so that the layout reaches a legal state, that is, all instances do not overlap and meet the minimum distance limit.

[0021] Preferably, in the step S1-1, in the selection of the legalization direction, the horizontal direction means that the instance can only move along the x-axis, and the vertical direction means that the instance can only move along the y-axis; the vertical direction is selected for illustration below, and the principle of the horizontal direction is the same;

[0022] In the data acquisition of step S1-2, a coordinate system is established on the chip layout, and grid lines parallel to the x-axis are laid with the minimum spacing as the width; the lower left corner of the instance is moved to the nearest grid line; and finally, the instance numbers of all instances that do not meet the spacing requirements are exported;

[0023] In the step S1-3 of cutting out the legalization processing area, for each instance to be legalized, a region near it is cut out in the y-axis direction as the legalization processing area of ​​the instance.

[0024] Preferably, in step S1-5, in constructing the instance directed acyclic graph DAG, a matrix A is used to record the connection relationship in the graph to avoid repeated addition of edges;

[0025] Specifically, when constructing an instance directed acyclic graph DAG, the basis for determining whether two nodes are directly connected is: the two instances overlap in projection onto the x-axis and the two instances cannot be reached through existing paths; in addition, when constructing the instance directed acyclic graph DAG, a matrix A representing the connection relationship is introduced, which is used to record the connection relationship between the edges that have been added and the nodes to avoid repeated addition of edges; the elements in the matrix A are first initialized to 0, and for node j, check whether A[i,j]=1 already exists between nodes i and j in the order of i=j-1,...,0; if not, check whether there is overlap; if there is overlap, add an edge i->j, and set the corresponding position in the matrix A to 1, that is, A[i, j]=1; at the same time, in order to record the connection relationship with the previous node, the i-th row element in A is added to the j-th row.

[0026] Preferably, the defined state space in step S2-1 specifically includes:

[0027] (1) The instance number, using one-hot embedding to form a feature vector;

[0028] (2) The distance to the nearest parent node and the distance to the nearest child node are standardized into two-dimensional features;

[0029] (3) The above feature vectors are concatenated and used as the input of the policy network.

[0030] Preferably, in the reward function designed in step S2-3, the reward function penalizes:

[0031] (1) Over-movement from the initial position to encourage the model to minimize adjustments and maintain the original layout;

[0032] (2) After the movement, the distance between the parent node instance and itself becomes lower than the minimum threshold, or even fails to meet the position relationship requirements, making all actions in the child node's action space infeasible. If this happens, the model will choose a movement as far away from the parent node as possible, and the reward function will impose a high penalty.

[0033] Preferably, the model optimization in step S2-4 specifically includes:

[0034] PPO adopts an actor-critic architecture, which consists of a policy and a value function network; the policy network inputs the features of each instance and outputs the probability of each action, which is adjusted by the action mask to zero invalid actions or horizontal coordinates; the value network inputs the current state and outputs an estimate of the expected return of the current state;

[0035] In a reinforcement learning setting using MaskablePPO, the rollout buffer stores the experience collected during each episode; the experience includes state, action, reward, next state, and whether the terminal state is reached; PPO then divides the rollout buffer into smaller batches and updates the policy multiple times using different subsets of the data; after collecting data, PPO's policy network uses a generalized advantage estimation update strategy to estimate the advantage of each state-action pair in the buffer; the advantage function measures how good it is to take a specific action compared to the average of all possible actions in a given state based on the expected reward, thereby selecting the most advantageous action in the current state.

[0036] Preferably, in the model optimization of step S2-4, the policy loss function of PPO also includes pruning alternative targets to prevent large policy updates that may undermine learning stability; the loss function of the policy network can be specifically expressed as:

[0037] ;

[0038] Among them, the advantage estimate According to GAE, represents the probability ratio of adopting the new and old strategies, Limit Probability Ratio exist between, is the model clipping parameter; finally, the policy network is updated by performing gradient ascent on this loss.

[0039] Preferably, in the model optimization of step S2-4, the value function network estimates the expected return of each state; this value function helps to calculate the advantage estimate in future iterations; the loss function of the value function network is expressed as:

[0040] ;

[0041] It measures the difference between the predicted values ​​and the actual observed returns.

[0042] Preferably, in the execution reinforcement learning training of step S2-5, the model learns how to adjust the positions of all instances through a series of legal moves in a complete layout cycle through multiple iterative training to achieve a legal configuration without overlap and in compliance with the minimum distance limit.

[0043] The advantages of the present invention are:

[0044] 1. The reinforcement learning-based legitimization method for chip instance placement proposed in this invention can dynamically adapt to various placement rules and manufacturing constraints, including processing cells of different heights and sizes. This makes the algorithm not only suitable for the placement of standard cells, but also effectively handles the placement of macros. In contrast, traditional algorithms such as Abacus are mainly applicable to standard cells with a single row height, and are not adaptable enough to multiple row heights and complex scenarios.

[0045] 2. The method of the present invention can learn from past placement practices and continuously optimize its placement strategy. This learning ability enables the algorithm to continue to improve with the development of technology and changes in design requirements, and better cope with the rapid iteration of integrated circuit design. In addition, when facing large-scale chips, the reinforcement learning algorithm can also complete the legal placement of chip instances within a limited time.

[0046] 3. In summary, the method of the present invention not only improves the flexibility and accuracy of chip placement by utilizing reinforcement learning technology, but also reduces the dependence on complex algorithm modifications, providing a more efficient and reliable solution for modern integrated circuit design. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0048] Figure 1 The diagram is a schematic diagram of the operation of the legalization method for chip instance placement based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0049] like Figure 1 As shown, the present invention takes local chip areas of 7 instances as examples to introduce the operating steps of the legalization method for chip instance placement based on reinforcement learning of the present invention.

[0050] Step 1: Data preprocessing.

[0051] 1. Select the legalization direction: Legalization can be done independently in the horizontal and vertical directions, or it can be done alternately and iteratively. Here, horizontal means that the instance can only move along the x-axis, and vertical means that the instance can only move along the y-axis. This embodiment uses the vertical direction for illustration, and the principle of the horizontal direction is the same.

[0052] 2. Data acquisition: Get the data of 7 instances, including the coordinates of the lower left corner, length and width. Establish a coordinate system on the chip layout and lay out grid lines with a minimum spacing of 50nm. Move the lower left corner of the instance to the nearest grid line (i.e. the position shown in the figure). Figure 1 In the example, the distance between Example 3 and Example 4 does not meet the minimum spacing requirement (50nm).

[0053] 3. Cutting off the legalization processing area: For the to-be-legalized instance 3, cut off an area near it in the y-axis direction as the legalization processing area of ​​the instance.

[0054] 4. Instance sorting: In each legalization processing area, the instances are sorted according to the size of the ordinate in the lower left corner of the instance (this order is used to construct the DAG. Figure 1 not indicated in the figure).

[0055] 5. Build instance directed acyclic graph (DAG): Based on the sorted instances, check the overlapping relationship between instances and build a DAG graph, where nodes represent line segments and edges represent parent-child relationships. Use matrix A to record the connection relationship in the graph to avoid repeated addition of edges. Figure 1 In the figure, the constructed parent-child edges are represented by arrows.

[0056] Specifically, when establishing a DAG, the basis for determining whether two nodes are directly connected is that the two instances overlap when projected onto the x-axis and the two instances cannot be reached through existing paths. In addition, when constructing the graph DAG, a matrix A representing the connection relationship is introduced. This matrix is ​​used to record the connection relationship between the edges that have been added and the nodes to avoid repeated addition of edges. The elements in A are first initialized to 0. For node j, check whether A[i,j]=1 already exists between nodes i and j in the order of i=j-1,...,0. If not, check whether there is overlap. If there is overlap, add an edge i->j and set the corresponding position in matrix A to 1, that is, A[i, j]=1. At the same time, in order to record the connection relationship with the previous node, the i-th row element in A is added to the j-th row.

[0057] 6. Find the connected branch: Start from the instance 3 to be processed in the DAG and find its connected branch. The connected branch contains instances 1, 2, 3, 4, and 5. The movement of instance 3 is only restricted by the instances inside the connected branch, so instances 6 and 7 outside the connected branch can be ignored to reduce the amount of calculation.

[0058] 7. Establish a topological sort: Topologically sort the instances within the connected branches to determine the order in which the instances are processed, ensuring that any instance is moved only after all its parent nodes have been processed. In this example, instance numbering is a feasible topological order.

[0059] Step 2: Legalization solution based on reinforcement learning.

[0060] 1. Define the state space: Define the state space of the reinforcement learning model to describe the characteristics of each instance, including the distance relationship with other line segments and the topological structure of the parent-child relationship. Specifically:

[0061] (1) The instance number is constructed using one-hot encoding to construct the feature vector. There are 5 instances in the connected branch in the figure, and the feature vector of instance 3 can be represented as [0, 0, 1, 0, 0].

[0062] (2) The distance to the nearest parent node and the distance to the nearest child node are standardized to become two-dimensional features.

[0063] The above feature vectors are concatenated and used as the input of the policy network.

[0064] 2. Define discrete action space: The movement of the instance in the y-axis direction can be selected as {-K,...,-1,0,1,...,K} as the discrete action space. Here, the unit distance of movement and the maximum distance represented by K are appropriately set.

[0065] 3. Design reward function: Design a reward function that imposes penalties on behaviors that violate the minimum distance requirement or destroy the topological order.

[0066] The reward function in this model penalizes:

[0067] (1) Over-movement from the initial position to encourage the model to minimize adjustments and maintain the original layout.

[0068] (2) After the movement, the distance between the parent node instance and itself is lower than the minimum threshold, or even does not meet the position relationship requirements, making all actions in the child node's action space infeasible. If this happens, the model will choose a movement as far away from the parent node as possible, and the reward function will impose a high penalty.

[0069] 4. Optimization strategy: The MaskablePPO strategy optimization algorithm is adopted, and the action mask is used to ensure that in each step, the model only considers legal actions that meet the constraints, thereby accelerating convergence and avoiding unnecessary exploration.

[0070] Specifically, PPO adopts an actor-critic architecture, which consists of a policy and a value function network. The policy network inputs the features of each instance and outputs the probability of each action, which is adjusted by the action mask to zero invalid actions. The value network inputs the current state (such as the current step number, that is, the topological order number) and outputs an estimate of the expected return of the current state.

[0071] In a reinforcement learning setting with MaskablePPO, the rollout buffer stores the experience collected during each episode. These experiences include state, action, reward, next state, and whether the terminal state is reached (a Boolean value). PPO then divides the rollout buffer into smaller batches and updates the policy multiple times using different subsets of the data. This approach not only improves efficiency but also helps stabilize policy changes. After collecting data, PPO's policy network updates the policy using generalized advantage estimation (GAE). It estimates the advantage of each state-action pair in the buffer. The advantage function measures how good it is to take a specific action compared to the average of all possible actions in a given state based on the expected reward, thereby selecting the most advantageous action in the current state. In addition, PPO's policy loss function also includes clipping alternative targets, which prevents large policy updates that may undermine learning stability. The loss function of this policy network can be specifically expressed as:

[0072] ;

[0073] Among them, the advantage estimate According to GAE, represents the probability ratio of adopting the new and old strategies, Limit Probability Ratio exist between, is the model clipping parameter; finally, the policy network is updated by performing gradient ascent on this loss.

[0074] The value function network estimates the expected reward for each state. This value function helps calculate the advantage estimate in future iterations. The loss function of the value function network can be expressed as:

[0075] ;

[0076] This is a standard mean squared error (MSE) loss function that measures the difference between the predicted value and the actual observed reward. Minimizing this loss ensures that the value function provides more accurate estimates of future rewards, which is critical for both advantage estimation and policy evaluation.

[0077] 5. Perform reinforcement learning training: In a complete layout cycle (episode), the goal of MaskablePPO is to adjust all instances through a series of actions so that the layout reaches a legal state, that is, all instances do not overlap and meet the minimum distance limit. The optimal solution of this embodiment is to move instance 3 up one grid so that its distance with instance 4 meets the minimum spacing requirement and the sum of the movement of all instances is minimized.

[0078] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any modifications made according to the spirit of the main technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A legitimization method for chip instance placement based on reinforcement learning, characterized in that: Includes steps: S1. Data preprocessing, including: S1-1. Select the legalization direction: Legalization is completed independently in the horizontal and vertical directions, or it is completed alternately in the two directions; S1-2, data acquisition: obtain the data of each instance, including the coordinates of the lower left corner, length and width values; S1-3, intercepting the legalization processing area: for each instance to be legalized, intercepting an area near it as the legalization processing area of ​​the instance; S1-4, instance sorting: sort the instances in each legalization processing area according to the size of the ordinate or abscissa of the lower left corner of the instance; S1-5. Construct instance directed acyclic graph DAG: Based on the sorted instances, check the overlapping relationship between instances and construct a DAG graph, where nodes represent line segments and edges represent parent-child relationships; S1-6, find connected branches: start from the instance to be processed in the DAG graph and find the connected branch where it is located; the movement of the instance is only restricted by the instances inside the connected branch, and the instances outside the connected branch are not considered to reduce the amount of calculation; S1-7, establish topological sorting: topologically sort the instances inside the connected branches to determine the order in which the instances are moved, ensuring that any instance is moved only after all its parent nodes have been processed; S2. Legalization solution based on reinforcement learning; specifically including: S2-1. Define the state space: Define the state space of the reinforcement learning model to describe the characteristics of each instance, including the distance relationship with other line segments and the topological structure of the parent-child relationship; S2-2, define discrete action space: select {-K,...,-1,0,1,...,K} as the discrete action space for movement in the selected direction, where the unit distance of movement and the maximum distance represented by K are set; S2-3. Design reward function: impose penalties on behaviors that violate minimum distance requirements or destroy topological order; S2-4, Model Optimization: The MaskablePPO strategy optimization algorithm is adopted, and the action mask is used to ensure that in each step, the model only considers legal actions that meet the constraints, thereby accelerating convergence; S2-5. Perform reinforcement learning training: In a complete layout cycle, all instances are adjusted through a series of actions so that the layout reaches a legal state, that is, all instances do not overlap and meet the minimum distance limit.

2. The legitimization method for chip instance placement based on reinforcement learning according to claim 1, characterized in that: In the step S1-1, in the selection of the legalization direction, the horizontal direction means that the instance can only move along the x-axis, and the vertical direction means that the instance can only move along the y-axis. The vertical direction is selected for illustration below, and the principle of the horizontal direction is the same. In the data acquisition of step S1-2, a coordinate system is established on the chip layout, and grid lines parallel to the x-axis are laid with the minimum spacing as the width; the lower left corner of the instance is moved to the nearest grid line; and finally, the instance numbers of all instances that do not meet the spacing requirements are exported; In the step S1-3 of cutting out the legalization processing area, for each instance to be legalized, a region near it is cut out in the y-axis direction as the legalization processing area of ​​the instance.

3. The method for legitimizing chip instance placement based on reinforcement learning according to claim 2, characterized in that: In step S1-5, in the example directed acyclic graph DAG, the matrix A is used to record the connection relationship in the graph to avoid repeated addition of edges; Specifically, when constructing an instance directed acyclic graph DAG, the basis for determining whether two nodes are directly connected is: the two instances overlap when projected onto the x-axis and the two instances cannot be reached through existing paths; in addition, when constructing the instance directed acyclic graph DAG, a matrix A representing the connection relationship is introduced, which is used to record the connection relationship between the edges that have been added and the nodes to avoid repeated addition of edges; the elements in the matrix A are first initialized to 0, and for node j, check whether A[i,j]=1 already exists between nodes i and j in the order of i=j-1,...,0; if not, check whether there is overlap; if there is overlap, add an edge i->j, and set the corresponding position in the matrix A to 1, that is, A[i, j]=1; at the same time, in order to record the connection relationship with the previous node, the i-th row element in A is added to the j-th row.

4. The method for legitimizing chip instance placement based on reinforcement learning according to claim 3, characterized in that: The definition state space of step S2-1 specifically includes: (1) The instance number, using one-hot embedding to form a feature vector; (2) The distance to the nearest parent node and the distance to the nearest child node are standardized into two-dimensional features; (3) Concatenate the above feature vectors and use them as the input of the policy network.

5. The method for legitimizing chip instance placement based on reinforcement learning according to claim 4, characterized in that: In the design reward function of step S2-3, the reward function will penalize: (1) Over-movement from the initial position to encourage the model to reduce adjustments and maintain the original layout; (2) After the movement, the distance between the parent node instance and itself becomes lower than the minimum threshold, or even fails to meet the position relationship requirements, making all actions in the child node's action space infeasible. If this happens, the model will choose the movement farthest from the parent node, and the reward function will impose a high penalty.

6. The method for legitimizing chip instance placement based on reinforcement learning according to claim 5, characterized in that: The model optimization in step S2-4 specifically includes: PPO adopts an actor-critic architecture, which consists of a policy and a value function network; the policy network inputs the features of each instance and outputs the probability of each action, which is adjusted by the action mask to zero the probability of invalid actions; the value network inputs the current state and outputs an estimate of the expected return of the current state; In a reinforcement learning setting using MaskablePPO, the rollout buffer stores the experience collected during each episode; the experience includes state, action, reward, next state, and whether the terminal state is reached; PPO then divides the rollout buffer into smaller batches and updates the policy multiple times using different subsets of the data; after collecting the data, PPO's policy network uses a generalized advantage estimation update strategy to estimate the advantage of each state-action pair in the buffer; the advantage function measures the expected reward for taking a specific action compared to the average of all actions in a given state, thereby selecting the most advantageous action in the current state.

7. The method for legitimizing chip instance placement based on reinforcement learning according to claim 6, characterized in that: In the model optimization of step S2-4, the policy loss function of PPO also includes clipping alternative targets to prevent large policy updates that undermine learning stability; the loss function of the policy network is specifically expressed as: ； Among them, the advantage estimate According to GAE, represents the probability ratio of adopting the new and old strategies, Limit Probability Ratio exist between, is the model clipping parameter; finally, the policy network is updated by performing gradient ascent on this loss.

8. The method for legitimizing chip instance placement based on reinforcement learning according to claim 7, characterized in that: In the model optimization of step S2-4, the value function network estimates the expected return of each state; this value function helps calculate the advantage estimate in future iterations; the loss function of the value function network is expressed as: ； The loss function measures the difference between the predicted value and the actual observed return.

9. The method for legitimizing chip instance placement based on reinforcement learning according to claim 8, characterized in that: In the execution reinforcement learning training of step S2-5, the model learns how to adjust the positions of all instances through a series of legal moves to achieve a legal configuration without overlap and in compliance with the minimum distance constraint through multiple iterations of training during the complete layout cycle.

Images