A floor planning method based on reinforcement learning and genetic algorithm
By combining reinforcement learning and genetic algorithms in the graph planning method, the graph planning of VLSI is optimized, solving the area and line length problems under bounding box constraints and achieving a more efficient global optimum search.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2022-11-29
- Publication Date
- 2026-05-29
AI Technical Summary
In the physical design of very large-scale integrated circuits, layout planning is difficult to effectively optimize area and line length while considering bounding box constraints. Existing reinforcement learning algorithms have long training times and are prone to getting stuck in local optima. Heuristic algorithms are inefficient in large-scale chip netlists.
Combining reinforcement learning and genetic algorithms, a population is generated by randomly perturbing a B*-tree structure. Individuals are optimized using crossover and mutation operators, and pre-trained reinforcement learning agents are used for local optimization. Dual benchmark reward value calculation is combined to avoid local optima and improve global search efficiency.
Effective optimization of graph layout planning under bounding box constraints improves the ability to search for the global optimum, avoids the local optimum problem of using reinforcement learning algorithms alone, and improves the success rate and efficiency of graph layout planning.
Smart Images
Figure CN115758981B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of integrated circuit design automation technology, and particularly relates to a layout planning method based on reinforcement learning and genetic algorithms. Background Technology
[0002] In VLSI physical design, layout planning is one of the key steps. Layout planning determines the geometric relationships between modules, as well as cost metrics such as layout area and line length. For layout planning with bounding box constraints, it is necessary to satisfy the basic constraints that all modules cannot overlap and cannot exceed the bounding box, which further increases the difficulty of macro-placement.
[0003] Several heuristic algorithms have been proposed to solve graph layout problems, such as simulated annealing, genetic algorithms, and meme algorithms. One such adaptive hybrid meme algorithm, which uses a genetic algorithm as the global search algorithm and a simulated annealing algorithm as the local search algorithm, is a multi-objective layout optimization method that has been shown to have better global optimum search characteristics.
[0004] Reinforcement learning achieves self-learning through the interaction between an agent and its environment. The agent changes the state of the environment through its actions, receives reward feedback from the environment, and updates the model gradient. Reinforcement learning has good problem-solving capabilities for many combinatorial optimization problems. It describes the problem to be solved as a Markov decision process and designs a reward function based on the solution objective, thus enabling the reinforcement learning model to learn how to maximize the cumulative reward value over a training cycle. Previously, researchers used the proximal policy optimization algorithm in reinforcement learning to train a reinforcement learning agent capable of completing macro-module placement. The placement completed by this agent has been shown to surpass the design of physical design experts. However, for large-scale chip netlists, this method requires a very long training time under limited computing power. In addition, a reinforcement learning algorithm based on offline policy learning has been used to train an agent for randomly adjusting and optimizing the placement. This method has also been shown to perform better than simulated annealing. However, this method lacks stability and is prone to getting trapped in local optima during the optimization process, thus wasting remaining search time. Summary of the Invention
[0005] The purpose of this invention is to provide a graph planning method based on reinforcement learning and genetic algorithms to solve the technical problem of better optimizing the area and line length of the graph and improving the success rate of graph planning.
[0006] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:
[0007] A graph planning method based on reinforcement learning and genetic algorithms includes the following steps:
[0008] Step S1: Generate an original population with a fixed number of individuals by randomly perturbing the B*-tree structure;
[0009] Step S2: Calculate the fitness of all individuals and determine the optimal individual in the population based on the fitness values;
[0010] Step S3: In each population update cycle, select whether to use the crossover operator in that cycle based on probability. If selected, select an individual from the population as the parent and use the crossover operator to generate offspring individuals.
[0011] Step S4: Select whether to use the mutation operator in this period based on probability. If selected, select individuals in the population as parents and use the genetic operator to generate offspring individuals.
[0012] Step S5: Further optimize the offspring individuals using a local optimization method based on reinforcement learning agents, and determine whether the offspring individuals should replace the parent individuals based on their fitness.
[0013] Step S6: Repeat steps S3-S5 until the number of iterations reaches the upper limit, and obtain the best individual in the population.
[0014] Furthermore, in step S1, there are four ways to perturb the B*-tree structure: rotating a module by 90 degrees; randomly deleting a node other than the root node and randomly placing the node in a new placeable location; randomly swapping the positions of two nodes; and randomly swapping two subtrees with the same height.
[0015] Furthermore, in step S2, fitness is an indicator for evaluating individuals in the genetic algorithm population. The fitness value is calculated using equation (1), and the fitness function is inversely proportional to the cost function.
[0016]
[0017] Where cost(F) is the cost function of the layout planning, and the cost is calculated by equation (2). The cost of layout planning includes three parts: area, line length, and penalty term.
[0018]
[0019] Where A represents area, B represents line length, P represents penalty term, α is the weight of area, β is the weight of line length, and γ is the weight of penalty term. The three costs are divided by their initial values A. * B * and P * We will standardize the three cost terms.
[0020] Furthermore, in step S3, the crossover operator is the core operator of the genetic algorithm. The crossover operator generates a new offspring structure by replacing and recombining parts of the structure of two parent B*-tree individuals. The crossover operator selects two parent B*-tree individuals p1 and p2 from the population. First, it randomly selects a node b (excluding the root node) from p1. This node b and all nodes below it form a subtree p1'. Then, it copies p2 to p2', deletes all nodes in p2' that exist in p1', and inserts p1' into the position of the original node b in p2', thereby generating a new offspring B*-tree individual c1. The crossover operator can improve the ability to search for the global optimum while preserving the characteristics of the parents.
[0021] Furthermore, in step S4, the mutation operator is another operator defined by the genetic algorithm. The mutation operator includes two mutation methods: the first is to randomly swap the two subtrees of the root node of the parent B*-tree to generate the offspring B*-tree individual c2; the second is to randomly select a node b and its subtree p of the parent B*-tree and move them to a new position to generate the offspring B*-tree individual c2.
[0022] Furthermore, in step S5, the reinforcement learning agent used is a pre-trained model; the training samples are randomly generated, containing a netlist of 100 macro modules and 100 wires, with each wire connected to 6 pins; the reinforcement learning model adopts a deep Q offline learning model, and the replay experience pool capacity of the model is set to 20000. When the model is updated, 128 samples are randomly sampled from the replay experience pool for gradient update of the model. The loss function value is calculated by equation (3). The model reduces the loss function value through gradient update.
[0023]
[0024] Where r is the reward value of the current round, γ is the discount factor, a' is the action distribution under adjacent state s', and θ P The model parameters θ represent the policy network. T The model parameters represent the target network, and Q(s,a;θ) is the value estimation function for taking action a in state s. The experience replay mechanism in deep Q-learning improves data utilization and model update efficiency.
[0025] The output of reinforcement learning is the action to be performed, and the input is a one-dimensional vector, which includes the following feature variables: the proportion of affected modules to the total number of modules; the proportion of the total area of disturbed modules to the total area of all modules; the standardized cost value of the current state; the standardized cost value of the adjacent states; the minimum standardized cost value during the search process; the average standardized reward value during the search process; and the index of the current step. All of these variables are necessary, and for cases where bounding box constraints need to be considered, three additional variables need to be added: the proportion of out-of-bounds modules to the total number of modules; the standardized width of the layout exceeding the bounding box; and the standardized length of the layout exceeding the bounding box.
[0026] The reinforcement learning reward value is calculated by equation (4), and the reward value function takes the layout cost as the only factor in the reward value calculation.
[0027]
[0028] In equation (2), the cost function f(·) is mentioned, and the baseline is the cost value of the initial state. Due to the diversity of training samples, if only the cost value of the initial state of the sample is used as the benchmark for the standardized reward value, the cumulative reward values of different samples during the training period will not match. To solve this problem, a dual-benchmark reward value calculation method as shown in equation (5) is adopted.
[0029]
[0030] Among them, s initial It is the initial state, s after The state is obtained by optimizing the sample using a simulated annealing algorithm.
[0031] Furthermore, in step S6, if the size of the optimized example is too large, and the termination condition is determined based on the number of iteration cycles, resulting in an excessively long running time, it is also possible to determine whether to terminate based on the preset running time; after termination, the individual with the highest fitness value in the population is taken as the output solution.
[0032] The graph planning method based on reinforcement learning and genetic algorithm proposed in this invention has the following advantages: Under the premise of considering the bounding box constraints, this invention can optimize the layout locally through the reinforcement learning policy model and optimize the global layout globally by combining the graph planning algorithm that combines reinforcement learning and genetic algorithm. This is conducive to better searching for the global optimum, improving search efficiency, and effectively avoiding the problem of getting trapped in local optima caused by using reinforcement learning algorithm alone. Attached Figure Description
[0033] Figure 1 This is a flowchart of a graph planning method based on reinforcement learning and genetic algorithms according to the present invention. Detailed Implementation
[0034] To better understand the purpose, structure, and function of this invention, a graph planning method based on reinforcement learning and genetic algorithms will be described in further detail below with reference to the accompanying drawings.
[0035] Example 1:
[0036] The technical solution of the invention will now be described in detail with reference to the accompanying drawings.
[0037] This invention provides a graph planning method based on reinforcement learning and genetic algorithms that considers bounding box constraints. The method first generates an initial population with a fixed number of individuals by randomly perturbing a B*-tree structure, calculates the fitness of all individuals, and marks the individual with the highest fitness as the optimal individual. Then, in each population iteration cycle, individuals in the population are selected as parents with a certain probability, and crossover or mutation operators are used to generate offspring individuals. Next, a graph optimization method based on reinforcement learning agents is used to further optimize the offspring individuals, and the fitness of the offspring individuals determines whether they should replace the parent individuals. Finally, the iteration ends, and the optimal individual in the population is obtained. Figure 1 As shown, the method includes the following steps:
[0038] 101, claiming a fixed number of original populations by perturbing the B*-tree structure;
[0039] 102. Calculate the fitness of all individuals;
[0040] 103, Use the crossover operator to generate offspring individuals;
[0041] 104. Use mutation operators to generate offspring individuals;
[0042] 105. Further optimization of offspring individuals using reinforcement learning agents;
[0043] 106. The iteration ends, and the optimal individual is obtained.
[0044] In B*-tree 101, to maintain the diversity of the original population, the perturbation method must be randomly selected. There are four ways to perturb the B*-tree structure: rotate a module by 90 degrees; randomly delete a node other than the root node and randomly place that node in a new position; randomly swap the positions of two nodes; and randomly swap two subtrees with the same height.
[0045] In equation (102), fitness is an indicator for evaluating individuals in the genetic algorithm population. The fitness value is calculated using equation (1), and the fitness function is inversely proportional to the cost function.
[0046]
[0047] Where cost(F) is the cost function of the layout, and the cost is calculated by equation (2). The layout cost includes three parts: area, line length, and penalty term.
[0048]
[0049] Where A, B, and P represent area, line length, and penalty term, respectively, and α, β, and γ are the weights of the three costs. The three costs are then divided by their initial values A. * B * and P * We will standardize the three cost terms.
[0050] In algorithm 103, the crossover operator is the core of the genetic algorithm. The crossover operator generates a new offspring structure by replacing and recombining parts of the structure of two parent B*-tree individuals. The crossover operator selects two parent B*-tree individuals p1 and p2 from the population. First, it randomly selects a node b (excluding the root node) from p1. This node b and all nodes below it form a subtree p1'. Then, it copies p2 to p2', deletes all nodes in p2' that existed in p1', and inserts p1' into the original position of node b in p2', thus generating a new offspring B*-tree individual c1. The crossover operator can improve the ability to search for the global optimum while preserving parent characteristics.
[0051] In 104, the mutation operator includes two mutation methods: the first is to randomly swap the two subtrees of the root node of the parent B*-tree to generate the child B*-tree individual c2; the second is to randomly select a node b and its subtree p of the parent B*-tree, and then move them to a new position to generate the child B*-tree individual c2.
[0052] In model 105, the reinforcement learning agent used is a pre-trained model. The training samples are randomly generated, containing a netlist of 100 macromodules and 100 wires, with each wire connected to 6 pins. The reinforcement learning model adopts a deep Q-learning offline model, and the replay experience pool capacity of the model is set to 20,000. When the model is updated, 128 samples are randomly sampled from the replay experience pool for gradient update of the model. The loss function value is calculated by equation (3). The model reduces the loss function value through gradient update.
[0053]
[0054] Where γ is the discount factor, a' is the action distribution under adjacent state s', and θ P and θ TLet θ represent the model parameters of the policy network and the target network, respectively. Q(s,a; θ) is the value estimation function for taking action a in state s. The experience replay mechanism in deep Q-learning improves data utilization and model update efficiency.
[0055] The output of reinforcement learning is the action to be performed, and the input is a one-dimensional vector, including the following feature variables: the proportion of affected modules to the total number of modules; the proportion of the total area of disturbed modules to the total area of all modules; the standardized cost value of the current state; the standardized cost value of the adjacent states; the minimum standardized cost value during the search process; the average standardized reward value during the search process; and the index of the current step. To ensure the model learns to conform to bounding box constraints, the following feature variables are also added: the proportion of out-of-bounds modules to the total number of modules; the standardized width of the layout exceeding the bounding box; and the standardized length of the layout exceeding the bounding box.
[0056] The reinforcement learning reward value is calculated by equation (4), and the reward value function takes the place value as the sole factor in the reward value calculation.
[0057]
[0058] In equation (2), f(·) represents the cost function, where s is the current state, s' is the adjacent state of s, and baseline is the cost of the initial state. Due to the diversity of training samples, if only the cost of the initial state of a sample is used as the benchmark for the standardized reward value, the cumulative reward values of different samples during the training period will not match. To solve this problem, a dual-benchmark reward value calculation method as shown in equation (5) is adopted.
[0059]
[0060] Among them, s initial It is the initial state, s after The state is obtained by optimizing the sample using a simulated annealing algorithm.
[0061] In step 106, if the number of optimized examples is too large, determining the termination condition based on the number of iterations may lead to excessively long runtime. Alternatively, a preset runtime can be used to determine whether to terminate. After termination, the individual with the highest fitness value in the population is taken as the output solution.
[0062] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.
Claims
1. A graph layout planning method based on reinforcement learning and genetic algorithms, characterized in that, Includes the following steps: Step S1: Through random perturbation The structure generates an original population with a fixed number of individuals; Step S2: Calculate the fitness of all individuals and determine the optimal individual in the population based on the fitness values; Step S3: In each population update cycle, select whether to use the crossover operator within that cycle with a fixed probability. If selected, select an individual from the population as the parent and use the crossover operator to generate offspring individuals. Step S4: Select whether to use the mutation operator within this period with a fixed probability. If selected, select individuals from the population as parents and use the genetic operator to generate offspring individuals. Step S5: Further optimize the offspring individuals using a local optimization method based on reinforcement learning agents, and determine whether the offspring individuals should replace the parent individuals based on their fitness. Step S6: Repeat steps S3-S5 until the number of iterations reaches the upper limit, and obtain the best individual in the population; In step S5, the reinforcement learning agent used is a pre-trained model; the training samples are randomly generated, containing a netlist of 100 macromodules and 100 wires, with each wire connected to 6 pins; the reinforcement learning model adopts a deep Q offline learning model, and the replay experience pool capacity of the model is set to 20000. When the model is updated, 128 samples are randomly sampled from the replay experience pool for gradient update of the model. The loss function value is calculated by equation (1). The model reduces the loss function value through gradient update. ； in It is the reward value for the current round. It is a discount factor. In the adjacent state The distribution of actions below, The model parameters representing the policy network, These represent the model parameters of the target network. It is in state Take action The value estimation function; The output of reinforcement learning is the action to be performed, and the input is a one-dimensional vector, which includes the following feature variables: the proportion of affected modules to the total number of modules; the proportion of the total area of disturbed modules to the total area of modules; the standardized cost value of the current state; the standardized cost value of the adjacent states; the minimum standardized cost value during the search process; the average standardized reward value during the search process; and the index of the current step. For cases where bounding box constraints need to be considered, three additional variables need to be added: the proportion of out-of-bounds modules to the total number of modules. The layout extends beyond the standardized width of the bounding box; The layout extends beyond the standardized length of the bounding box; The reinforcement learning reward value is calculated by equation (2), and the reward value function takes the layout cost as the sole factor in the reward value calculation. ； in The formula is the cost function. It is the cost of the initial state; the reward value calculation method with dual benchmarks as shown in equation (3) is adopted. ； in, It is the initial state. The state is obtained by optimizing the sample using a simulated annealing algorithm.
2. The graph planning method based on reinforcement learning and genetic algorithm according to claim 1, characterized in that, In step S1, the disturbance There are four ways to structure a module: rotate a module by 90 degrees; randomly delete a node other than the root node and randomly place the node in a new placeable location; randomly swap the positions of two nodes; and randomly swap two subtrees with the same height.
3. The graph planning method based on reinforcement learning and genetic algorithm according to claim 1, characterized in that, In step S2, the fitness value is calculated using equation (4), and the fitness function is inversely proportional to the cost function. ； in, It is the cost function of the layout planning, and the cost is calculated by equation (5). The cost of layout planning includes three parts: area, line length, and penalty term. ； in Represents area, Represents line length, Represents the penalty item. It is the area weight. It is the weight of line length. The weights of the penalty terms are calculated by dividing each of the three costs by their initial values. , and We will standardize the three cost terms.
4. The graph planning method based on reinforcement learning and genetic algorithm according to claim 3, characterized in that, In step S3, the crossover operator combines the two parent generations. An individual generates a new offspring structure by replacing and recombining parts of its structure; the crossover operator selects two parent generations from the population. individual and First from Randomly select one node other than the root node. This node The subtree consists of all nodes below it. Then copy for , and delete It exists in All nodes in, and insert Central Plains Node The position, thereby generating new offspring. individual .
5. The graph planning method based on reinforcement learning and genetic algorithm according to claim 3, characterized in that, In step S4, the mutation operator includes two mutation methods: the first is to randomly swap the parent generation. The two subtrees of the root node generate children. individual The second method is to randomly select the parent generation. A node and its subtree Then move to a new location to generate offspring. individual .
6. The graph planning method based on reinforcement learning and genetic algorithm according to claim 5, characterized in that, In step S6, using the number of iteration cycles as a termination condition would result in excessively long running time. The system determines whether to terminate based on the preset running time. After termination, the individual with the highest fitness value in the population is taken as the output solution.