Improved reinforcement learning-based macro layout optimization system and method

Abstract

The invention discloses an improved reinforcement learning-based macro layout optimization system and method, and the method comprises the steps: setting a heuristic macro sorting module at the most front end of the system, and determining a placement sequence of a macro module; the return decomposition value network, the strategy network and the experience pool form a reinforcement learning agent, and the return decomposition value network and the strategy network cooperatively work to update the experience pool; and the state characterization module and the density perception reward function form a layout environment, receive the output of the reinforcement learning agent, and feed back layout state features and rewards to the reinforcement learning agent. According to the invention, a technical scheme of cooperative work is constructed, and through a series of innovative modules, the problems of sparse rewards, short view of decisions, calculation bottleneck and the like are systematically solved.

Description

Technical Field

[0001] This invention belongs to the field of electronic design automation technology, and specifically relates to an improved reinforcement learning-based macro-layout optimization system and method. Background Technology

[0002] Macro placement is a critical step in the VLSI physical design process. Its goal is to determine the precise locations of various macro modules on the chip while meeting strict physical constraints such as non-overlap, in order to optimize performance metrics such as half-circle length, timing, power consumption, and routerizability. High-quality macro placement schemes are essential for improving chip performance, power consumption, and area (PPA).

[0003] In recent years, reinforcement learning (RL) technology has provided a new paradigm for macro-layout automation due to its powerful capabilities in complex sequential decision-making problems. RL methods model the layout process as a Markov decision process, where the agent learns placement strategies step by step through interaction with the environment. MaskPlace is a representative work in this technical approach, introducing the concepts of WireMask and PositionMask to estimate wire length cost and ensure layout validity, respectively. It employs a reward function based on the change in wire length (-ΔHPWL) at each step, achieving results superior to many traditional methods.

[0004] However, existing reinforcement learning methods most similar to this invention, including MaskPlace and its subsequent improvements, still suffer from the following technical shortcomings that urgently need to be addressed: The reward signal is sparse and semantically ambiguous: the reward mechanism based on instantaneous ΔHPWL has inherent flaws. When the macromodule to be placed has no direct connection with the already placed modules, the reward is always zero, failing to provide an effective learning gradient for the agent. More importantly, for key intermediate modules in the connection path, their placement may be unreasonably penalized due to placement order factors, providing incorrect optimization signals and misleading the policy's convergence direction.

[0005] Short-sighted value estimation and difficulties in credit allocation: Existing methods typically use a single value network to estimate state value, making it difficult to accurately quantify the long-term impact of individual placement actions on the final layout quality. This leads agents to make "short-sighted" decisions, lacking forward-looking planning for the overall layout structure and future placeability, thus limiting the global optimality of the solution.

[0006] Computational efficiency is a bottleneck: Taking MaskPlace as an example, the generation of its WireMask and PositionMask relies on serial, module-level traversal computations, and the time complexity increases sharply with the increase of circuit size and layout mesh density. This becomes the main speed bottleneck of the entire training and inference process, severely restricting the practicality and scalability of the method in industrial-scale designs. Summary of the Invention

[0007] Based on an in-depth analysis of the shortcomings of existing reinforcement learning macro-layout methods in the background section, the purpose of this invention is to provide a novel reinforcement learning macro-layout optimization scheme to systematically solve the following technical problems: 1. Addressing the issues of sparse reward signals and semantic ambiguity: Overcoming the lack of guidance signals or misleading methods caused by relying on instantaneous line length changes (-ΔHPWL) as rewards in existing methods, providing agents with continuous, consistent, and semantically clear learning gradients in all placement steps (including placement of modules without direct connections and placement of intermediate modules in the connection chain).

[0008] 2. Alleviating the problems of short-sighted value estimation and difficulty in credit allocation: Improve the structure of traditional single value estimation networks so that they can effectively distinguish and quantify the immediate benefits and long-term strategic value of current deployment decisions, thereby guiding agents to make more forward-looking decisions.

[0009] 3. Solve the bottleneck problem of low computational efficiency in mask generation: fundamentally optimize the generation logic of WireMask and PositionMask, and innovate them from serial, module-level computation mode to unified, parallel tensor operations based on the overall layout grid, significantly reducing computational complexity and achieving more than 5 times acceleration to support the rapid training and inference of large-scale circuits.

[0010] To achieve the aforementioned objectives, this invention provides a reinforcement learning macro-layout optimization system called RDCPlace. Its core lies in constructing a collaborative technical solution that systematically addresses problems such as reward sparsity, short-sighted decision-making, and computational bottlenecks through a series of innovative modules.

[0011] This invention provides an improved reinforcement learning-based macro-layout optimization system, comprising a heuristic macro-ranking module, a reward decomposition value network, a policy network, an experience pool, a state representation module, and a density-aware reward function, wherein... The heuristic macro sorting module is set at the front of the system to determine the placement sequence of macro modules; The reward decomposition value network, the policy network, and the experience pool constitute a reinforcement learning agent, and the reward decomposition value network and the policy network work together to update the experience pool. The state representation module and the density-aware reward function constitute a layout environment, which receives the output of the reinforcement learning agent and feeds back the layout state features and rewards to the reinforcement learning agent.

[0012] Preferably, the heuristic macro sorting module uses a heuristic greedy sorting algorithm to determine the placement sequence of macro modules, taking the macro module with the largest area as the starting seed, and dynamically selecting the unplaced module with the largest area that is connected to any module in the sequence as the next placement target.

[0013] Preferably, the reward decomposition value network is related to the total state value. Perform explicit decomposition: Immediate Value Head : Use lightweight convolutional branches to evaluate the short-term impact of the direct line length benefits of the current layout action; Future Value Head : Utilize a Long Short-Term Memory (LSTM) network module to process high-level state features extracted from a shared fully connected layer, in order to infer the long-term strategic value of the current decision for the feasibility and quality of subsequent layout; Total value : Learn the comprehensive state value through a multilayer perceptron as the advantage function The foundation of computation; The training objective of the reward decomposition value network is determined by a composite loss function. The guidance states that this function simultaneously constrains the accuracy of the total value forecast and its consistency with the decomposed components: ; in, For actual returns, To balance the hyperparameters.

[0014] Preferably, the policy network is based on the current state. The algorithm generates a placement probability distribution at all legal grid locations using a dual-path feature extraction and adaptive gating architecture.

[0015] Preferably, the policy network includes: Data path: The state input is processed through a lightweight convolutional neural network to extract layout preference features guided by the wiremask. ; Prior path: Generate spatial prior features based on geometric heuristics in the canvas, local blankness ratio, and boundary distance. ; Adaptive gating mechanism: Dynamically calculates a gating value based on the signal strength of the WireMask channel. When connectivity is greater than or equal to a preset value, gating tends to follow the data path; when connectivity is less than the preset value, gating automatically switches to the prior path, ensuring that reasonable layout decisions can be made even without explicit network guidance. Strategy generation: Final placement probability distribution It is generated by the following formula: ; in, It is a learnable temperature parameter used to control the balance between exploration and exploitation.

[0016] Preferably, the state characterization module includes treating the shape of the macro module to be placed as a structural element and taking a two-dimensional matrix representing the canvas occupancy state as an input image; by performing a max pooling operation, it detects the overlap between the module and the already placed module when placing the module at all possible positions at once: when the pooling result is 1, it indicates that the position overlaps with the existing module.

[0017] Preferably, the density-aware reward function introduces a local density penalty term: ; in: The decrease in the half-circumference length is caused by the placement action. It is an indicator function. When the currently placed macro module is not connected to the previous module, that is, when the chain is broken, it takes the value 1; otherwise, it is 0. It represents the local module density around the current placement location, measured by calculating the number of placed modules within a predetermined radius centered on that location. It is the penalty coefficient.

[0018] To achieve the above objectives, the present invention also provides an improved reinforcement learning-based macro-layout optimization method, which, based on the above system, includes the following steps: S1, Data preparation and preprocessing; S2, Constructing the layout environment and training the deep reinforcement learning model; S3 uses the trained model to perform macro module layout and outputs location files.

[0019] Preferably, S1 specifically includes: S1.1: Obtain the chip circuit netlist and extract macro module information; S1.2: Perform initial sorting of macro modules.

[0020] Preferably, S2 specifically includes: S2.1: Build the layout environment; S2.2: Construct the policy network; S2.3: Construct a value network for reward decomposition; S2.4: Training the model.

[0021] The beneficial effects of the present invention include at least the following: Overall superior performance: The method proposed in this invention outperforms other methods in layout optimization across various macro circuits. The method of this invention has stable and superior layout optimization capabilities on circuits of different sizes and characteristics.

[0022] Furthermore, the parallelized mask generation engine of this invention exhibits significant advantages in computational efficiency. Under different circuit and mesh configurations, the mask generation speed of this invention is several times faster than the baseline method MaskPlace.

[0023] The macro layout optimization method provided by this invention is significantly superior to existing technical solutions in terms of core indicators such as layout computation efficiency (mask generation speed), especially when dealing with large-scale complex circuits, and effectively achieves the intended purpose of the invention. Attached Figure Description

[0024] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration: Figure 1 This is a schematic diagram of the structure of the improved reinforcement learning-based macro-layout optimization system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the improved reinforcement learning-based macro-layout optimization method according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the policy network structure of the improved reinforcement learning macro-layout optimization system according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the reward decomposition value network of the improved reinforcement learning macro-layout optimization system according to an embodiment of the present invention. Figure 5 This is a detailed diagram of the data path module in the policy network of the improved reinforcement learning macro-layout optimization system according to an embodiment of the present invention; Figure 6 This is a detailed diagram of the prior path module in the policy network of the improved reinforcement learning macro-layout optimization system according to an embodiment of the present invention. Figure 7 This is an overall architecture diagram of the reward decomposition value network in an improved reinforcement learning macro-layout optimization system according to an embodiment of the present invention. Detailed Implementation

[0025] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] See Figure 1An improved reinforcement learning-based macro-layout optimization system includes a heuristic macro-ranking module 10, a reward decomposition value network 21, a policy network 22, an experience pool 23, a state representation module 31, and a density-aware reward function 32, wherein... The heuristic macro sorting module 10 is located at the very front of the system. It receives the connection chip netlist information 11 and determines the placement sequence of the macro modules. This module runs before placement begins and is used to determine the placement sequence of the macro modules using a heuristic greedy sorting algorithm. Its process is as follows: Use the macro module with the largest area as the starting seed.

[0027] Dynamically select the unplaced module that is connected to any module in the sequence and has the largest area as the next placement target.

[0028] This method aims to cluster strongly connected modules early in the layout process, forming continuous connection chains. This maximizes the number of steps covered by the effective reward signal, thus mitigating the reward sparsity problem caused by premature breakage of connection chains at its source. The reward decomposition value network 21, the policy network 22, and the experience pool 23 form a reinforcement learning agent. The reward decomposition value network 21 and the policy network 22 work together to update the experience pool 23. The state representation module 31 and the density-aware reward function 32 constitute a layout environment, which receives the output of the reinforcement learning agent and feeds back the layout state features and rewards to the reinforcement learning agent.

[0029] See Figure 4 The return decomposition value network 21 pairs of total state values Perform explicit decomposition: Immediate Value Head : Use lightweight convolutional branches to evaluate the short-term impact of the direct line length benefits of the current layout action; Future Value Head : Utilize a Long Short-Term Memory (LSTM) network module to process high-level state features extracted from a shared fully connected layer, in order to infer the long-term strategic value of the current decision for the feasibility and quality of subsequent layout; Total value : Learn the comprehensive state value through a multilayer perceptron as the advantage function The foundation of computation; The training objective of the reward decomposition value network 21 is determined by the composite loss function. The guidance states that this function simultaneously constrains the accuracy of the total value forecast and its consistency with the decomposed components: ; in, For actual returns, To balance the hyperparameters.

[0030] See Figure 3 Policy network 22 based on the current state The algorithm generates a placement probability distribution at all legal grid locations using a dual-path feature extraction and adaptive gating architecture.

[0031] Policy network 22 includes: Data path: The state input is processed through a lightweight convolutional neural network to extract layout preference features guided by the wiremask. ; Prior path: Generate spatial prior features based on geometric heuristics in the canvas, local blankness ratio, and boundary distance. ; Adaptive gating mechanism: Dynamically calculates a gating value based on the signal strength of the WireMask channel. When connectivity is greater than or equal to a preset value, gating tends to follow the data path; when connectivity is less than the preset value, gating automatically switches to the prior path, ensuring that reasonable layout decisions can be made even without explicit network guidance. Strategy generation: Final placement probability distribution It is generated by the following formula: ; in, It is a learnable temperature parameter used to control the balance between exploration and exploitation.

[0032] The core improvement of the state representation module 31 lies in the fundamental reconstruction of the WireMask and PositionMask generation algorithms. Traditional methods typically employ nested loops to process each macromodule or each net one by one, resulting in a sharp increase in computational overhead as the problem size expands.

[0033] This invention proposes a unified parallel computing paradigm based on tensor operations, which is implemented as follows: (1) Parallel generation of PositionMask: This invention transforms the determination of position validity into an image morphology operation. Specifically, the shape of the macromodule to be placed is considered as a structuring element, and a two-dimensional matrix representing the canvas occupancy state is used as the input image. By performing a max pooling operation, the overlap between the module and existing modules can be detected at all possible positions in one go: a pooling result of 1 indicates that the position overlaps with an existing module. This process compresses the original O(M×G²) loop (where M is the number of macromodules) into a single O(G²) pooling operation and automatically handles boundary overflow cases.

[0034] (2) Parallel generation of WireMask: Traditional methods for wireframe length estimation require calculating bounding box constraints for each wireframe. This invention utilizes the broadcast mechanism of the PyTorch framework to completely eliminate these loops. First, obtain the bounding box information (min_x, max_x, min_y, max_y) and pin offsets of all relevant nets; Secondly, the bounding box position is adjusted through tensor translation operations, taking into account the pin offset of the current module; Then, using the ReLU function and broadcasting mechanism, the row and column out-of-bounds distances from all grid positions to each wireframe bounding box are calculated in parallel at once; Finally, a weighted sum is calculated based on the net weights to directly generate a complete WireMask heatmap, which guides module placement to optimize line length.

[0035] This improvement reduces the computational complexity from O(N×G²) to O(G²), (where N is the number of nets), and the actual speedup is significant due to the use of underlying optimized GPU parallel computing throughout.

[0036] The density-aware reward function 32 provides guidance and balances layout density when chain breaks. Based on the traditional line length reward, the reward function 32 of this invention introduces a local density penalty term: ; in: This is the decrease in the semi-circumference length caused by the placement action. It is an indicator function. It takes the value 1 when the currently placed macro module is not connected to the previous module (i.e., a chain break occurs), and 0 otherwise. It is the local module density around the current placement location, which is measured by calculating the number of placed modules within a predetermined radius centered on that location. It is the penalty coefficient.

[0037] This design incentivizes agents to proactively choose sparsely distributed areas when they have to break the connection chain, thereby reserving space for modules that may be connected to them later and spontaneously optimizing the overall layout density.

[0038] The entire system is based on the Proximal Policy Optimization (PPO) algorithm for end-to-end training and optimization. As an advanced policy gradient method, the PPO algorithm effectively ensures the stability and sample efficiency of the training process by introducing key technologies such as pruning mechanisms, avoiding the drastic performance fluctuations or degradation caused by excessively large policy update steps in traditional policy gradient methods. The training framework of this invention is carefully designed to fully leverage the advantages of PPO and deeply collaborate with the aforementioned innovative modules (such as the reward decomposition value network 21 and the density-aware reward function 32) to jointly constitute an efficient and stable learning system.

[0039] The core process and key design of the training framework are as follows: 1. Experience collection and data storage: Intelligent agents (i.e., policy networks 22) In the layout environment, following the sequence determined by the heuristic macro-sorting module 10, it interacts with the state representation module 31 and the parallelized mask generation engine. At each time step... The agent, based on the current state The probability distribution of the output of policy network 22 Sample a placement action After the environment performs this action, a new state is generated. And calculate the immediate reward based on the density-aware reward function 32. The experience tuples obtained from each step of the interaction. The data is stored in a fixed-size experience pool 23. This experience pool 23 helps to break the temporal correlation between data, improve sample utilization, and make the distribution of training data more stable, thereby improving the stability and efficiency of training.

[0040] 2. Dominance Function Estimation: At the start of each policy update round, a batch of empirical data is randomly sampled from the experience replay buffer. To more accurately evaluate the quality of actions, we employ the Generalized Dominance Estimation (GAE) method to calculate the dominance function. GAE introduces an attenuation factor. and This balances the bias and variance of the dominance estimate, and its calculation formula is as follows: ; in, For timing difference error, Output from the reward decomposition value network 21 of this invention. The unique value decomposition architecture in this invention ( This provides a more stable and forward-looking baseline for GAE calculations, resulting in more reliable advantage estimates. This effectively guides the direction of strategy updates.

[0041] 3. PPO Pruning Objective Function: The update objective of Policy Network 22 is to maximize the pruning objective function of PPO. This function ensures stable updates by limiting the differences between the old and new strategies: ; in, This represents the probability ratio between the old and new strategies. The parameters of the policy network before the update are 22. This is for pruning hyperparameters. The objective function, through minimization and pruning operations, prevents drastic changes in the strategy during a single update, while preserving room for performance improvement.

[0042] 4. Cooperative update of value network 21 and policy network 22: value network 21 updates by minimizing its composite loss function. To perform updates, policy network 22 maximizes... Updates are performed. In practice, we adopt an alternating update approach. In one iteration, the value network 21 is first updated multiple times using the current batch of data to better fit the value function, and then the policy network 22 is updated. The optimizers for both networks are the Adam optimizer with an adaptive learning rate, which dynamically adjusts the learning rate of each parameter based on the first and second moments of the parameter gradients, balancing rapid convergence in the early stages of training with stable fine-tuning in later stages.

[0043] Through the aforementioned systematic training framework, this invention successfully combines the stability and efficiency of the PPO algorithm with the specific characteristics of the macro-layout problem. The precise value signal provided by the reward decomposition value network 21 and the dense, semantically explicit guidance provided by the density-aware reward function 32 complement each other, jointly guiding the policy network 22 to learn a high-performance macro-layout strategy that combines immediate line length optimization capabilities with a long-term layout planning perspective.

[0044] The key technical point of this invention lies in its systematic solution to the fundamental problem of reinforcement learning in macro-layout applications through a series of collaboratively designed modules. Its core protection points are as follows: 1. Dynamically Connected Macro-Module Sorting Method: A heuristic sorting algorithm is proposed, which starts with the macro-module with the largest area and dynamically selects the unplaced module with the highest connection strength to the last module in the current sequence as the next placement object at each step. The aim is to maximize the length of the continuous connection chain and optimize the continuity of the reward signal.

[0045] 2. Density-aware composite reward function: Design a reward mechanism that rewards at a standard line length (e.g., Based on this, a local density-based approach is introduced. The penalty term, and the penalty term is indicated by a chain breakage indicator function. Control activation. This incentivizes the agent to automatically select low-density areas for placement when connectivity is interrupted, in order to achieve a spontaneous balance in layout density.

[0046] 3. Parallelized mask generation method: The calculation process of net mask and position mask is reconstructed from serial traversal based on a single module or net to a unified tensor operation based on the entire layout mesh state. Parallel computing hardware (such as GPU) is used to generate all masks at once, so as to significantly reduce the computational complexity.

[0047] 4. Policy network with adaptive gating mechanism: Design a policy network with a dual-path structure, which dynamically integrates data-driven features and spatial geometric prior features through a learnable gating signal, so that the system can follow connectivity when guided by wired network and rely on heuristic rules when guided by wireless network, thus ensuring the robustness of decision-making.

[0048] 5. Value estimation architecture based on reward decomposition: Constructing a new type of value network, the core of which lies in the total state value Explicitly decomposed into immediate value and future value And through a composite loss function (such as Joint optimization is used to stabilize training and provide long-term decision guidance. Specifically, the architecture may include computation heads corresponding to immediate, future, and total value, with the future value head preferably employing an LSTM network.

[0049] This invention also provides an improved reinforcement learning-based macro-layout optimization method. Based on the aforementioned system and a specific embodiment of the complete macro-layout optimization process based on RDCPlace, a macro-layout optimization method based on the foregoing technical solution is provided, the process of which is as follows: Figure 2 As shown, the method is explained in detail using the bigblue3 circuit in the ISPD2005 reference circuit as an example. The method specifically includes the following steps: S1: Data preparation and preprocessing; S1.1: Obtain the chip circuit netlist and extract macro module information; Input the netlist file of the BigBlue3 circuit, parse and extract the physical attributes (width, height, area), pin information, and connection relationships (net information) between all macro modules.

[0050] S1.2: Perform initial sorting of macro modules; The heuristic macro sorting module is invoked. First, the macro module with the largest area is selected as the starting point of the sequence. Then, the unplaced modules with the largest area that are connected to the current sequence modules are iteratively selected to form the final macro module placement sequence.

[0051] S2: Constructing the layout environment and training the deep reinforcement learning model; S2.1: Build the layout environment; Initialize an empty layout canvas and divide it into a 448×448 grid. Define the state space (a multi-channel tensor containing canvas channels, wire mesh mask channels, and position mask channels), the action space (all valid grid positions), and the density-aware composite reward function.

[0052] S2.2: Construct the policy network; Build as Figure 3 The dual-path policy network is shown. This network receives a state tensor, extracts features through the data path and prior path respectively, and fuses them through an adaptive gating mechanism to finally output the probability distribution at the legal positions. Figure 5 for Figure 3 Detailed diagram of the data path module in the policy network. Figure 6 for Figure 3 Detailed diagram of the prior path module in the policy network; S2.3: Construct a value network for reward decomposition; Build as Figure 4 The network shown is a value decomposition network for returns. This network receives a state tensor and estimates the immediate value, future value, and total value through different computation heads. Figure 7 for Figure 4 A detailed diagram of the overall architecture in the reward decomposition value network.

[0053] S2.4: Training the model The model is trained based on the PPO algorithm. The agent progressively places macromodules according to the sequence generated in S1.2, collects experience data, and stores it in the experience pool. When the data reaches a certain batch size, the value network is updated using a composite loss function, and the policy network is updated using the PPO objective function. The training process continues for multiple rounds (e.g., 1000 epochs) until the model converges.

[0054] S3: Use the trained model to lay out macro modules and output the location file.

[0055] The trained optimal model parameters are loaded, and inference is performed on the BigBlue3 circuit. Following the learned strategy, the model sequentially places macromodules on the canvas, ultimately outputting the coordinate positions of all macromodules to complete the layout. To verify the effectiveness of the proposed RDCPlace macro-layout optimization method, the inventors conducted extensive simulation experiments and compared its performance with several current advanced methods.

[0056] 1. Experimental Environment and Parameter Settings Operating System: Ubuntu 18.04 LTS, Programming Language: Python 3.8, Deep Learning Framework: PyTorch 1.10, Computing Architecture: CUDA 11.3, CPU: AMD EPYC 7513 32-Core Processor, Graphics Processor: NVIDIA GeForce RTX 3090 GPU (24GB RAM), System Memory: 125 GB RAM. The Adam optimizer was used during training, which adaptively adjusts the learning rate, balancing rapid convergence in the early stages of training with stable fine-tuning in later stages.

[0057] 2. Control group and dataset To comprehensively evaluate the performance of the present invention, we selected several advanced layout methods as a control group, including: GraphPlace, DeepPR, MaskPlace, and Chipformer.

[0058] The experiments used the widely adopted ISPD2005 benchmark suite, and the selected circuits covered circuit designs of varying sizes. All methods were evaluated under the same experimental settings.

[0059] 3. Experimental Data and Analysis The table below shows a comparison of the optimized half-cycle length and line length results of the proposed method and the control method on various reference circuits. All RL methods were run for 1000 training cycles, and the results were taken as the average (Mean) and standard deviation (Std) of 5 independent runs.

[0060] Table 1. Comparison of HPWL for different macro layout methods (unit: ×10^5) Layout scheme adaptec1 adaptec2 adaptec3 adaptec4 bigblue1 bigblue3 GraphPlace 30.10±2.98 351.71 ± 38.20 358.18 ± 13.93 151.42 ± 9.72 10.58±1.29 357.48±47.83 DeepPR 19.91±2.13 203.51 ± 6.27 347.16 ± 4.32 311.86± 56.74 23.33±3.65 430.48±12.18 MaskPlace 7.62±0.67 75.16 ± 4.97 100.24 ± 13.54 87.99 ± 3.25 3.04±0.06 90.04±4.83 Chipformer 6.62 ± 0.05 67.10 ± 5.46 76.70 ± 1.15 68.80 ± 1.59 2.95 ± 0.04 72.92 ± 2.56 RDCPlace 5.96 ± 0.20 36.72 ± 1.18 56.02 ± 0.66 53.09 ± 0.34 2.15 ± 0.01 41.13 ± 1.00 Analysis of Table 1 reveals the following: Superior overall performance: The method proposed in this invention outperforms other methods in layout optimization across various macrocircuits. This demonstrates that the method of this invention possesses stable and superior layout optimization capabilities across circuits of different sizes and characteristics.

[0061] Furthermore, the parallelized mask generation engine of this invention exhibits a significant advantage in terms of computational efficiency. As shown in Table 2, under different circuit and mesh configurations, the mask generation speed of this invention is several times faster than the baseline method MaskPlace.

[0062] Table 2 Comparison of mask generation times (unit: seconds / cycle) Dataset Grid size MaskPlace RDCPlace acceleration factor adaptec1 224 5.09 1.06 4.8x adaptec3 224 10.58 1.40 7.6x adaptec4 224 17.85 2.63 6.8x adaptec1 448 10.16 1.15 8.8x adaptec3 448 22.74 1.53 14.9x adaptec4 448 36.38 2.93 12.4x Conclusion: Simulation results fully demonstrate that the macro layout optimization method provided by this invention is significantly superior to existing technical solutions in terms of core indicators such as layout computation efficiency (mask generation speed), especially when dealing with large-scale complex circuits, and effectively achieves the intended purpose of the invention.

[0063] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. An improved reinforcement learning-based macro-layout optimization system, characterized in that, It includes a heuristic macro-ranking module, a reward decomposition value network, a policy network, an experience pool, a state representation module, and a density-aware reward function, among which, The heuristic macro sorting module is set at the front of the system to determine the placement sequence of macro modules; The reward decomposition value network, the policy network, and the experience pool constitute a reinforcement learning agent, and the reward decomposition value network and the policy network work together to update the experience pool. The state representation module and the density-aware reward function constitute a layout environment, which receives the output of the reinforcement learning agent and feeds back the layout state features and rewards to the reinforcement learning agent.

2. The improved reinforcement learning-based macro-layout optimization system according to claim 1, characterized in that, The heuristic macro sorting module uses a heuristic greedy sorting algorithm to determine the placement sequence of macro modules. It takes the macro module with the largest area as the starting seed and dynamically selects the unplaced module with the largest area that is connected to any module in the sequence as the next placement target.

3. The improved reinforcement learning-based macro-layout optimization system according to claim 1, characterized in that, The reward decomposition value network for total state value Perform explicit decomposition: Immediate Value Head : Use lightweight convolutional branches to evaluate the short-term impact of the direct line length benefits of the current layout action; Future Value Head : Utilize long short-term memory network modules to process high-level state features extracted from shared fully connected layers, in order to infer the long-term strategic value of current decisions for the feasibility and quality of subsequent layouts; Total value : Learn the comprehensive state value through a multilayer perceptron as the advantage function The foundation of computation; The training objective of the reward decomposition value network is determined by a composite loss function. The guidance states that this function simultaneously constrains the accuracy of the total value forecast and its consistency with the decomposed components: ； in, For actual returns, To balance the hyperparameters.

4. The improved reinforcement learning-based macro-layout optimization system according to claim 1, characterized in that, The policy network is based on the current state. The algorithm generates a placement probability distribution at all legal grid locations using a dual-path feature extraction and adaptive gating architecture.

5. The improved reinforcement learning-based macro-layout optimization system according to claim 1, characterized in that, The policy network includes: Data path: The state input is processed through a lightweight convolutional neural network to extract layout preference features guided by the wiremask. ; Prior path: Generate spatial prior features based on geometric heuristics in the canvas, local blankness ratio, and boundary distance. ; Adaptive gating mechanism: Dynamically calculates a gating value based on the signal strength of the WireMask channel. When connectivity is greater than or equal to a preset value, gating tends to follow the data path; when connectivity is less than the preset value, gating automatically switches to the prior path, ensuring that reasonable layout decisions can be made even without explicit network guidance. Strategy generation: Final placement probability distribution It is generated by the following formula: ； in, It is a learnable temperature parameter used to control the balance between exploration and exploitation.

6. The improved reinforcement learning-based macro-layout optimization system according to claim 1, characterized in that, The state characterization module includes treating the shape of the macro module to be placed as a structural element and taking a two-dimensional matrix representing the canvas occupancy state as an input image; by performing a max pooling operation, it detects the overlap between the module and the already placed module when placing the module at all possible positions at once: when the pooling result is 1, it indicates that the position overlaps with the existing module.

7. The improved reinforcement learning-based macro-layout optimization system according to claim 1, characterized in that, The density-aware reward function introduces a local density penalty term: ； in: The decrease in the half-circumference length is caused by the placement action. It is an indicator function. When the currently placed macro module is not connected to the previous module, that is, when the chain is broken, it takes the value 1; otherwise, it is 0. It represents the local module density around the current placement location, measured by calculating the number of placed modules within a predetermined radius centered on that location. It is the penalty coefficient.

8. An improved reinforcement learning-based macro-layout optimization method, characterized in that, The system based on any one of claims 1-7 includes the following steps: S1, Data preparation and preprocessing; S2, Constructing the layout environment and training the deep reinforcement learning model; S3 uses the trained model to perform macro module layout and outputs location files.

9. The improved reinforcement learning-based macro-layout optimization method according to claim 8, characterized in that, S1 specifically includes: S1.1: Obtain the chip circuit netlist and extract macro module information; S1.2: Perform initial sorting of macro modules.

10. The improved reinforcement learning-based macro-layout optimization method according to claim 8, characterized in that, S2 specifically includes: S2.1: Build the layout environment; S2.2: Construct the policy network; S2.3: Construct a value network for reward decomposition; S2.4: Training the model.

Images