An IC design optimization method and system based on reinforcement learning

Abstract

The application discloses an IC design optimization method and system based on reinforcement learning, comprising the following steps: obtaining IC data to be optimized and performing an EDA process to generate a design checkpoint that can be rolled back; performing timing analysis on the design checkpoint and merging critical path families to determine a design object to be adjusted; generating a design state vector according to design evaluation data; constructing a tree-shaped BDQN model to sequentially determine a target design checkpoint, a target critical path family and a local optimization action; copying the target design checkpoint to generate a design branch, batch-executing the local optimization action and continuing the EDA process to generate an optimized design branch; generating a reward value according to the optimized design branch and training the tree-shaped BDQN model to obtain an optimized IC design. The application realizes IC design bifurcation optimization, improves timing convergence efficiency and design optimization quality.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit design automation technology, and in particular to an IC design optimization method and system based on reinforcement learning. Background Technology

[0002] As integrated circuit scale and process complexity increase, IC design typically requires the use of EDA tools to complete synthesis, placement, clock tree synthesis, routing, and sign-off analysis. The design is then repeatedly adjusted based on timing, power consumption, area, congestion, and design rule checks. Existing optimization methods largely rely on manual experience, built-in strategies in EDA tools, or fixed script flows, making tentative adjustments to critical paths, placement parameters, cell sizes, and buffer insertions.

[0003] Existing technologies also utilize reinforcement learning for IC design optimization, treating design evaluation data as states, design adjustments as actions, and generating reward values ​​based on changes in optimized metrics. Current methods typically only continue optimizing along the most recent design result, making it difficult to revert to a more suitable design stage and re-establish optimization branches.

[0004] Existing methods often focus on patching single critical timing paths, failing to fully utilize the shared connections, start-end relationships, and spatial proximity between multiple paths. This can easily lead to repeated adjustments of the same batch of circuit instances, interconnects, or physical regions. Current BDQN structures typically use parallel Q-value headers to estimate the same set of actions, with branches not linked to design checkpoints, critical path families, and local optimization actions. This makes it difficult to simultaneously determine the optimization start point, optimization object, and optimization method, resulting in problems such as a single optimization path, fragmented path processing, and insufficient coupling between the model structure and the EDA process.

[0005] Therefore, how to provide an IC design optimization method and system based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an IC design optimization method and system based on reinforcement learning. This invention employs a tree-structured BDQN model, design checkpoint branching optimization, and critical path family batch optimization to determine the optimization starting point, optimization object, and local optimization actions of the IC design. It has the advantages of flexible optimization paths, centralized critical path processing, and high design iteration efficiency.

[0007] An IC design optimization method and system based on reinforcement learning according to an embodiment of the present invention includes the following steps: Acquire the design data and constraint data of the IC to be optimized, execute the EDA design flow, save the design checkpoints that can be rolled back at different design stages, and retain the corresponding design database, stage status and subsequent process entry for each design checkpoint. Perform a timing analysis on each design checkpoint, group key timing paths that satisfy at least one of the following relationships: connection sharing relationship, path start-end relationship, spatial proximity relationship into a critical path family, establish a one-to-many mapping between design checkpoints and critical path families, and associate paths within the same critical path family with the same batch of design objects to be adjusted. Generate a design state vector based on the design evaluation data from the design checkpoints; A tree-structured BDQN model is constructed, which includes checkpoint selection nodes, path family selection nodes, and action leaf nodes. The checkpoint selection nodes determine the target design checkpoints, the path family selection nodes determine the target critical path family in the critical path family corresponding to the target design checkpoints, and the action leaf nodes determine the local optimization actions that are applied to the design objects associated with the target critical path family. Copy the target design checkpoint to generate a design branch. Within the design branch, perform local optimization actions in batches on the target critical path family associated design objects. Continue the EDA design process from the subsequent process entry point of the target design checkpoint to generate the optimized design branch. Reward values ​​are generated based on the design evaluation data of the optimized design branch. The tree-shaped BDQN model is trained iteratively until the optimized IC design is obtained.

[0008] Optionally, saving rollbackable design checkpoints at different design stages specifically includes: Obtain the design data and constraint data of the IC to be optimized, and perform design processing according to the preset EDA stage sequence; After each design phase is completed, the design database formed in the current design phase is extracted and used as the corresponding design checkpoint. Read the design evaluation data, design completion status and phase transition status generated in the current design phase, and generate the phase status of the corresponding design checkpoints; Based on the execution position of the current design phase in the EDA design process, determine the EDA phase that needs to be executed after the current design phase, and generate the subsequent process entry point for the corresponding design checkpoint. After the target design checkpoint is selected by the tree-shaped BDQN model, the design database corresponding to the target design checkpoint is called, the stage status corresponding to the target design checkpoint is read, and the EDA design process continues to be executed according to the subsequent process entry point corresponding to the target design checkpoint. The target design checkpoint forms a design branch that can be further optimized.

[0009] Optionally, performing timing analysis for each design checkpoint specifically includes: Perform static timing analysis on the design database corresponding to each design checkpoint to extract key timing paths with timing margins less than a preset margin threshold; Analyze the circuit instances, interconnects, starting timing units, ending timing units, and physical location information traversed by each critical timing path; Generate connection sharing relationships based on the shared circuit instances and shared interconnects among different key timing paths; Generate path start-end relationships based on the cases where different key time-series paths have the same starting time-series unit, the same ending time-series unit, and the same combination of starting and ending points; Spatial proximity relationships are generated based on the fact that the physical location information corresponding to different key time-series paths are in the same physical region; Key time-series paths that satisfy at least one of the following relationships—connection sharing relationship, path start-end relationship, and spatial proximity relationship—are grouped into the same critical path family. Establish a one-to-many mapping between each design checkpoint and the corresponding generated critical path family; The circuit instances, interconnects, and physical regions that are commonly associated with each critical timing path within the same critical path family are identified as the design objects to be adjusted.

[0010] Optionally, generating the design state vector based on the design evaluation data from the design checkpoints specifically includes: Read the design evaluation data corresponding to each design checkpoint; The design evaluation data is normalized based on the constraint data to generate normalized evaluation data for the corresponding design checkpoints. The normalized evaluation data is concatenated with the stage status of the corresponding design checkpoint to generate the state sub-vector of the corresponding design checkpoint. The design state vector is generated by combining the state sub-vectors according to the stage sequence of the design checkpoints in the EDA design process. Input the design state vector into the tree-shaped BDQN model.

[0011] Optionally, constructing the tree-shaped BDQN model specifically includes: Using the design state vector as the root node input of the tree-shaped BDQN model, a checkpoint selection root node is established; Under the root node of the checkpoint selection, create first-level child nodes according to the design checkpoints. Each first-level child node is bound to the design database, stage status and subsequent process entry of a design checkpoint. Under each first-level child node, a second-level child node is established according to the one-to-many mapping of the corresponding design checkpoint. Each second-level child node is bound to a critical path family and the design object to be adjusted associated with that critical path family. Create an action leaf node under each second-level child node, and bind each action leaf node to a local optimization action that acts on the corresponding design object to be adjusted. The design state vector is input to the root node of the checkpoint. The Q value of each first-level child node is calculated, and the design checkpoint corresponding to the first-level child node with the largest Q value is selected as the target design checkpoint. The design state vector is passed to the second-level child nodes under the first-level child node corresponding to the target design checkpoint. The Q value of each second-level child node is calculated, and the critical path family corresponding to the second-level child node with the largest Q value is selected as the target critical path family. The design state vector is passed to the action leaf nodes under the corresponding second-level child nodes of the target critical path family. The Q value of each action leaf node is calculated, and the local optimization action corresponding to the action leaf node with the largest Q value is selected as the target local optimization action. The selected first-level child nodes, second-level child nodes, and action leaf nodes form a tree path. The tree path corresponds to the combined optimization decision consisting of the target design checkpoint, the target critical path family, and the target local optimization actions.

[0012] Optionally, the step of generating a design branch by copying the target design checkpoint specifically includes: Based on the tree path, call the design database corresponding to the target design checkpoint and copy the design database corresponding to the target design checkpoint as a design branch; Based on the target critical path family, locate the associated design objects to be adjusted, and perform target local optimization actions in batches on the design objects to be adjusted within the design branch to generate a branch design database; Continue executing the EDA design process according to the target design checkpoints and the corresponding subsequent process entry points, generate optimized design branches, and obtain the design evaluation data, design completion status, and stage transition status of the optimized design branches. When the design completion status of the optimized design branch meets the stage completion conditions, the stage connection status meets the subsequent execution conditions, and the design evaluation data meets the preset conditions for continued optimization, the optimized design branch will be used as an optional checkpoint. In subsequent optimization rounds, existing design checkpoints and optional checkpoints are used together as checkpoint selection objects for the tree-structured BDQN model. The tree-structured BDQN model redetermines the target design checkpoint from the existing design checkpoints and optional checkpoints. The process generates several optimized design branches in a loop, and selects the optimized IC design that satisfies the constraint data from all optimized design branches.

[0013] Optionally, the iterative training of the tree-structured BDQN model specifically includes: Read the design evaluation data of the optimized design branch and the target design checkpoint, calculate the change in the design evaluation data of the optimized design branch relative to the target design checkpoint, and generate a reward value. The design state vector, tree path, reward value, and the next design state vector corresponding to the optimized design branch are used to generate training samples. Update the Q values ​​of the selected first-level child nodes, second-level child nodes, and action leaf nodes in the tree-structured BDQN model based on the training samples; When the optimized design branch satisfies the constraint data, the optimized design branch will be output as the optimized IC design. When the optimized design branch does not meet the constraint data but meets the preset conditions for continuing optimization, the optimized design branch will be used as an optional checkpoint for subsequent optimization rounds, and the training of the tree-shaped BDQN model will continue.

[0014] An IC design optimization system based on reinforcement learning according to an embodiment of the present invention includes: The EDA flow control module is used to call EDA tools to execute the design flow of the IC to be optimized, and to form design checkpoints, design database, stage status and subsequent process entry points at the end of the design phase. The critical path family construction module is used to perform time series analysis on the design database corresponding to the design checkpoints, merge critical time series paths into critical path families, and determine the design objects to be adjusted associated with the critical path families. The state construction module is used to generate design state vectors based on design evaluation data and stage states from design checkpoints. The tree-structured BDQN model module is used to set the root node, first-level child nodes, second-level child nodes, and action leaf nodes for checkpoint selection. It generates a tree path consisting of target design checkpoints, target critical path families, and target local optimization actions based on the design state vector. The branch optimization execution module is used to copy the target design checkpoint to generate design branches. Within the design branches, it performs local optimization actions on the target critical path family-related design objects to be adjusted in batches, and generates optimized design branches according to the subsequent process entry. The training and output module is used to generate reward values ​​based on the design evaluation data of the optimized design branch, update the nodes selected by the tree path in the tree BDQN model module, and output the optimized IC design that meets the constraint data from the optimized design branch.

[0015] The beneficial effects of this invention are: (1) This invention saves rollback design checkpoints in multiple EDA design stages and uses design checkpoints as checkpoint selection objects in the tree-shaped BDQN model, so that the optimization process can re-form design branches from different design stages instead of just continuing along the last optimization result, thereby improving the flexibility of IC design optimization path.

[0016] (2) By performing time-series analysis on each design checkpoint, the present invention merges the critical time-series paths that satisfy the connection sharing relationship, path start-end relationship and spatial proximity relationship into a critical path family, and performs local optimization actions on the design objects to be adjusted associated with the critical path family in batches, thereby reducing the repeated repair of a single critical path and improving the consistency of critical path optimization.

[0017] (3) This invention constructs a tree-shaped BDQN model that includes a root node for checkpoint selection, first-level child nodes, second-level child nodes and action leaf nodes, so that the tree path corresponds simultaneously to the target design checkpoint, the target critical path family and the target local optimization action, and the decision results of the reinforcement learning model are directly mapped to the optimization starting point, optimization object and optimization method in the IC design process, thereby improving the matching degree between the model decision and the EDA execution process. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an IC design optimization method based on reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram of an IC design optimization system based on reinforcement learning proposed in this invention; Figure 3 This is a schematic diagram of design checkpoint bifurcation optimization, a reinforcement learning-based IC design optimization method proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 An IC design optimization method and system based on reinforcement learning, comprising the following steps: Acquire the design data and constraint data of the IC to be optimized, execute the EDA design flow, save the design checkpoints that can be rolled back at different design stages, and retain the corresponding design database, stage status and subsequent process entry for each design checkpoint. Perform a timing analysis on each design checkpoint, group key timing paths that satisfy at least one of the following relationships: connection sharing relationship, path start-end relationship, spatial proximity relationship into a critical path family, establish a one-to-many mapping between design checkpoints and critical path families, and associate paths within the same critical path family with the same batch of design objects to be adjusted. Generate a design state vector based on the design evaluation data from the design checkpoints; A tree-structured BDQN model is constructed, which includes checkpoint selection nodes, path family selection nodes, and action leaf nodes. The checkpoint selection nodes determine the target design checkpoints, the path family selection nodes determine the target critical path family in the critical path family corresponding to the target design checkpoints, and the action leaf nodes determine the local optimization actions that are applied to the design objects associated with the target critical path family. Copy the target design checkpoint to generate a design branch. Within the design branch, perform local optimization actions in batches on the target critical path family associated design objects. Continue the EDA design process from the subsequent process entry point of the target design checkpoint to generate the optimized design branch. Reward values ​​are generated based on the design evaluation data of the optimized design branch. The tree-shaped BDQN model is trained iteratively until the optimized IC design is obtained.

[0021] In this embodiment, saving reversible design checkpoints at different design stages specifically includes: The design data and constraint data of the IC to be optimized are obtained, and the design processing is performed according to the preset EDA stage sequence. The design data of the IC to be optimized may include one or more of the following: RTL design file, gate-level netlist, standard cell library file, macro cell library file, and layout constraint file. The constraint data may include one or more of the following: clock constraint, area constraint, power consumption constraint, routing constraint, and process rule constraint. The preset EDA stage sequence can be set to logic synthesis stage, placement stage, clock tree synthesis stage, routing stage, and approval analysis stage. Alternatively, at least two consecutive stages can be selected as the preset EDA stage sequence according to the design flow of the IC to be optimized. After each design phase is completed, the design database formed in the current design phase is extracted and used as the corresponding design checkpoint. The design database refers to the data set formed by the EDA tool after the current design phase is completed, which can be reloaded and subsequent design processing can continue. Read the design evaluation data, design completion status, and stage transition status generated in the current design phase, and generate the stage status of the corresponding design checkpoints; the design completion status is used to characterize whether the current design phase has met the execution conditions for entering the next design phase; the stage transition status is used to characterize the data handover conditions between the current design phase and the subsequent design phase, which may include at least one of the following: whether the input files required for the subsequent phase have been generated, whether the constraint files have been updated, whether the design database can be loaded, whether the entry command for the subsequent phase can be executed, and whether the output data of the current phase matches the input requirements of the subsequent phase; Based on the execution position of the current design phase in the EDA design flow, determine the EDA phase that needs to be executed after the current design phase, and generate the subsequent process entry point for the corresponding design checkpoint; the subsequent process entry point refers to the starting phase and execution configuration required to continue the EDA design flow from the current design checkpoint. After the target design checkpoint is selected by the tree-shaped BDQN model, the design database corresponding to the target design checkpoint is called, the stage status corresponding to the target design checkpoint is read, and the EDA design process continues to be executed according to the subsequent process entry point corresponding to the target design checkpoint. The target design checkpoint forms a design branch that can be further optimized.

[0022] In this embodiment, performing timing analysis for each design checkpoint specifically includes: Static timing analysis is performed on the design database corresponding to each design checkpoint to extract critical timing paths with timing margins less than a preset margin threshold. The timing margin can be represented by the path margin value in the static timing analysis report, and the preset margin threshold can be set to 0.1ns. When the path margin is less than 0ns, the critical timing path is a timing violation path. When the path margin is greater than or equal to 0ns and less than the preset margin threshold, the critical timing path is a critical timing path. Analyze the circuit instance, interconnect, starting timing unit, ending timing unit, and physical location information of each critical timing path; the starting timing unit and ending timing unit can be one of registers, latches, memory ports, or input / output ports; the physical location information can be represented by at least one of the following: the coordinates of the circuit instance in the layout coordinate system, the bounding rectangle area corresponding to the critical timing path, and the wiring area crossed by the interconnect. A connection sharing relationship is generated based on the shared circuit instances and shared interconnects between different critical timing paths; a sharing ratio threshold is set, and a connection sharing relationship is generated when the ratio of the number of shared circuit instances between two critical timing paths to the number of circuit instances in the shorter path of the two critical timing paths is greater than or equal to 20%. Generate path start-end relationships based on the cases where different key time-series paths have the same starting time-series unit, the same ending time-series unit, and the same combination of starting and ending points; Spatial proximity relationships are generated based on the physical location information corresponding to different key timing paths being in the same physical region. The same physical region can be determined by dividing the layout grid, dividing the chip layout area into several rectangular grids. When the outer rectangular regions corresponding to two key timing paths fall into the same grid, spatial proximity relationships are generated. Key time-series paths that satisfy at least one of the following relationships—connection sharing, path start-end, and spatial proximity—are grouped into the same key path family. When merging key path families, each key time-series path can be treated as a node, and the connection sharing, path start-end, and spatial proximity relationships can be treated as edges between nodes to form a path relationship graph. The path relationship graph is then partitioned into connected components, and key time-series paths within the same connected component are grouped into the same key path family. The number of key time-series paths within each key path family can be set to no less than two. When a single key time-series path does not form any relationship with other key time-series paths, that single key time-series path can be grouped into a separate key path family. Establish a one-to-many mapping between each design checkpoint and the corresponding generated critical path family; The circuit instances, interconnects, and physical regions that are commonly associated with each critical timing path within the same critical path family are identified as the design objects to be adjusted.

[0023] In this embodiment, generating the design state vector based on the design evaluation data of the design checkpoints specifically includes: Read the design evaluation data corresponding to each design checkpoint; The design evaluation data is normalized based on the constraint data to generate normalized evaluation data for the corresponding design checkpoints. The normalized evaluation data is concatenated with the stage status of the corresponding design checkpoint to generate the state sub-vector of the corresponding design checkpoint. The design state vector is generated by combining the state sub-vectors according to the stage sequence of the design checkpoints in the EDA design process. Input the design state vector into the tree-shaped BDQN model.

[0024] In this embodiment, constructing the tree-shaped BDQN model specifically includes: Using the design state vector as the root node input of the tree-shaped BDQN model, a checkpoint selection root node is established; after receiving the design state vector, the checkpoint selection root node outputs the Q value corresponding to each design checkpoint. Under the root node of the checkpoint selection, create first-level child nodes according to the design checkpoints. Each first-level child node is bound to the design database, stage status and subsequent process entry of a design checkpoint; binding means establishing a fixed correspondence between the first-level child node and the design checkpoint. Under each primary child node, secondary child nodes are created according to the one-to-many mapping of the corresponding design checkpoint. Each secondary child node is bound to a critical path family and the design objects to be adjusted associated with that critical path family. The number of secondary child nodes is consistent with the number of critical path families generated under the corresponding design checkpoint. Under each secondary child node, an action leaf node is created. Each action leaf node is bound to a local optimization action that acts on the corresponding design object to be adjusted. A local optimization action is an EDA optimization action whose scope is limited to the design object to be adjusted corresponding to the secondary child node. Local optimization actions can be at least one of the following: cell size adjustment, threshold voltage replacement, buffer insertion, local layout density adjustment, routing layer preference adjustment, and local logic refactoring. The specific number is determined by the type of design object to be adjusted and the EDA stage in which the target design checkpoint is located. The design state vector is input to the root node of the checkpoint. The Q value of each first-level child node is calculated, and the design checkpoint corresponding to the first-level child node with the largest Q value is selected as the target design checkpoint. The design state vector is passed to the second-level child nodes under the first-level child node corresponding to the target design checkpoint. The Q value of each second-level child node is calculated, and the critical path family corresponding to the second-level child node with the largest Q value is selected as the target critical path family. The tree-shaped BDQN model only selects second-level child nodes under the first-level child node corresponding to the target design checkpoint, and does not calculate the critical path family under the unselected first-level child node. The design state vector is passed to the action leaf nodes under the corresponding second-level child nodes of the target critical path family. The Q value of each action leaf node is calculated, and the local optimization action corresponding to the action leaf node with the largest Q value is selected as the target local optimization action. After the target critical path family is determined, the tree-shaped BDQN model only selects action leaf nodes under the corresponding second-level child nodes of the target critical path family. The target local optimization action is applied to the design objects to be adjusted associated with the target critical path family, and is not applied to the critical path families that are not selected. The selected first-level child nodes, second-level child nodes, and action leaf nodes form a tree path. The tree path corresponds to the combined optimization decision consisting of the target design checkpoint, the target critical path family, and the target local optimization actions.

[0025] In this embodiment, generating a design branch by copying the target design checkpoint specifically includes: The tree path calls the design database corresponding to the target design checkpoint and copies the design database corresponding to the target design checkpoint as a design branch. The tree path is a combination optimization decision consisting of the target design checkpoint, the target critical path family, and the target local optimization actions. Copying means generating an independent design copy with the design database corresponding to the target design checkpoint as the starting data. The independent design copy has the same starting circuit data, physical data, and constraint data as the design database corresponding to the target design checkpoint. The local optimization actions performed on the independent design copy do not cover the design database corresponding to the target design checkpoint. Based on the target critical path family, locate the design objects to be adjusted that are associated with it, and perform target local optimization actions in batches on the design objects to be adjusted within the design branch to generate a branch design database; batch execution refers to generating and executing the same type of local optimization action at once for the design objects to be adjusted that are associated with multiple critical time-series paths within the target critical path family. The EDA design process continues from the entry point corresponding to the target design checkpoint, generating an optimized design branch and obtaining its design evaluation data, design completion status, and stage transition status. If the target design checkpoint corresponds to the layout stage, clock tree synthesis, routing, and approval analysis continue from the layout stage onwards. If the target design checkpoint corresponds to the clock tree synthesis stage, routing and approval analysis continue from the clock tree synthesis stage onwards. The optimized design branch is the updated design version formed after the design branch performs the target local optimization actions and subsequent EDA design processes. When the optimized design branch meets the stage completion condition, the stage transition condition meets the subsequent execution condition, and the design evaluation data meets the preset conditions for continued optimization, the optimized design branch will be used as an optional checkpoint. The stage completion condition is set to the normal termination of the EDA tool. The preset conditions for continued optimization can be set to the design evaluation data not exceeding the preset elimination threshold and at least one evaluation indicator improving relative to the target design checkpoint. The preset elimination threshold includes at least one of the following: total negative timing margin not exceeding 150% of the total negative timing margin corresponding to the target design checkpoint, power consumption not exceeding 130% of the upper limit of power consumption in the constraint data, and area not exceeding 130% of the upper limit of area in the constraint data. In subsequent optimization rounds, existing design checkpoints and optional checkpoints are used together as checkpoint selection objects for the tree-structured BDQN model. The tree-structured BDQN model redetermines the target design checkpoint from the existing design checkpoints and optional checkpoints. Satisfying the constraint data means that the timing, power consumption, area, congestion, and design rule check results of the optimized design branch all meet the corresponding constraints set by the constraint data. The process generates several optimized design branches in a loop, and selects the optimized IC design that satisfies the constraint data from all optimized design branches.

[0026] In this embodiment, the iterative training of the tree-shaped BDQN model specifically includes: Read the design evaluation data of the optimized design branch and the target design checkpoint, calculate the change in design evaluation data of the optimized design branch relative to the target design checkpoint, and generate a reward value; the change in design evaluation data is calculated according to the optimization direction of the evaluation indicators; the reward value is composed of the weighted sum of the changes in timing evaluation, power consumption evaluation, area evaluation, congestion evaluation, design rule evaluation, and the changes corresponding to the design completion status and stage transition status, with a weight sum of 1; The design state vector, tree path, reward value, and the next design state vector corresponding to the optimized design branch are used to generate training samples; the training samples are generated from the design evaluation data and stage states of the optimized design branch; the training samples can be represented as (S, P, R, S'), where S is the design state vector, P is the tree path, R is the reward value, and S' is the next design state vector. The Q-values ​​of the selected first-level child nodes, second-level child nodes, and action leaf nodes in the tree-structured BDQN model are updated based on the training samples. When updating the Q-values, only the first-level child nodes, second-level child nodes, and action leaf nodes corresponding to the tree paths in the training samples are updated; nodes not selected by the tree paths are not updated based on the current training samples. For the selected nodes, the reward value is multiplied by the maximum Q-value of the selectable nodes corresponding to the next design state vector by a discount factor, and the result is added to obtain the target Q-value. The difference between the target Q-value and the current Q-value of the selected node is calculated. The parameters of the tree-structured BDQN model are updated by backpropagation based on this difference. The discount factor is 0.9, and the learning rate is 0.001. When the optimized design branch satisfies the constraint data, the optimized design branch will be output as the optimized IC design. When the optimized design branch does not meet the constraint data but meets the preset conditions for continued optimization, the optimized design branch will be used as an optional checkpoint for subsequent optimization rounds, and the training of the tree-shaped BDQN model will continue. The preset conditions for continued optimization can be determined based on the stage completion conditions, subsequent execution conditions, and design evaluation data. If the optimized design branch does not meet the constraint data, but the design completion status meets the stage completion conditions, the stage transition status meets the subsequent execution conditions, and at least one design evaluation indicator improves compared to the target design checkpoint, then the optimized design branch will be used as an optional checkpoint. If the optimized design branch does not meet the stage completion conditions or does not meet the subsequent execution conditions, then it will not be used as an optional checkpoint for subsequent optimization rounds.

[0027] An IC design optimization system based on reinforcement learning according to an embodiment of the present invention includes: The EDA flow control module is used to call EDA tools to execute the design flow of the IC to be optimized, and to form design checkpoints, design database, stage status and subsequent process entry points at the end of the design phase. The critical path family construction module is used to perform time series analysis on the design database corresponding to the design checkpoints, merge critical time series paths into critical path families, and determine the design objects to be adjusted associated with the critical path families. The state construction module is used to generate design state vectors based on design evaluation data and stage states from design checkpoints. The tree-structured BDQN model module is used to set the root node, first-level child nodes, second-level child nodes, and action leaf nodes for checkpoint selection. It generates a tree path consisting of target design checkpoints, target critical path families, and target local optimization actions based on the design state vector. The branch optimization execution module is used to copy the target design checkpoint to generate design branches. Within the design branches, it performs local optimization actions on the target critical path family-related design objects to be adjusted in batches, and generates optimized design branches according to the subsequent process entry. The training and output module is used to generate reward values ​​based on the design evaluation data of the optimized design branch, update the nodes selected by the tree path in the tree BDQN model module, and output the optimized IC design that meets the constraint data from the optimized design branch.

[0028] Example 1: To verify the feasibility of this invention in practice, it was applied to a scenario of large-scale high-performance integrated circuit design optimization. In this scenario, the IC design flow includes logic synthesis, placement, clock tree synthesis, routing, and checklist analysis, each stage generating a large amount of design data and constraint information. In traditional methods, design optimization often relies on manual experience or fixed rule scripts, with critical path optimization handled at single points, resulting in low optimization efficiency, scattered path processing, and repetitive timing violation repairs. To address this issue, this invention introduces a reinforcement learning-based IC design optimization method, constructs a tree-structured BDQN model, and sets multiple rollbackable design checkpoints in the EDA flow to achieve cross-stage branching optimization and batch processing of critical path families, thereby improving optimization efficiency and design quality.

[0029] In practical applications, design and constraint data of the IC to be optimized are acquired, and logic synthesis, placement, clock tree synthesis, and routing are performed according to the EDA process. Design checkpoints are generated at the end of each stage, and the design database, stage status, and subsequent process entry points are recorded. Static timing analysis is performed on each design checkpoint to extract critical paths with timing margins less than a preset threshold. These paths are then grouped into critical path families based on their connection sharing relationships, start-end relationships, and spatial proximity. Design objects within the critical path families are then uniformly optimized in batches, improving the consistency of circuit instances, interconnects, and physical regions within the same batch.

[0030] Design state vectors are generated based on the design evaluation data at each design checkpoint and input into a tree-structured BDQN model. The model determines target design checkpoints through checkpoint selection nodes, target critical path families through path family selection nodes, and local optimization actions through action leaf nodes. Design branches are generated for each target design checkpoint, and local optimization actions are executed in batches within these branches. Simultaneously, the EDA process continues according to subsequent process entry points, forming optimized design branches. The design evaluation data from these optimized branches is used to calculate reward values. The tree-structured BDQN model is then iteratively trained until the design meets timing, power, area, and design rule constraints, ultimately yielding the optimized IC design.

[0031] Traditional methods mainly rely on human experience and built-in rules of EDA tools. They gradually adjust the critical path, layout parameters, cell size and buffer insertion through fixed scripts and single-point repairs. The optimization process follows the most recent design result and lacks cross-stage forking and batch processing mechanisms, resulting in low optimization efficiency, many repeated repairs and slow timing convergence.

[0032] Table 1: Comparison of Reinforcement Learning-Based IC Design Optimization Methods with Traditional Optimization Methods

[0033] The comparative analysis shows that the method of this invention outperforms traditional optimization methods in all key indicators. The reduction in total negative timing margin increased from 25% to 48%, demonstrating significant effectiveness in timing optimization. Power consumption and area optimization improved by 12% and 7% respectively, reflecting the effective control of resource usage by local optimization actions. The number of design rule violations decreased by more than 60%, and the number of critical path repairs increased by more than 50%, showcasing the advantages of batch critical path family optimization. By performing branching optimization at multiple design checkpoints, the number of EDA iterations was reduced by 40%, improving the efficiency of the optimization process. The setting of rollbackable design checkpoints and tree-structured BDQN model training rounds provides greater flexibility and controllability in the optimization process, offering strong technical support for the automated optimization of large-scale IC designs.

[0034] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An IC design optimization method based on reinforcement learning, characterized in that, Includes the following steps: Acquire the design data and constraint data of the IC to be optimized, execute the EDA design flow, save the design checkpoints that can be rolled back at different design stages, and retain the corresponding design database, stage status and subsequent process entry for each design checkpoint. Perform a timing analysis on each design checkpoint, group key timing paths that satisfy at least one of the following relationships: connection sharing relationship, path start-end relationship, spatial proximity relationship into a critical path family, establish a one-to-many mapping between design checkpoints and critical path families, and associate paths within the same critical path family with the same batch of design objects to be adjusted. Generate a design state vector based on the design evaluation data from the design checkpoints; A tree-structured BDQN model is constructed, which includes checkpoint selection nodes, path family selection nodes, and action leaf nodes. The checkpoint selection nodes determine the target design checkpoints, the path family selection nodes determine the target critical path family in the critical path family corresponding to the target design checkpoints, and the action leaf nodes determine the local optimization actions that are applied to the design objects associated with the target critical path family. Copy the target design checkpoint to generate a design branch. Within the design branch, perform local optimization actions in batches on the target critical path family associated design objects. Continue the EDA design process from the subsequent process entry point of the target design checkpoint to generate the optimized design branch. The process of generating design branches by replicating target design checkpoints specifically includes: Based on the tree path, call the design database corresponding to the target design checkpoint and copy the design database corresponding to the target design checkpoint as a design branch; Based on the target critical path family, locate the associated design objects to be adjusted, and perform target local optimization actions in batches on the design objects to be adjusted within the design branch to generate a branch design database; Continue executing the EDA design process according to the target design checkpoints and the corresponding subsequent process entry points, generate optimized design branches, and obtain the design evaluation data, design completion status, and stage transition status of the optimized design branches. When the design completion status of the optimized design branch meets the stage completion conditions, the stage connection status meets the subsequent execution conditions, and the design evaluation data meets the preset conditions for continued optimization, the optimized design branch will be used as an optional checkpoint. In subsequent optimization rounds, existing design checkpoints and optional checkpoints are used together as checkpoint selection objects for the tree-structured BDQN model. The tree-structured BDQN model redetermines the target design checkpoint from the existing design checkpoints and optional checkpoints. The process involves iteratively generating several optimized design branches, and then selecting the optimized IC design that satisfies the constraint data from all optimized design branches. Reward values ​​are generated based on the design evaluation data of the optimized design branch. The tree-shaped BDQN model is trained iteratively until the optimized IC design is obtained.

2. The IC design optimization method based on reinforcement learning according to claim 1, characterized in that, The specific provisions for saving reversible design checkpoints at different design stages include: Obtain the design data and constraint data of the IC to be optimized, and perform design processing according to the preset EDA stage sequence; After each design phase is completed, the design database formed in the current design phase is extracted and used as the corresponding design checkpoint. Read the design evaluation data, design completion status and phase transition status generated in the current design phase, and generate the phase status of the corresponding design checkpoints; Based on the execution position of the current design phase in the EDA design process, determine the EDA phase that needs to be executed after the current design phase, and generate the subsequent process entry point for the corresponding design checkpoint. After the target design checkpoint is selected by the tree-shaped BDQN model, the design database corresponding to the target design checkpoint is called, the stage status corresponding to the target design checkpoint is read, and the EDA design process continues to be executed according to the subsequent process entry point corresponding to the target design checkpoint. The target design checkpoint forms a design branch that can be further optimized.

3. The IC design optimization method based on reinforcement learning according to claim 2, characterized in that, The timing analysis performed on each design checkpoint specifically includes: Perform static timing analysis on the design database corresponding to each design checkpoint to extract key timing paths with timing margins less than a preset margin threshold; Analyze the circuit instances, interconnects, starting timing units, ending timing units, and physical location information traversed by each critical timing path; Generate connection sharing relationships based on the shared circuit instances and shared interconnects among different key timing paths; Generate path start-end relationships based on the cases where different key time-series paths have the same starting time-series unit, the same ending time-series unit, and the same combination of starting and ending points; Spatial proximity relationships are generated based on the fact that the physical location information corresponding to different key time-series paths are in the same physical region; Key time-series paths that satisfy at least one of the following relationships—connection sharing relationship, path start-end relationship, and spatial proximity relationship—are grouped into the same critical path family. Establish a one-to-many mapping between each design checkpoint and the corresponding generated critical path family; The circuit instances, interconnects, and physical regions that are commonly associated with each critical timing path within the same critical path family are identified as the design objects to be adjusted.

4. The IC design optimization method based on reinforcement learning according to claim 3, characterized in that, The process of generating a design state vector based on the design evaluation data from design checkpoints specifically includes: Read the design evaluation data corresponding to each design checkpoint; The design evaluation data is normalized based on the constraint data to generate normalized evaluation data for the corresponding design checkpoints. The normalized evaluation data is concatenated with the stage status of the corresponding design checkpoint to generate the state sub-vector of the corresponding design checkpoint. The design state vector is generated by combining the state sub-vectors according to the stage sequence of the design checkpoints in the EDA design process. Input the design state vector into the tree-shaped BDQN model.

5. The IC design optimization method based on reinforcement learning according to claim 4, characterized in that, The construction of the tree-shaped BDQN model specifically includes: Using the design state vector as the root node input of the tree-shaped BDQN model, a checkpoint selection root node is established; Under the root node of the checkpoint selection, create first-level child nodes according to the design checkpoints. Each first-level child node is bound to the design database, stage status and subsequent process entry of a design checkpoint. Under each first-level child node, a second-level child node is established according to the one-to-many mapping of the corresponding design checkpoint. Each second-level child node is bound to a critical path family and the design object to be adjusted associated with that critical path family. Create an action leaf node under each second-level child node, and bind each action leaf node to a local optimization action that acts on the corresponding design object to be adjusted. The design state vector is input to the root node of the checkpoint. The Q value of each first-level child node is calculated, and the design checkpoint corresponding to the first-level child node with the largest Q value is selected as the target design checkpoint. The design state vector is passed to the second-level child nodes under the first-level child node corresponding to the target design checkpoint. The Q value of each second-level child node is calculated, and the critical path family corresponding to the second-level child node with the largest Q value is selected as the target critical path family. The design state vector is passed to the action leaf nodes under the corresponding second-level child nodes of the target critical path family. The Q value of each action leaf node is calculated, and the local optimization action corresponding to the action leaf node with the largest Q value is selected as the target local optimization action. The selected first-level child nodes, second-level child nodes, and action leaf nodes form a tree path. The tree path corresponds to the combined optimization decision consisting of the target design checkpoint, the target critical path family, and the target local optimization actions.

6. The IC design optimization method based on reinforcement learning according to claim 5, characterized in that, The recurrent training tree-structured BDQN model specifically includes: Read the design evaluation data of the optimized design branch and the target design checkpoint, calculate the change in the design evaluation data of the optimized design branch relative to the target design checkpoint, and generate a reward value. The design state vector, tree path, reward value, and the next design state vector corresponding to the optimized design branch are used to generate training samples. Update the Q values ​​of the selected first-level child nodes, second-level child nodes, and action leaf nodes in the tree-structured BDQN model based on the training samples; When the optimized design branch satisfies the constraint data, the optimized design branch will be output as the optimized IC design. When the optimized design branch does not meet the constraint data but meets the preset conditions for continuing optimization, the optimized design branch will be used as an optional checkpoint for subsequent optimization rounds, and the training of the tree-shaped BDQN model will continue.

7. An IC design optimization system based on reinforcement learning, applied to the IC design optimization method based on reinforcement learning as described in any one of claims 1 to 6, characterized in that, include: The EDA flow control module is used to call EDA tools to execute the design flow of the IC to be optimized, and to form design checkpoints, design database, stage status and subsequent process entry points at the end of the design phase. The critical path family construction module is used to perform time series analysis on the design database corresponding to the design checkpoints, merge critical time series paths into critical path families, and determine the design objects to be adjusted associated with the critical path families. The state construction module is used to generate design state vectors based on design evaluation data and stage states from design checkpoints. The tree-structured BDQN model module is used to set the root node, first-level child nodes, second-level child nodes, and action leaf nodes for checkpoint selection. It generates a tree path consisting of target design checkpoints, target critical path families, and target local optimization actions based on the design state vector. The branch optimization execution module is used to copy the target design checkpoint to generate design branches. Within the design branches, it performs local optimization actions on the target critical path family-related design objects to be adjusted in batches, and generates optimized design branches according to the subsequent process entry. The training and output module is used to generate reward values ​​based on the design evaluation data of the optimized design branch, update the nodes selected by the tree path in the tree BDQN model module, and output the optimized IC design that meets the constraint data from the optimized design branch.

Images