A model training method, integrated circuit layout method and device

Abstract

The present invention relates to the field of integrated circuit layout technology, specifically to a model training method, an integrated circuit layout method, and an apparatus, designed to address the instability and inefficiency of existing simulated annealing for integrated circuit layout. The present invention processes circuit component data, eliminating interference caused by the varying specifications of standard cells in the netlist data, making it easier to extract relevant features and more effectively satisfying network input. In terms of network structure, the strategy network module is combined with the random network distillation module to assign nearly equal probabilities to actions with similar state values ​​while exploring randomization, avoiding the suboptimal consequences of repeatedly selecting the same action. Furthermore, by maximizing rewards and abandoning clearly unpromising layout solutions, the present invention can explore more effective layout solutions and improve exploration efficiency compared to traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of integrated circuit layout technology, and in particular to a model training method, and a target layout scheme exploration model obtained based on the training method, and also to an integrated circuit layout method using the model and an integrated circuit layout device using the integrated circuit layout method. Background Art

[0002] As integrated circuits (ICs) become increasingly complex following Moore's Law, IC system design becomes increasingly challenging. Electronic Design Automation (EDA) tools play a leading role in addressing these IC design challenges.

[0003] Physical IC design is one of the most critical and complex steps in EDA tools. It determines the quality of the final result. Layout is one of the most time-consuming and challenging stages of physical design. The layout step involves placing a set of instances from the netlist onto the chip canvas so that performance-related objectives such as power, performance, and area (PPA) can be minimized while satisfying constraints such as routing congestion and density. Due to its critical role in IC design, layout has been extensively studied for decades.

[0004] However, the most commonly used heuristic algorithm for integrated circuit layout is simulated annealing, which is unstable and inefficient due to its long processing time, inefficient layout structure, and large number of dead zones. Summary of the Invention

[0005] Based on this, it is necessary to provide a model training method, an integrated circuit layout method and an apparatus to address the instability and inefficiency problems existing when using simulated annealing for integrated circuit layout.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] In a first aspect, the present invention discloses a model training method for training an initial layout solution exploration model to obtain a trained target layout solution exploration model.

[0008] The model training method includes the following steps:

[0009] S1.1, build an initial layout solution exploration model; the initial layout solution exploration model includes a policy network module, an environment setting module, an experience pool module, a random network distillation module, an online criticism network module, and a target criticism network module;

[0010] S1.2, obtaining a training data set; wherein the training data set is a random netlist including a plurality of components;

[0011] S1.3, perform multiple rounds of iterative training on the initial layout solution exploration model based on the training dataset;

[0012] Among them, the environment setting module is used to calculate a part of the data required for model training and store it in the experience pool module; the random network distillation module is used to calculate another part of the data required for model training and store it in the experience pool module; the policy network module is used to make decisions and determine what actions the environment setting module should take; the online criticism network module and the target criticism network module assist the policy network module in making decisions based on the sampled data of the experience pool module;

[0013] S1.4, when the stopping condition is met, stop training and obtain the trained target layout solution exploration model.

[0014] The implementation of this model training method is based on the method or process of the embodiment of the present disclosure.

[0015] In a second aspect, the present invention discloses a model training device, which uses the model training method disclosed in the first aspect.

[0016] The model training device includes: an initial model building module, a training data acquisition module, a model training module, and a target model building module.

[0017] The initial model building module is used to build an initial layout solution exploration model. The training data acquisition module is used to obtain a training dataset. The model training module is used to iteratively train the initial layout solution exploration model based on the training dataset. The target model building module is used to stop training when a stopping condition is met, resulting in a trained target layout solution exploration model.

[0018] The implementation of this model training device is based on the method or process of the embodiment of the present disclosure.

[0019] In a third aspect, the present invention discloses an integrated circuit layout method, comprising the following steps:

[0020] S2.1, obtaining circuit component data and processing it to obtain a component netlist data set;

[0021] S2.2, using the target layout solution exploration model to perform I rounds of processing on the component netlist dataset to obtain a preliminary layout solution; wherein the target layout solution exploration model is obtained using the model training method disclosed in the first aspect;

[0022] The strategy network module obtains the current component processing value s of the j+1th processing round j ″, calculate the current action processing value a j″;

[0023] The environment setting module receives j ″、a j ″, calculate the next component processing value s j+1 ″, and adjust the layout scheme code; where j∈[0, I]; if j=0, the current component processing value s0″ of the first processing round is randomly obtained from the component netlist data set; if j>0, the current component processing value s0″ of the j+1th processing round is randomly obtained from the component netlist data set; j ″ is the next component processing value s obtained in the jth processing round j ″;

[0024] The component netlist data set is laid out according to the preliminary layout scheme code obtained after the I-round processing to obtain a preliminary layout scheme;

[0025] S2.3, taking the preliminary layout plan as the initial state, adjusting it through simulated annealing to obtain the final layout plan.

[0026] The integrated circuit layout method is implemented according to the method or process of the embodiment of the present disclosure.

[0027] In a fourth aspect, the present invention discloses an integrated circuit layout device, which utilizes the integrated circuit layout method disclosed in the third aspect.

[0028] The integrated circuit layout device includes: a component data processing module, a preliminary layout scheme building module, and a final layout scheme building module.

[0029] The component data processing module is used to acquire circuit component data and process it to obtain a component netlist dataset. The preliminary layout solution construction module is used to perform one round of processing on the component netlist dataset using the target layout solution exploration model to obtain a preliminary layout solution. The final layout solution construction module uses the preliminary layout solution as the initial state and adjusts it through simulated annealing to obtain the final layout solution.

[0030] The integrated circuit layout device is implemented according to the method or process of the embodiment of the present disclosure.

[0031] In a fifth aspect, the present invention discloses a readable storage medium having computer program instructions stored therein. When the computer program instructions are read and executed by a processor, the model training method disclosed in the first aspect and / or the integrated circuit layout method disclosed in the third aspect are executed.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1) The strategy network module and random network distillation module proposed in this paper can assign nearly equal probabilities to actions with similar state values ​​while exploring randomization. They do not assign very high probabilities to any one action within the action range, thus avoiding the suboptimal situation caused by repeatedly selecting the same action. At the same time, by maximizing rewards, they abandon obviously unpromising layout plans, thereby improving the exploration efficiency of layout plans. Moreover, they are robust and can better adapt to interference, ultimately increasing training speed.

[0034] 2) This paper utilizes a random network distillation method to define the intrinsic reward generated during the exploration process of the policy network based on the differences in state characteristics between the prediction network and the target network. Due to the complexity of the network model and the limited capacity of the neural network, more specific layout solutions are explored and the prediction error is large. The intrinsic reward will be used as an additional item to the extrinsic reward generated by the layout solution and participate in the network training, thereby further improving the novelty of the explored layout solutions and improving the exploration efficiency.

[0035] 3) The present invention combines reinforcement learning with layout algorithms: In terms of netlist data processing, through data cleaning, data reduction, data transformation and other processing operations, the netlist data set is effectively processed, eliminating the interference caused by the different specifications of each standard unit in the netlist data, making it easier to extract relevant features and more effectively meet the network input; in terms of network structure, the strategy network module is combined with the random network distillation module. Compared with traditional methods, it can explore more effective layout solutions and improve exploration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is a brief flowchart of the model training method proposed in Example 1 of the present invention;

[0038] Figure 2 A data flow diagram between modules in the model training method proposed in Example 1 of the present invention;

[0039] Figure 3 for Figure 2 The structure diagram of the prediction network part in the random network distillation module;

[0040] Figure 4 for Figure 2 The structure diagram of the target network part in the random network distillation module;

[0041] Figure 5 for Figure 2 The structural diagram of the strategy network module;

[0042] Figure 6 for Figure 2 The structural diagram of the target criticism network module;

[0043] Figure 7 for Figure 2 The structural diagram of the online criticism network module;

[0044] Figure 8 This is a flow chart of the integrated circuit layout method proposed in Example 1 of the present invention;

[0045] Figure 9 for Figure 8 The data flow diagram between various modules in the integrated circuit layout method;

[0046] Figure 10 is the initial discrete state diagram of ami49 used in Example 1 of the present invention;

[0047] Figure 11 This is the final layout diagram obtained by directly using simulated annealing to layout ami49 in Example 1 of the present invention;

[0048] Figure 12 This is the final layout diagram obtained by using the integrated circuit layout method to layout ami49 in Example 1 of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component. When a component is considered to be "fixed to" another component, it may be directly fixed to the other component or there may be a central component.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0052] Example 1

[0053] Please see Figure 1 , which is a brief flow chart of the model training method proposed in this embodiment 1, is used to train the initial layout solution exploration model to obtain a trained target layout solution exploration model.

[0054] In general, the model training method includes the following steps:

[0055] S1.1: Build an initial layout solution exploration model. The initial layout solution exploration model includes a policy network module, an environment setting module, an experience pool module, a random network distillation module, an online critic network module, and a target critic network module.

[0056] See Figure 2 , which is a data flow diagram between various modules in the model training method proposed in Example 1.

[0057] The following is an introduction to each module one by one:

[0058] 1. The environment setting module is constructed by customizing the gym reinforcement learning training environment: including setting the state space, setting the action space, setting the interaction logic, and setting the reset condition.

[0059] The state space refers to the range of states that a component can reach. Generally, it is set as a discrete state.

[0060] The action space is the range of actions that a component can make, which generally includes five types: 1. Select a specific component and other random components in Γ + 2. Select specific elements and randomly select other elements in Γ - 3. Select specific elements and other random elements in Γ + , Γ - 4. Delete specific elements and insert them into Γ + , Γ - 5. Select a specific component and change R e The value in .

[0061] The interaction logic is how the calculations are performed:

[0062] (1) When the initial layout solution exploration model is trained, the current component training value s m , current action training value a m Calculate the next element training value s m+1 and the current action external reward training value r i ext :Specifically, the environment setting module receives the current component training value s m , current action training value a m , randomly select an element as the next element training value s m+1 , based on the current action training value a m Adjust the layout scheme encoding during training <Γ + , Γ - , R e >.

[0063] Among them, <Γ + , Γ - > represents a sequence pair, and Γ represents a sequence with N element numbers. e is a sequence of length N consisting of 0 or 1, that is, for R e The E+1th number in the sequence represents the element with the sequence number E. The corresponding value (0 or 1) determines whether the element has been rotated (0 means no rotation has been generated, 1 means rotation has been generated). E∈[0,N-1], N is the number of all elements in the selected dataset (in this case, the selected dataset is the training dataset). This means taking one of the five actions mentioned above: 1. Select the current element training value s m and a random one of the remaining elements in Γ + 1. Exchange in the middle, optimize the change mode of the previous sequence; 2. Select the current component training value s m and a random one of the remaining elements in Γ - Exchange in the middle, optimize the change mode of the next sequence; 3, select the current component training value s m and one of the remaining elements in Γ + , Γ - Exchange in the middle, optimize the change mode of the overall sequence; 4, delete the current component training value s m , and insert into Γ + , Γ - Random position; 5, select the current component training value s m As the component with serial number E, if the current E is 0, change it to 1, and vice versa, optimize R e The change method of . In this way, the new layout scheme code <Γ +′ , Γ -′ , R e ′>.

[0064] Then, according to the topological sorting, calculate <Γ +′ , Γ -′ , R e The sum of the layout space occupied by all components and the length of the connecting lines in ′> is used as the external reward r for the current action i ext Specifically, according to <Γ +′ , Γ -′ >, construct a directed acyclic graph with horizontal constraints, and then use the longest path algorithm to obtain the result in <Γ +′ , Γ -′ >Constraint, the layout scheme encoding <Γ +′ , Γ -′ , R e ′>The maximum width that can be achieved and the width coordinates of each component; Similarly, construct a directed acyclic graph of vertical constraints and obtain <Γ +′ , Γ -′ , R e ′>The maximum height that can be achieved and the height coordinates of each component; Among them, if R e ', if the value of the position where the serial number is E is 1, the height and width are swapped, indicating that the component has been rotated 90 degrees.

[0065] (2) When the target layout scheme exploration model performs layout processing, the current component processing value s j ″、Current action processing value a j Calculate the next component processing value s j+1 ″: Specifically, the environment setting module receives the current component processing value s j ″、Current action processing value a j ″, randomly select an element as the next element processing value s j+1 ″, adjust the layout scheme encoding during processing according to the current action processing value <Γ +″ , Γ -″ , R e ″>, similar to (1), a new layout scheme encoding <Γ +″′ , Γ -″′ , R e However, at this time, there is no need to calculate the external reward for the current action. Instead, the component netlist dataset will be laid out according to the new layout scheme encoding.

[0066] The reset condition is to replace the current component training value s m , current action training value a m Source:

[0067] When the preset M rounds are not reached (M is generally set to 20,000 steps), the current component training value s0 of the first round is randomly obtained from the random netlist, and an action is randomly set as the current action training value a0 of the first round;

[0068] If 0<m′≤M, calculate the next component training value s in the m′th round m′ As the current component training value s of the m′+1th round m′ , and randomly set an action as the current action training value a of the m′+1th round m′ .

[0069] After reaching the preset M rounds, the current component training value s of the M+1 round is M Reset to 0, and set s M The current state training value a of the M+1th round is calculated by the strategy network module M ;

[0070] If m″>M, calculate the next component training value s in the m″th round m″ As the current component training value s of the m″+1th round m″ , and the current action training value a of the m″+1th round is calculated by the strategy network module m″ .

[0071] After reaching the preset M rounds, the same number of current component sampling values ​​s are randomly collected from the experience pool module in each round i ', current action sampling value a i ', the next element sampling value s i+1 ′, the final reward sample value of the current action r i ', termination condition sampling value; wherein, the termination condition sampling value is not used.

[0072] It should be noted that each random sampling is 5 groups, and the number of each group is the same: that is, each random sampling to the current component group (that is, the current component sampling value s i ′), the current action group (i.e. the current action sampling value a i ', the current action group), the next component group (that is, the next component sampling value s i+1 ′), the final reward group of the current action (that is, the final reward sampling value r of the current action i ′), the final reward group of the current action), the termination condition group (that is, the termination condition group composed of the termination condition sampling values), and the data of each group of data is Batch.

[0073] 2. The random network distillation module consists of a prediction network part and a target network part. The parameters of the target network part are fixed, and only the parameters of the prediction network part are updated.

[0074] See Figure 3 The prediction network consists of a three-layer 2D convolutional network and a two-layer fully connected network. The three layers of the 2D convolutional network are connected by the LeakyReLU function; the two-layer fully connected network consists of Linear1 and Linear2; adjacent 2D convolutional networks and fully connected layers are connected by the LeakyReLU+Flatten function; and Linear1 and Linear2 are connected by the ReLU function.

[0075] Among them, the convolution kernels of the three-layer two-dimensional convolutional network of the prediction network are all 1*1, and the step size is 1. The random network distillation module receives the current component training value s m , the prediction network outputs the current component training value s m Predictive features of The target network outputs the current component training value s m Target characteristics And calculate the current action intrinsic reward training value

[0076] in,

[0077] See Figure 4 The target network consists of three layers of two-dimensional convolutional networks and one fully connected layer. The three layers of the two-dimensional convolutional networks in the target network are connected by the LeakyReLU function, and the adjacent two-dimensional convolutional networks and fully connected layers are connected by the LeakyReLU+Flatten function.

[0078] Among them, the convolution kernels of the three-layer two-dimensional convolutional network of the target network are all 1*1, and the step size is 1. The random network distillation module receives the next element sampling value s i+1 ′, the prediction network outputs the next element sampling value s i+1 ′’s predictive features The target network outputs the next element sampling value s i+1 Target features of ′ And calculate the loss function loss of the prediction network predict , and then back propagation is used to update the parameters of the prediction network (i.e., back propagation is used to update the parameters of the three-layer two-dimensional convolutional network and the two-layer fully connected layer network in the prediction network). In the formula, Batch represents the number of samples from the experience pool module, which is generally set to 64.

[0079] 3. The experience pool module is used to store the current component training value, current action training value, next component training value, current action final reward training value, and termination condition training value for each round of training.

[0080] The termination condition training value is used to limit whether the model terminates at an early stage. In other words, before the termination condition training value of each round of training is stored back to the experience pool module, it is first checked to see if it meets the requirements; if so, training is stopped early.

[0081] Generally, the threshold comparison method can be used, that is, to calculate the final reward r of the current action m and the final reward r of the previous action m-1 The difference Δr m =r m -r m-1 , if Δr i =0, the calculation stops.

[0082] In addition, the experience pool module also undertakes some calculation work: for example, based on the current action external reward training value Current action intrinsic reward training value Calculate the final reward training value r of the current action m .

[0083] It should be noted that the experience pool module is designed to eliminate correlations: because the previous and subsequent actions in reinforcement learning are usually strongly correlated, if they are not broken up, it is easy to cause interference between the actions.

[0084] Therefore, in the early stage of network training, a random setting method is adopted to form a sufficient amount of current component training value, current action training value, next component training value, current action final reward training value, and termination condition training value for each round of training, and store them in the experience pool module; after reaching the preset M rounds, the current component training value is reset to 0, and the policy network module is used to process it to obtain the current action, and then the current component training value, current action training value, next component training value, current action final reward training value, and termination condition training value are obtained, and iterated, and each round of iteration is also stored in the experience pool module; on the other hand, the experience pool module is randomly sampled (the preset hyperparameter weight is 1), and the current component sampling value, current action sampling value, next component sampling value, current action final reward sampling value, and termination condition sampling value are used to calculate the errors of the random network distillation module, the online criticism network module, and the policy network module.

[0085] 4. See Figure 5The policy network module includes a policy network unit and a classification function unit. The policy network unit includes three layers of two-dimensional convolutional networks and two layers of fully connected layers. The three layers of the two-dimensional convolutional networks in the policy network unit are connected by ReLU functions, the adjacent two-dimensional convolutional networks and fully connected layers are connected by ReLU+Flatten functions, and the two fully connected layers are connected by ReLU functions. The convolution kernels of the three layers of the two-dimensional convolutional networks in the policy network unit are all 1*1, and the stride is all 1.

[0086] The classification function part is connected to the output of the policy network part, including the softmax function and the categorical function set in sequence. Among them, the softmax function is used to obtain the action probability, and the categorical function is used to obtain the corresponding action (i.e. training value, processing value).

[0087] When the initial layout solution exploration model is trained, the policy network module starts working after reaching the preset M rounds:

[0088] On the one hand, the strategy network receives the current element training value s m″ , calculate the current action training value a m″ The probability of the action caused And classify to get the current action training value a m″ Among them, the action probability Just calculate the current action training value a m″ The intermediate value will not be used later.

[0089] On the other hand, the strategy network receives the current element sampling value s i ′, calculate the current action sampling value a i ′The action probability caused by The strategy network receives the next element sampling value s i+1 ', calculate the next action sampling value a i+1 ′The action probability caused by

[0090] It should be noted that, on the other hand, the classification function is based on the action probability The new current action value and next action value will also be calculated, but they are redundant and useless.

[0091] When the target layout solution exploration model performs layout processing, the strategy network department receives the current component processing value s j ″, calculate the current action processing value a j The probability of the action caused by And classify to get the current action processing value a j ″. Among them, the action probability It only calculates the intermediate value of the current action processing value and will not be used later.

[0092] 5. See Figure 6 ,The online criticism network module includes two identical parallel networks ,one, each of which includes three layers of two-dimensional convolutional networks and two layers of ,fully connected layers.

[0093] The three-layer two-dimensional convolutional networks of parallel network 1 are connected by ReLU function, the adjacent two-dimensional convolutional networks and fully connected layer networks are connected by ReLU+Flatten, and the two fully connected layer networks are connected by ReLU function.

[0094] Among them, the convolution kernels of the three-layer two-dimensional convolutional network of the parallel network 1 are all 1*1, and the step size is 1. The online criticism network module receives the current component sampling value s i ′, where one of the parallel networks outputs the current action sampling value a i ′’s action value characteristics Another parallel network outputs the current action sampling value a i ′’s action value characteristics

[0095] 6. See Figure 7 ,The target criticism network module consists of two identical parallel networks and a Bellman ,computation unit.

[0096] Each parallel network 1 consists of three layers of 2D convolutional networks and two fully connected layers. The three layers of 2D convolutional networks in parallel network 2 are connected via ReLU functions, adjacent 2D convolutional networks and fully connected layers are connected via ReLU+Flatten, and two fully connected layers are connected via ReLU functions. The convolution kernels of the three layers of 2D convolutional networks in parallel network 2 are all 1*1, with a stride of 1.

[0097] The Bellman calculation unit is connected to the output ends of the two parallel networks.

[0098] The target criticism network module receives the next element sampling value s i+1 ′, where one of the parallel networks outputs the next action sampling value a i+1 ′’s action value characteristics Another parallel network 2 outputs the next action sampling value a i+1 ′’s action value characteristics The Bellman calculation unit calculates the estimated state value Q i .

[0099] in, α is an adaptive parameter, and γ is a hyperparameter, which is set to 0.99.

[0100] in addition, Q i Used to calculate the error of the online critic network module.

[0101] S1.2. Obtain a training dataset. The training dataset is a random netlist consisting of several components. Typically, a random netlist uses 100 components with a width and height between [10, 100], each with three pins at random locations. Generate 100 random signals, each connected to three random pins.

[0102] The number of random pins assigned to each component represents the maximum number of connections it can have with other components. Randomly generated signals represent the state of connections between components. Each signal is connected to three random pins, indicating that there must be two connections connecting all three components. The sum of the connection lengths calculated by the environment setup module later refers to the sum of the Manhattan distances between all connected pins.

[0103] S1.3, perform multiple rounds of iterative training on the initial layout solution exploration model based on the training dataset.

[0104] In terms of functionality, the environment setup module calculates a portion of the data required for model training and stores it in the experience pool module. The random network distillation module calculates another portion of the data required for model training and stores it in the experience pool module. The policy network module makes decisions, determining the actions to be taken by the environment setup module. The online critic network module and the target critic network module assist the policy network module in making decisions based on the sampled data from the experience pool module.

[0105] Specifically, with the preset M rounds as the boundary, it is divided into two modes: when the preset M rounds are not reached, it is mode 1; after the preset M rounds are reached, it is mode 2.

[0106] Mode 1:

[0107] The environment setting module is based on the current component training value s m , current action training value a m , calculate the next element training value s m+1 and the current action external reward training value Random network distillation module based on s m Calculate the intrinsic reward training value of the current action The current action external reward training value Current action intrinsic reward training value Perform weighted sum to get the final reward training value r for the current action m ,in, coef ext 、coef int are all hyperparameters, but coef extTake a negative value.

[0108] In addition, in order to make the training more stable and avoid non-convergence, it is recommended to first calculate the r m Perform Clip operation: Clip(r m )=max(min(r m ,1.0),-1.0). This calculation can be performed by the experience pool module, and the calculated result is used as the final reward training value r of the current action. m for storage.

[0109] Among them, the current component training value s0 of the first round is randomly obtained from the random netlist, and an action is randomly set as the current action training value a0 of the first round;

[0110] If 0<m′≤M, calculate the next component training value s in the m′th round m ′ is the current component training value s in the m′+1th round m ', and randomly set an action as the current action training value a of the m'+1th round m ';

[0111] The training values ​​for the current component, current action, next component, final reward, and termination condition for each training round are stored back in the experience pool module. Before the termination condition for each training round is stored back in the experience pool module, it is first checked to see if it meets the requirements; if so, training is terminated prematurely.

[0112] Mode 2: See Figure 2 , the environment setting module is based on the current component training value s m , current action training value a m , calculate the next element training value s m+1 and the current action external reward training value Random network distillation module based on s m Calculate the intrinsic reward training value of the current action The current action external reward training value Current action intrinsic reward training value Perform weighted sum to get the final reward training value r for the current action m ,in, coef ext 、coef int are all hyperparameters, but coef ext Take a negative value.

[0113] In addition, in order to make the training more stable and avoid non-convergence, it is recommended to first calculate the r m Perform Clip operation: Clip(rm )=max(min(r m ,1.0),-1.0). This calculation can be performed by the experience pool module, and the calculated result is used as the final reward training value r of the current action. m for storage.

[0114] Among them, the current component training value s of the M+1 round M Reset to 0, and set s M The current state training value a of the M+1th round is calculated by the strategy network module M ;

[0115] If m″>M, calculate the next component training value s in the m″th round m ″ is the current component training value s of the m″+1th round m ″, and the current action training value a of the m″+1th round is calculated by the strategy network module m ″.

[0116] The training values ​​for the current component, current action, next component, final reward, and termination condition for each training round are stored back in the experience pool module. Before the termination condition for each training round is stored back in the experience pool module, it is first checked to see if it meets the requirements; if so, training is terminated prematurely.

[0117] Synchronously, each round also randomly collects the same number of current component sampling values ​​s from the experience pool module i ', current action sampling value a i ', the next element sampling value s i+1 ′, the final reward sample value of the current action r i ', the termination condition sampling value; wherein the termination condition sampling value is not used (ie, the Batch current element sampling value s i ′, Batch current action sampling value a i ', Batch next element sampling value s i+1 ′, Batch current action final reward sampling value r i ′, Batch termination condition sampling value);

[0118] The random network distillation module also calculates the next element sampling value s i+1 Calculate the error of the random network distillation module - predict the loss function of the network part predict , and back-propagation is used to update the parameters of the prediction network part (that is, back-propagation is used to update the parameters of the three-layer two-dimensional convolutional network and the two-layer fully connected layer network in the prediction network part).

[0119] The strategy network module also uses the current component sampling value si 'Calculate the current action sampling value a i ′The action probability caused by According to the next component sampling value s i+1 'Calculate the next action sampling value a i+1 ′The action probability caused by

[0120] Then calculate the action probability The natural logarithm of Maximum Entropy Calculating action probabilities The natural logarithm of in, The above calculations can be performed by the experience pool module.

[0121] Then, the online criticism network module is based on the current component sampling value s i 'Calculate the current action sampling value a i ′’s action value characteristics

[0122] The target criticism network module is based on the next component sampling value s i+1 'Calculate the next element sampling value s i+1 ′’s action value characteristics Based on r i ′, adaptive parameter α calculates the evaluation state value Q i ;

[0123] Then, according to Q i Calculate the loss of the online criticism network module separately - loss q1 、loss q2 (also known as eigenvalue loss);

[0124] in, This calculation can be performed by the experience pool module.

[0125] Based on Calculate the current action sampling value a i ′’s evaluated action value q i ;in, This calculation can be performed by the experience pool module.

[0126] According to q i 、 Preset hyperparameter weight and calculate the error of the policy network module - the loss function of the policy network module Policy, and back-propagate to update the parameters of the strategy network part (that is, use back-propagation to update the parameters of the three-layer two-dimensional convolutional network and the two-layer fully connected layer network in the strategy network part).

[0127] in, This calculation can be performed by the experience pool module.

[0128] Based on Calculate the maximum entropy loss loss Ent , and back-propagate to update the adaptive parameter α (i.e., use back-propagation to update the adaptive parameter α);

[0129] in, A represents the number of random actions, generally set to 5; the ratio is set to 0.98. This calculation can be performed by the experience pool module.

[0130] S1.4, when the stopping condition is met, stop training and obtain the trained target layout solution exploration model.

[0131] Among them, the stopping conditions include loss q1 or / and loss q2 Convergence, or reaching the set stop round (generally set to 200,000 steps), can be set according to actual conditions.

[0132] For example, loss is detected before step 200,000 q1 Stop when convergence occurs; or, stop after reaching 200,000 steps.

[0133] Based on the target layout solution exploration model, this embodiment 1 also provides an integrated circuit layout method. Figure 8 , which is a flow chart of the integrated circuit layout method proposed in Example 1, comprising the following steps:

[0134] S2.1, obtaining circuit component data and processing it to obtain a component netlist data set;

[0135] This step involves first digitizing the circuit components to be laid out. Typically, this netlist dataset is obtained through data cleaning, data reduction, and data transformation. This eliminates interference caused by varying specifications of standard cells within the netlist data, making it easier to extract relevant features and more effectively satisfy the network input requirements for subsequent layout solution exploration models.

[0136] S22, using the target layout solution exploration model to perform I rounds of processing on the component netlist dataset to obtain a preliminary layout solution. The target layout solution exploration model is obtained using the above-mentioned model training method.

[0137] Specifically, the strategy network module obtains the current component processing value s of the j+1th processing round j ″, calculate the current action processing value a j ″;

[0138] The environment setting module receives j ″、a j ″, calculate the next component processing value s j+1 ″, and adjust the layout scheme code; where j∈[0, I]; if j=0, the current component processing value s0″ of the first processing round is randomly obtained from the component netlist data set; if j>0, the current component processing value s0″ of the j+1th processing round is randomly obtained from the component netlist data set; j ″ is the next component processing value s obtained in the jth processing round j ″;

[0139] The component netlist data set is laid out according to the preliminary layout scheme code obtained after the I-round processing to obtain a preliminary layout scheme.

[0140] S2.3, taking the preliminary layout plan as the initial state, adjusting it through simulated annealing to obtain the final layout plan.

[0141] Simulated annealing is a commonly used layout calculation method that utilizes topological sorting and sequence pairs for layout. It performs operations such as permuting and swapping some elements in the code, and calculates the objective function difference with the new solution. It then determines whether the new solution is acceptable. If the new solution is determined to be acceptable, it replaces the current solution with the new solution—the transformation corresponding to the new solution is pre-implemented, and the objective function value is modified. At this point, the current solution has completed one iteration, and the next round of experiments can be started based on this. If the new solution is deemed discarded, the next round of experiments continues based on the original current solution. The topological order determined by simulated annealing is used to generate the final layout result.

[0142] However, existing simulated annealing methods directly use the initial discrete states of components as the initial state, which takes a relatively long time to calculate. This method, however, uses the aforementioned model to first calculate a preliminary layout solution using the initial discrete states of components, and then uses simulated annealing to make adjustments. This generally saves time and produces a final layout solution with better performance.

[0143] This embodiment 1 also uses the above method to layout ami49 (the initial discrete state of the components of ami49 is as follows Figure 8 ), the final layout solution is determined by the maximum occupied area (i.e., the product of the maximum height and maximum width of the layout area, where the units of height and width are both unit length): the smaller the maximum occupied area, the better the final layout solution.

[0144] First, the existing simulated annealing is used to layout ami49, and the results are as follows Figure 9 As shown, it takes 17.34s; using this method, the result is as follows Figure 10 As shown, it took 14.29 seconds, a speedup of nearly 20%. Furthermore, using existing simulated annealing for ami49 layout, the maximum area occupied by the final layout solution is 6440*7994; using this method, the maximum area occupied by the final layout solution is 7056*6510, a reduction of more than 10%.

[0145] It has been verified that the efficiency and effect of this method are improved.

[0146] Example 2

[0147] This embodiment 2 discloses an integrated circuit layout device, which uses the integrated circuit layout method disclosed in embodiment 1.

[0148] The integrated circuit layout device of the second embodiment includes: a component data processing module, a preliminary layout plan building module, and a final layout plan building module.

[0149] The component data processing module is used to obtain circuit component data and process it to obtain a component netlist dataset. The preliminary layout solution construction module is used to perform one round of processing on the component netlist dataset using the trained layout solution exploration model to obtain a preliminary layout solution. The final layout solution construction module uses the preliminary layout solution as the initial state and adjusts it through simulated annealing to obtain the final layout solution.

[0150] Example 3

[0151] This embodiment 3 also discloses a readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and run by a processor, the model training method or / and the integrated circuit layout method disclosed in embodiment 1 are executed.

[0152] When the method of Example 1 is applied, it can be applied in the form of software, such as a program designed as a computer-readable storage medium that can run independently. The computer-readable storage medium can be a USB flash drive designed as a USB shield, and the USB flash drive is designed to start the program of the entire method through external triggering.

[0153] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A model training method for training an initial layout solution exploration model to obtain a trained target layout solution exploration model; characterized in that: The model training method comprises the following steps: S1.1, constructing an initial layout solution exploration model; the initial layout solution exploration model includes a policy network module, an environment setting module, an experience pool module, a random network distillation module, an online criticism network module, and a target criticism network module; S1.2, obtaining a training data set; wherein the training data set is a random netlist including a plurality of components; S1.3, perform multiple rounds of iterative training on the initial layout solution exploration model based on the training dataset; The environment setting module is used to calculate a portion of the data required for model training and store it in the experience pool module; The random network distillation module is used to calculate another part of the data required for model training and store it in the experience pool module; The policy network module is used to make decisions and determine what actions the environment setting module should take; The online criticism network module and the target criticism network module assist the strategy network module in making decisions based on the sampling data of the experience pool module; S1.4, when the stopping condition is met, stop training and obtain the trained target layout solution exploration model; In S1.3, the environment setting module obtains the current component training value s of the m+1th training round m , current action training value a m , calculate the next element training value s m+1 and the current action external reward training value Random network distillation module based on s m Calculate the intrinsic reward training value of the current action Will Perform weighted sum to get the final reward training value r for the current action m ; After reaching the preset M rounds, the same number of current component sampling values ​​s are randomly collected from the experience pool module in each round i ', current action sampling value a i ', the next element sampling value s i+1 ′, the final reward sample value of the current action r i ', termination condition sampling value; wherein, the termination condition sampling value is not used; Random network distillation module based on s i+1 'Calculate the error of the random network distillation module and back-propagate to update the parameters of the prediction network; the strategy network module is based on s i 'Calculate the current action sampling value a i ′The action probability caused by According to s i+1 'Calculate the next action sampling value a i+1 ′The action probability caused by calculate The natural logarithm of Maximum Entropy calculate The natural logarithm of Online Criticism Network Module Based on s i 'Calculate the current action sampling value a i ′’s action value characteristics The target criticism network module is based on s i+1 'Calculate the next element sampling value s i+1 ′’s action value characteristics in accordance with r i ′, adaptive parameter α calculates the evaluation state value Q i ;in accordance with Q i Calculate the loss of the online criticism network module separately q1 、loss q2 ;in accordance with Calculate the current action sampling value a i ′’s evaluated action value q i ; Based on q i 、 Preset the hyperparameter weight, calculate the error of the policy network module, and backpropagate to update the parameters of the policy network module; Calculate the maximum entropy loss loss Ent , and back-propagate the updated α.

2. The model training method according to claim 1, characterized in that When the preset M rounds are not reached, the current component training value s0 of the first round is randomly obtained from the random netlist, and an action is randomly set as the current action training value a0 of the first round; If \(0 < m' \leq M\), the training value \(s\) of the next component calculated in the \(m'\)-th round is used m′ as the current training value \(s\) of the component in the \((m' + 1)\)-th round m′ and a random action is set as the current training value \(a\) of the action in the \((m' + 1)\)-th round m′ .

3. The model training method according to claim 2, characterized in that After reaching the preset M rounds, the current component training value s of the M+1 round is M Reset to 0, and set s M The current state training value a of the M+1th round is calculated by the strategy network module M ; If m″>M, calculate the next component training value s in the m″th round m″ As the current component training value s of the m″+1th round m″ , and the current action training value a of the m″+1th round is calculated by the strategy network module m″ .

4. The model training method according to claim 3, characterized in that The current component training value, current action training value, next component training value, current action final reward training value, and termination condition training value of each round of training are stored back into the experience pool module; Among them, before the termination condition training value of each round of training is stored back to the experience pool module, it is first judged whether it meets the requirements; if so, the training is stopped in advance.

5. A model training device, characterized in that: The model training method according to any one of claims 1 to 4 is used; the model training device comprises: An initial model building module, which is used to build an initial layout solution exploration model; the initial layout solution exploration model includes a strategy network module, an environment setting module, an experience pool module, a random network distillation module, an online criticism network module, and a target criticism network module; A training data acquisition module, which is used to acquire a training data set; wherein the training data set is a random netlist including a plurality of components; The model training module is used to perform multiple rounds of iterative training on the initial layout solution exploration model based on the training data set; wherein the environment setting module is used to calculate a portion of the data required for model training and store it in the experience pool module; the random network distillation module is used to calculate another portion of the data required for model training and store it in the experience pool module; the policy network module is used to make decisions on what actions the environment setting module should take; the online criticism network module and the target criticism network module assist the policy network module in making decisions based on the sampled data of the experience pool module; as well as The target model construction module is used to stop training when the stopping condition is met and obtain a trained target layout solution exploration model.

6. An integrated circuit layout method, characterized in that: The following steps are involved: S2.1, obtaining circuit component data and processing it to obtain a component netlist data set; Wherein, the method for processing the circuit element data includes data cleaning, data reduction, and data transformation; S2.2, using a target layout solution exploration model to perform one round of processing on the component netlist dataset to obtain a preliminary layout solution; wherein the target layout solution exploration model is obtained using the model training method according to any one of claims 1 to 4 or / and the model training device according to claim 5; The strategy network module obtains the current component processing value s of the j+1th processing round j ″, calculate the current action processing value a j ″; The environment setting module receives j ″、a j ″, calculate the next component processing value s j+1 ″, and adjust the layout scheme code; where j∈[0, I]; if j=0, the current component processing value s0″ of the first processing round is randomly obtained from the component netlist data set; if j>0, the current component processing value s0″ of the j+1th processing round is randomly obtained from the component netlist data set; j ″ is the next component processing value s obtained in the jth processing round j ″; The component netlist data set is laid out according to the preliminary layout scheme code obtained after I round of processing to obtain a preliminary layout scheme; S2.3, taking the preliminary layout plan as the initial state, adjusting it through simulated annealing to obtain the final layout plan.

7. An integrated circuit layout device, characterized in that: Using the integrated circuit layout method according to claim 6; The integrated circuit layout device comprises: A component data processing module is used to obtain circuit component data and process it to obtain a component netlist data set; a preliminary layout solution building module, configured to perform a round of processing on a component netlist dataset using a target layout solution exploration model to obtain a preliminary layout solution; and The final layout scheme building module is used to take the preliminary layout scheme as the initial state and adjust it through simulated annealing to obtain the final layout scheme.

8. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the model training method according to any one of claims 1 to 4, or / and the integrated circuit layout method according to claim 6 are executed.

Images