A hierarchical power electronic circuit layout method based on reinforcement learning

Abstract

The present invention discloses a hierarchical power electronic circuit layout method based on reinforcement learning, which belongs to the field of integrated circuits. The method is mainly divided into three levels: component sequence generation, layout optimization, and wiring optimization. First, the power flow direction of the power electronic circuit is analyzed, and the component layout sequence is reasonably arranged, thereby achieving basic optimization from the top level; then, based on the component sequence, a convolutional neural network is used to extract layout features, and a graph convolutional neural network is used to extract graph features of circuit connection relationships. The unique design rules of the power electronic circuit are embedded in the reinforcement learning reward system, and a deep Q network (DQN) is used to realize the layout of components; through reinforcement learning, the intelligent agent can summarize experience in trial and error without human intervention, and finally generate a layout result that maximizes the reward.

Description

Technical Field

[0001] The present invention belongs to the field of integrated circuits, and more specifically, relates to a hierarchical power electronic circuit layout method based on reinforcement learning. Background Art

[0002] With the continuous development of power electronics technology, the requirements for the efficiency, power density, and reliability of power electronic circuits are constantly increasing, and therefore the factors that need to be considered in design are becoming increasingly complex. For example, the design of power electronic circuits requires the separation of power and signal lines, the "commutation loop" in the power circuit needs to be minimized in area, different voltage levels need to be reliably isolated, and local heating devices must have sufficient heat dissipation capabilities. To ensure the performance of the designed power electronic circuits, engineers with extensive product design experience and multidisciplinary knowledge are usually required to conduct repeated iterative design and testing, which is very time-consuming, labor-intensive, and costly. To improve design efficiency, various electronic design automation (EDA) tools are usually required to assist designers in completing the tedious and mechanical aspects of the design process, checking and avoiding potential design risks, and thus reducing the number of modification iterations.

[0003] After years of research and development, general-purpose electronic design automation tools now offer automated layout and routing. However, these tools are primarily designed for electronic circuit development, focusing solely on signal flow and connectivity when addressing related issues. Due to the unique design characteristics of power electronics circuits, with numerous components and varying packaging, the layout's layers, positions, and angles all impact the design, creating a vast design solution space. Connectivity is no longer the sole optimization objective; factors such as thermal, electrical, electromagnetic interference (EMI), and power density all significantly impact the design outcome and need to be factored into evaluation metrics.

[0004] Therefore, existing electronic automation design tools are not applicable in the field of power electronics. There is still a need to develop targeted intelligent layout methods that can meet the design requirements of various power electronics to assist power electronics designers in solving design problems efficiently and accurately. Summary of the Invention

[0005] In response to the above-mentioned shortcomings or improvement needs of the existing technology, the present invention provides a hierarchical power electronic circuit layout method based on reinforcement learning, thereby solving the technical problems that the existing power electronic circuit layout design is highly dependent on expert knowledge and experience, has a long design cycle, high cost, and a single optimization target.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a hierarchical power electronic circuit layout method based on reinforcement learning is provided, comprising:

[0007] S1, builds a deep neural network as an intelligent agent;

[0008] The input of the intelligent agent is the layout state matrix O of each grid point after gridding the PCB board and the coordinate state matrix P of each component of the circuit to be laid out. The layout state of each grid point includes no component and within the layout range, outside the layout range, occupied by a component placed in the past action, and occupied by a component placed in the current action. The coordinate state of each component includes the X and Y coordinates of each component and the number of grid points occupied by the component in the X and Y directions when the coordinates are used as the reference point. The output is the quality vector Q of all possible layout actions of the circuit. The layout action is the direction of the next component placement, and the quality is the probability of obtaining the optimal layout result when selecting each layout action. The deep neural network includes CNN, GCN and MLP.

[0009] S2, the layout state matrix O of step t t and the coordinate state matrix P t Input to the agent, CNN from O t Extract the layout features of the PCB plane, GCN from P t Extract circuit diagram features from the MLP, and generate Q according to the layout and circuit diagram features. t ; Select an action a from t , get the layout state matrix O of step t+1 t+1 and the coordinate state matrix P t+1 , and evaluate it through the reward rule to get the reward R t ; O t 、P t 、a t 、R t , O t+1 、P t+1 As a set of data, it is stored in the memory pool; where t = 0, 1, 2, ... K, where K is the number of components in the circuit to be laid out; the reward rule is: the placement positions of the components do not conflict and the total routing length is as short as possible;

[0010] S3, repeat S2 until all components are placed in a preset order; randomly extract a batch of data from the memory pool to train the intelligent agent; wherein the preset order is consistent with the power flow direction of the circuit.

[0011] S4, repeating S2-S3 until the training stop condition is met, and using the trained intelligent agent to obtain the optimal layout result of the circuit to be laid out.

[0012] According to a second aspect of the present invention, a hierarchical power electronic circuit layout and routing method based on reinforcement learning is provided. The method described in the first aspect is used to obtain an optimal layout result of the circuit to be laid out, that is, the pin positions of each component of the circuit to be laid out, for wiring. The wiring method includes:

[0013] S1, build an intelligent agent based on Q-table; where the state is the l×n×m coordinate points of the virtual PCB board, and the action is to move from the current coordinate point to the "above", "below", "left", or "right" of the xth layer (x=1,2…l); define the mass matrix Q' as the mass of each action of all coordinate points on each layer, and define the action reward value of all coordinate points on each layer as the reward matrix R';

[0014] S2: Divide the components into N groups according to their pin potentials. In the yth group of pins, take the zth pin and the z-1th group of traces as the end points, and the z+1th pin as the starting point to form the zth group of starting and ending points of the yth group of pins, and initialize its mass matrix Q. y,z ` and the reward matrix R y,z `; where y = 1, 2…N, z = 1, 2…M y -1,M y is the total number of pins in the yth group, and the z-1th group of traces is the line between the zth pin and the z-1th pin; the initial Q y,z All elements in ` are 0; if the initial R y,z If the coordinate point corresponding to the element in ` is the end point, the value of the element is positive. If the coordinate point is a pin in a different group or is occupied by a trace in a different group, the value of the element is negative. Otherwise, the value of the element is a smaller negative value.

[0015] S3, based on the current coordinate point and the current mass matrix Q y,z `Select action, according to the initial R y,z `And reward rules determine its reward value, and update Q according to the reward value and Q-learning Q value calculation formula y,z `; The reward rules are: the total routing length is as short as possible, and the routing paths of different potential points cannot overlap;

[0016] S4, repeating S3 until a preset reward value is obtained after taking the latest action; the preset reward value is the end point reward value;

[0017] S5, repeat S2-S4 until the training stop condition is met; use the trained intelligent agent to obtain the optimal wiring result of the zth group of starting and end point combinations.

[0018] S6, repeating S2-S5 until all pins are connected, and finally obtaining the optimal wiring result of the circuit to be wired.

[0019] According to a third aspect of the present invention, there is provided an electronic device comprising: a computer-readable storage medium and a processor;

[0020] The computer-readable storage medium is used to store executable instructions;

[0021] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to the first aspect or the second aspect.

[0022] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to the first aspect or the second aspect.

[0023] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0024] The present invention provides a hierarchical power electronic circuit layout method based on reinforcement learning, which is mainly divided into three levels: component sequence generation, layout optimization, and wiring optimization. First, the power flow analysis for the power electronic circuit is used to reasonably arrange the component layout order, thereby achieving basic optimization from the top level; then, based on the component sequence, a convolutional neural network is used to extract layout features, and a graph convolutional neural network is used to extract graph features of circuit connection relationships, and the unique design rules of the power electronic circuit are embedded in the reinforcement learning reward system, and a deep Q network (DQN) is used to implement the layout of components; through reinforcement learning, the intelligent agent can summarize experience in trial and error without human intervention, and finally generate a layout result that maximizes the reward; the power flow of the power electronic circuit obtained based on circuit modal analysis is used as a reference for formulating the component layout sequence, and physical laws and expert experience are used as prior knowledge for deep neural network learning to improve the algorithm optimization efficiency. In addition, the hierarchical power electronic circuit wiring method based on reinforcement learning provided by the present invention uses a reinforcement learning method based on Q-table based on the optimal layout result obtained by the above layout method to quickly complete the wiring. In summary, the layout and routing method provided by the present invention transforms the layout and routing problem of power electronic circuits into a problem similar to "playing chess" or "navigating a maze." It uses deep reinforcement learning to continuously try various layout and routing structures, efficiently searching for the optimal layout and routing results in a process of trial and error and learning, avoiding complex manual design and iterative processes and improving design efficiency. The conflicting rules and optimization objectives of layout and routing are converted into reinforcement learning rewards, thereby transforming the layout and routing problem into a reinforcement learning problem of finding the maximum sum of rewards. The deep neural network can automatically explore and learn the layout and routing results that maximize the rewards. In particular, a simplified method is used to evaluate the wire length indicator during the layout optimization process, achieving decoupling of the layout and routing processes and reducing time complexity.

[0025] Furthermore, when training the intelligent agent, the present invention preferably adopts a method of fully utilizing the CPU and GPU, splitting the program and assigning it to the CPU or GPU for operation according to the operation requirements, and adopting a programming method of multi-threaded synchronous operation, thereby realizing the simultaneous evaluation of component layout optimization based on different sequences, shortening the program running time; on this basis, the deep neural network "equivalent parallel training" model is adopted to train multiple DQNs simultaneously on a single-core GPU, which can simultaneously learn and layout and route multiple component sequences, significantly accelerating the entire design process and further improving the program running efficiency.

[0026] Furthermore, the present invention utilizes the world knowledge of LLM to obtain the generation of the component sequence of the circuit to be laid out, fully utilizing the LLM's powerful understanding ability and ability to learn relevant knowledge and experience of power electronics, and uses it for global planning and detailed assistance in layout and routing optimization problems, completing tasks such as component sequence generation, user language and algorithm language conversion, and hierarchical design process interaction; fully combining the high flexibility but uncertain output of LLM with the autonomous learning and trial-and-error characteristics of reinforcement learning, providing a new approach to solving layout and routing problems, reducing dependence on human expert experience, and improving design efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of a hierarchical power electronic circuit layout method based on reinforcement learning provided by an embodiment of the present invention;

[0028] Figure 2 A topological diagram of a Cuk converter provided in an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a Cuk converter drawn in Altium Designer software according to an embodiment of the present invention;

[0030] Figure 4 The operating interface designed based on Python language provided by the embodiment of the present invention;

[0031] Figure 5 A current loop when a switch of a Cuk converter provided by an embodiment of the present invention is turned on;

[0032] Figure 6 A current loop when the switch of the Cuk converter provided by an embodiment of the present invention is disconnected;

[0033] Figure 7 The optimal layout result of the Cuk converter is obtained by using the layout method provided by the embodiment of the present invention;

[0034] Figure 8 The optimal wiring result of the Cuk converter is obtained by using the layout method provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0036] The traditional layout and wiring process of power electronic circuits is cumbersome and complicated, requiring experienced professionals to go through multiple rounds of iterations and modifications to complete the design. It is time-consuming, labor-intensive and costly, so it is in urgent need of assistance from new technologies and new ideas. The layout and wiring problem of power electronic circuits is essentially to find the optimal solution in a huge solution space. To address this problem, artificial intelligence (AI) is an effective method. Among various AI methods, reinforcement learning (RL) allows continuous exploration in the search space and continuous learning from experience in the trial and error process, and ultimately finds the optimal solution efficiently. This process does not require human intervention, so it can significantly reduce the design burden of designers. Based on this, an embodiment of the present invention provides a hierarchical power electronic circuit layout method based on reinforcement learning, such as Figure 1 Shown, including:

[0037] S1, builds a deep neural network as an intelligent agent;

[0038] The input of the intelligent agent is the layout state matrix O of each grid point after the PCB board is gridded, and the coordinate state matrix P of each component of the circuit to be laid out. The layout state of each grid point includes no component and within the layout range, occupied by a component placed in the past action, and outside the layout range and occupied by a component placed in the current action. The coordinate state of each component includes the X and Y coordinates of each component and the number of grid points occupied by the component in the X and Y directions when the coordinates are used as the reference point. The output is the quality vector Q of all possible layout actions of the circuit. The layout action is the direction of the next component placement, and the quality is the probability of obtaining the optimal layout result when selecting each layout action. The deep neural network includes a convolutional neural network, a graph convolutional neural network, and a multi-layer perceptron.

[0039] S2, the layout state matrix O of step t t and the coordinate state matrix P t Input to the agent, the convolutional neural network is from O t Extracting the layout features of the PCB plane, the graph convolutional neural network, from P t The circuit diagram features are extracted from the multilayer perceptron, and the multilayer perceptron generates Q according to the layout and circuit diagram features. t ; Select an action a from t , get the layout state matrix O of step t+1 t+1 and the coordinate state matrix P t+1 , and evaluate it through the reward rule to get the reward R t ; O t 、P t 、a t 、R t , Ot+1 、P t+1 As a set of data, it is stored in the memory pool; where t = 0, 1, 2, ... K, where K is the number of components in the circuit to be laid out; the reward rule is: the placement positions of the components do not conflict and the total routing length is as short as possible;

[0040] S3, repeat S2 until all components are placed in a preset order, randomly extract a batch of data from the memory pool, and train the intelligent agent using a reinforcement learning algorithm; wherein the preset order is consistent with the power flow direction of the circuit.

[0041] S4, repeating S2-S3 until a preset condition is met, and using the trained intelligent agent to obtain the optimal layout result of the circuit to be laid out.

[0042] Specifically, the method provided by the present invention comprises the following steps:

[0043] (1) Define a set of two-dimensional state matrices O t and P t , O t Used to describe the coordinate occupancy information in the virtual PCB layout, indicating whether a component has been placed at the coordinate point; t Used to describe the characteristics of each component, including the current coordinates and placement angle.

[0044] Among them, the layout state matrix O t The matrix represents the current layout characteristics of the virtual PCB board. It is an n×m matrix, which essentially divides the virtual PCB board into an n×m grid (it can be understood that when the layout area of ​​the PCB board is a square, n=m). If there is no component at a grid point and it is within the allowable layout range, the matrix element value of the point is 0; if a component is placed at the grid point (that is, it is occupied by a component placed in the past action) or it is outside the allowable layout range, the matrix element value of the point is 1; to highlight the influence of the "current action", if the grid point is occupied by a component placed by the current operation, the matrix element value of the point is 2.

[0045] Coordinate state matrix P t The characteristics of the circuit to be laid out are represented by a K×4 matrix, where K is the number of components in the circuit to be laid out. Each row of the matrix represents a component, and the four elements in a row represent the X and Y coordinates of the component and the number of grid points in the X and Y directions occupied by the component with respect to the coordinates as the reference point.

[0046] The placement order of components in the circuit to be laid out can be obtained by manually analyzing the power flow direction of the circuit to be laid out.

[0047] Considering that the newly emerging large language model provides a new perspective for solving many problems with its rich knowledge and powerful natural language understanding capabilities, but because LLM is essentially a probabilistic generation model, the output is uncertain. If the flexibility of the large language model and the trial-and-error characteristics of reinforcement learning can be combined together, it will inevitably bring new breakthroughs to the solution of layout problems. Based on this, in order to improve the efficiency of obtaining the placement order, as a further optimization of the present invention, the above-mentioned manual analysis process is replaced by LLM, that is, the preset order of all the components is obtained by inputting the information of each component of the circuit and the connection relationship of each component into LLM.

[0048] Specifically, the netlist of a power electronic circuit is input, and the LLM uses its extensive knowledge and understanding capabilities to generate a component sequence that meets the power flow requirements by analyzing the netlist and specific prompt words.

[0049] Using circuit design software such as Altium Designer, a netlist containing component information and connection relationship information can be directly exported from the circuit schematic. The netlist is input into the LLM along with corresponding prompts, which include requirements for the LLM, supplementary circuit descriptions, and input and output examples. Requirements are the expectations that the LLM should output a set of component sequences after reading the netlist information. Supplementary descriptions include a description of the basic information and characteristics of the input circuit, such as the circuit topology type, input, output, and commutation loop information. These supplementary information will be prioritized by the LLM, increasing the likelihood of successful optimization. Input and output examples are combinations of other input netlists and output component sequences, providing reference and optimization direction for the LLM. After receiving this netlist input, the LLM can reconstruct it into a power electronic circuit topology and perform operating modal analysis to generate a component sequence that meets power flow requirements.

[0050] Since the sequence generated by LLM may not be completely correct or meet specific design requirements, in order to ensure the correctness and practicality of the generated element sequence, as a further preference of the present invention, the LLM also includes a verification and post-processing module for checking whether the number of elements in the generated element sequence is correct. If not, the missing elements are inserted into the element sequence.

[0051] Specifically, a verification and post-processing procedure is added to the LLM to check whether the number of components in the generated component sequence is correct. If missing components are found, a random placement strategy can be adopted to insert them into the component sequence.

[0052] If it is not possible to directly determine the most reasonable sequence from the generated series of component sequences, each sequence can be used as input to train the corresponding intelligent agent, and then the optimal layout result can be selected.

[0053] Define two lists, Library and Component, to convert the component model package and layout status (including position coordinates, placement angle, potential label, component name and special attribute flag) into digital expression, which is used to store component information and node connection network.

[0054] The expression of the component package table Library is as follows:

[0055] Library={Footprint,Size,N1,N2,…,N T}

[0056] Where Footprint is a string representing the package name of the component. Size is a 1×N vector. The size of N is related to the shape of the package and is the number of parameters required to uniquely describe the package shape. For example, if the package is a rectangle, then the value of N is 2. The first element in Size is the length of the component in the X direction on the virtual PCB board, and the second element is the length of the component in the Y direction on the virtual PCB board. N j (j=1,2…T) represents the position of each pin of the component in the package, and T is the total number of pins contained in the component. N j It is a 1×2 vector that describes the coordinates of the pin relative to the package reference point (the point in the upper left corner). The package name Footprint can be used to accurately index the required package information, and then the package size Size and pin coordinate N can be used to accurately index the required package information. j (j=1,2…T) can define the space occupied by a component on the virtual PCB board, laying the foundation for subsequent layout and routing operations.

[0057] The expression of the component information table Component is as follows:

[0058] Component={Footprint,Angle,P,N,Name,Flag1,Flag2}

[0059] Footprint is a string representing the component's package name. Angle is also a string representing the component's placement angle, meaning the component can be rotated 0°, 90°, 180°, or 270° relative to the default package definition. P represents the coordinates of the component on the virtual PCB board and is a 1×2 vector, where the first element represents the X-coordinate and the second element represents the Y-coordinate. N represents the potential of each pin of the component (pins with the same potential need to be connected) and is a 1×T vector, where T is the total number of pins in the component. Each element in N corresponds to a pin one-to-one, defining the potential number of each pin. Pins with the same potential number are electrically connected. Name is a string representing the name of each component and is used to distinguish different components. Flag1 and Flag2 are both scalars with values ​​of 0 or 1, representing whether the component position is fixed and cannot be moved and whether the component has been placed on the virtual PCB board, respectively.

[0060] (2) Define a multi-input and multi-output deep neural network as the agent, with the state matrix O at step t t and P t As input to the agent (i.e., deep neural network), the agent outputs the quality vector Q of all possible layout actions for the circuit to be laid out. t The layout action refers to the position of the next component placement, and the quality is the possibility of obtaining the optimal layout result by selecting each action pair. The higher the action quality, the more likely it is to obtain the optimal layout result.

[0061] The deep neural network has two input channels: a convolutional neural network (CNN) channel and a graph convolutional neural network (GCN) channel. The CNN is used to extract the layout features of the PCB plane, while the GCN is used to extract the circuit diagram features. The output results of the two channels are then concatenated and encoded. Finally, the encoded results are decoded by a multilayer perceptron (MLP) to obtain the final action output, which is as follows:

[0062] 1) State matrix O t Sent into the CNN input channel, through the convolutional neural network, the two-dimensional matrix O t is mapped to a vector CNN of size 1×c OUT :

[0063]

[0064] Represents a convolutional neural network, whose structure needs to be appropriately selected according to the complexity of the design. represents the trainable parameters in a convolutional neural network, namely the weights and biases. g(·) encodes a multidimensional matrix into a one-dimensional vector. This operation is used to concatenate the outputs of deep neural networks of different dimensions for subsequent calculations.

[0065] 2) State matrix P t Sent into the GCN input channel, through the graph convolutional neural network, the two-dimensional matrix P t is mapped to a vector GCN of size 1×g OUT :

[0066]

[0067] represents a graph convolutional neural network, Represents the trainable parameters in a graph convolutional neural network.

[0068] 3) Output the convolutional neural network to CNN OUT And graph convolutional neural network output GCN OUT The end-to-end splicing code is then sent to the multi-layer perceptron for decoding, and finally a vector Q of size 1×s is obtained. t :

[0069]

[0070] represents a multilayer perceptron, Represents the trainable parameters in the multilayer perceptron, v(·) represents the CNN OUT and GCN OUT Concatenate the ends into a 1×(c+g) vector.

[0071] 4) Quality vector Q t As shown in the following formula:

[0072] Q t ={q(a1),q(a2),…,q(a t )}

[0073] Among them, a i (i=1,2…s) is the action of placing the element, and s is the number of optional actions. Each element value Indicates "select component placement action a i "The quality of the action. Vector Q t The size is 1×s.

[0074] 5) All possible actions are the optional orientations for placing components. The quality of an action is a numerical value. The higher the value, the better the action, and choosing this action is more likely to generate an optimal PCB layout.

[0075] (3) According to all the element values in the vector Q t select an action a from the action space using the ε-greedy algorithm t , that is, an orientation for placing a component, to obtain the state matrices O t+1 and P t+1 at the (t + 1)-th step, specifically including:

[0076] 1) Apply the ε-greedy algorithm, that is, with a certain probability ε, select the action a according to the maximum element value in Q t , and randomly select an action with a probability of 1 - ε. The action a t is a value, 1 < a t < s, representing the orientation for placing the component corresponding to the a t -th element in the vector Q t . Different values of a t represent different orientations for placing components. t

[0077] [[ID=3l]]2) After selecting the orientation for placing the component, place the new component on the virtual PCB board by selecting this action, and then the state matrices O t and P t at the (t + 1)-th step can be obtained from the current state matrices O t+1 and P t+1 .

[0078] (4) Define a reward rule to evaluate the state matrices O t+1 and P t+1 (that is, the new virtual PCB board layout) to obtain the reward R t . It can be understood that the reward R t is used to train the deep neural network. The higher the R t , the greater the probability of taking the corresponding action when encountering the same input O t and P t later; conversely, the smaller the R t , the smaller the probability of taking the corresponding action when encountering the same input O t and P t later.

[0079] The said reward rule is: the placement positions of each component have no conflicts and the total wire length is as short as possible. Based on this reward rule, the reward can be measured comprehensively by multiple indicators, such as:

[0080] 1) The overlapping area of components;

[0081] 2) The overlapping area between the regions enclosed by different potential points;

[0082] 3) Total trace length.

[0083] Among them, the component overlap area reward r overlap is the number of overlapping coordinate points raised to the power of a (a>1), as shown in the following formula. i (c_list) represents the number of components at the i-th coordinate point (the range beyond the boundary is considered to be always distributed with a large component). If there is no component distributed at this coordinate point, it will not participate in the calculation. overlap The physical meaning is that as the overlapping area increases, its impact on the layout becomes more and more serious.

[0084]

[0085] The overlapping reward r between the areas surrounded by different potential points overlap_net_area The overlap area between any two groups of pins at different potentials, when connected, is expressed as follows: S represents the intersection of the area enclosed by the ith potential and the area enclosed by the jth potential. Since this has less impact on layout than component overlap, it is not squared.

[0086]

[0087] Considering that the layout algorithm needs to constantly evaluate the line length, efficiency is an important factor, so the line length reward r hpwl The Half-Perimeter Wire Length (HPWL) model is used. HPWL is the most commonly used routing mode for automatic layout and routing algorithms. It only cares about the lengths of the top, bottom, left, and right sides, as shown in the following formula, where Δx i , Δy i They are the maximum spacing between the pins in the pin group of the i-th potential in the horizontal X direction and the vertical Y direction, denoted by Δx i For example, Δx i It is the difference between the x-coordinate of the rightmost pin and the x-coordinate of the leftmost pin in the horizontal X direction under the potential distribution of the i-th group. This formula means the semi-perimeter of the rectangle surrounded by the two pins on the diagonal. If there are N potentials in the circuit, the reward r hpwl It is the cumulative sum of the maximum differences between the horizontal X direction and the vertical Y direction under each potential distribution. The main features of HPWL are simplicity, fast evaluation, and an actual wiring length error of about 10%.

[0088]

[0089] Each time the agent takes an action, it places a component on the virtual PCB and calculates the current reward. Once all components are placed, the current round of learning stops and the next round begins.

[0090] It is understandable that the potential information of each pin required for calculating the above-mentioned reward is obtained from the component information table Component.

[0091] It should be noted that the reward metrics are not fixed and can be further adjusted, refined, or expanded based on the operational performance of the circuit to be laid out and routed. For example, factors such as electromagnetic interference, loss and temperature rise of the circuit can also be considered, with the addition of metrics such as "keeping high-frequency, high-current traces as short as possible" and "keeping the peak temperature of the virtual PCB as low as possible" to meet design requirements. The above reward metrics and their calculation formulas are provided for illustrative purposes only and are not intended to be exclusive to the present invention.

[0092] (5) Use reinforcement learning algorithms to use rewards to modify the weights and biases of the agent (i.e., deep neural network). t ,P t ), a t , R t ,(O t+1 ,P t+1 ) is stored as a set of data in the memory pool, and a batch of data is randomly extracted from the memory pool to train the agent (ie, deep neural network) through the reinforcement learning algorithm, and the weights and biases of the deep neural network are updated.

[0093] The memory pool contains a large number of (O t , P t ), a t , R t ,(O t+1 , P t+1 ) is a set of sample data. It has a maximum capacity. When the sample data reaches the maximum capacity, the first-in-first-out principle is adopted to eliminate the sample data that entered the memory pool first. Randomly extracting a batch of sample data means randomly selecting a specified number of sample data from all the data in the memory pool.

[0094] A batch of data is randomly extracted from the memory pool to train the deep neural network through the reinforcement learning algorithm, and the weights and biases of the deep neural network are updated. The specific process is as follows:

[0095] 1) Use the DQN algorithm to train a deep neural network. The training goal is to obtain a layout strategy with the highest possible reward. The formula for the DQN algorithm is:

[0096]

[0097] Rt represents the reward after each step, max(·) represents the selection of the maximum element; k represents the output vector Q of the deep neural network t The kth one in the list corresponds to the selected component placement action, i.e., a t ; Represents a complete deep neural network consisting of a convolutional neural network, a graph convolutional neural network, and a multilayer perceptron.

[0098] 2) Due to the real q k Given by the deep neural network, it is necessary to adjust the weight and bias parameters of the deep neural network Make q k accessible The loss function is defined as:

[0099]

[0100] 3) By calculating parameters For the gradient of Loss, it can be adjusted by the following algorithm

[0101]

[0102] Where α is the learning rate of the deep neural network. After training, the loss becomes smaller, indicating that the deep neural network has updated its design strategy.

[0103] (6) Define the “step” of the above strategy as: according to O t and P t , the deep neural network outputs Q t , and then choose action a t , get O t+1 ,P t+1 and R t A "working cycle" for the above strategy is defined as repeating S1 and S3 until the virtual PCB layout state completes a cycle from the initial state. When one cycle is completed, the next cycle is restarted. The deep neural network training stop sign is defined as: when the deep neural network training success requirement is met or the preset maximum number of training times is reached, the operation stops.

[0104] It can be understood that in the initial state, the state matrix O t The element value corresponding to the allowed layout area is 0, and the element value corresponding to the area outside the preset layout range is 1; the state matrix P t Then all elements are 0.

[0105] Each time you run a step, you get an action a t , that is, place the component at the specified position, and then update the state matrix to O t+1 and Pt+1 , and receive a reward R t When all components are placed, the circuit layout is complete and the current round of learning stops. Training stops when the preset maximum number of training times is reached or the final accumulated reward reaches a certain threshold.

[0106] In order to improve the learning efficiency of the system, as a further optimization of the present invention, when training the intelligent agent, the weight parameter update of the deep neural network is completed in the GPU, and the update of the memory pool is completed in the CPU.

[0107] Specifically, the training process uses a training method that combines the CPU and GPU to improve the learning efficiency of the system. By adopting a multi-process programming method, the program that takes a long time to "train the neural network and update the weight parameters" is placed on the GPU with a faster matrix operation speed to run as the main process; the programs that take a short time to "place components", "update the simulated PCB board", "calculate rewards", etc. (collectively referred to as the memory pool update program) are placed on the CPU to run as sub-processes. In this way, the CPU and GPU can divide the work and cooperate with each other. The main process is always training the neural network, and the sub-process is always updating the memory pool, so that the program length is always determined by the program length of training the neural network. In addition, the sub-process can make full use of time to simultaneously update the memory pool under multiple component sequences, thereby evaluating the impact of different sequences on the layout results.

[0108] When laying out multiple circuits to be laid out, in order to further improve the system learning efficiency, as a further optimization of the present invention, deep neural networks of multiple intelligent agents are trained simultaneously in the GPU; the multiple intelligent agents correspond one-to-one to the multiple circuits to be laid out, and are respectively used to obtain the optimal layout results of the corresponding circuits to be laid out.

[0109] When training a neural network on a GPU, the inputs of several neural networks are concatenated into a large matrix X = [X1... X i ... X n ], the weights are also concatenated into a large weight matrix W = [W1 ... W i ... W n ], and the same applies to bias. By computing these large matrices in parallel, it is possible to perform the equivalent of computing multiple neural networks simultaneously (i.e., several agents performing parallel layout and routing of multiple circuits to be laid out).

[0110] On this basis, by summing and averaging the loss functions and performing reverse gradient propagation on the total loss function, we can achieve the effect of parallel training of multiple neural networks on a single-core GPU. Therefore, the loss function should be changed to:

[0111]

[0112] The essence of this training method is not to train multiple different neural networks simultaneously, but to calculate the loss functions of multiple neural networks in sequence and then perform backpropagation together as a large neural network model. The calculation of the loss function is still performed in series, but the backpropagation calculation process achieves a parallel effect. Therefore, this model is called the "equivalent parallel training" model.

[0113] (7) After the layout agent training is completed, the layout state matrix and coordinate state matrix of the circuit to be laid out are input into the trained layout agent to obtain the optimal layout result of the circuit to be laid out. The optimal layout result is the placement position of each component of the circuit to be laid out, which is represented by the pin coordinates of each component.

[0114] It is understandable that during the training phase of the layout agent, the selection of actions is random. During the application phase of the layout agent, that is, when the trained layout agent is used to obtain the optimal layout result of the circuit to be laid out, the selection of actions is based only on the quality of the actions.

[0115] An embodiment of the present invention provides a hierarchical power electronic circuit layout and routing method based on reinforcement learning. The layout method described in the above embodiment is used to obtain an optimal layout result of the circuit. Based on the current optimal layout result, the layout result is used as the initial input. The routing algorithm is started according to principles and processes similar to those of the layout algorithm, and finally the optimal layout and routing result is obtained. The method includes:

[0116] S1, builds an intelligent agent based on Q-table; the state is the l×n×m coordinate points of the virtual PCB board, and the action is to move from the current coordinate point to the "above", "below", "left" or "right" of the xth layer (x=1,2…l); the mass matrix Q' is defined as the mass of each action of all coordinate points in each layer, and the action reward value of all coordinate points in each layer is defined as the reward matrix R'.

[0117] Specifically, the wiring problem is divided into a reinforcement learning optimization problem with finite states and actions. The states are the l×n×m coordinate points on a virtual PCB board, and the actions are moving from the current coordinate point to the "above," "below," "left," or "right" of the xth layer (x=1, 2…l). For any state, any action is associated with an action quality Q value, and such a (state, action)-Q key-value pair is defined as the quality matrix Q'. For any state, any action is associated with a reward R value, which evaluates the current action, and such a (state, action)-R key-value pair is defined as the reward matrix R'.

[0118] S2, divide the components into N groups according to their pin potentials, and select the pins of the zth (z = 1, 2 ... M) group from among the pins of the yth (y = 1, 2 ... N) group.y -1,M y The total number of pins in the yth group) and the z-1th group of traces are used as the end point, the z+1th pin is used as the starting point, and the zth group of starting and ending points of the yth group of pins are formed. The mass matrix Q of the zth group of starting and ending points of the yth group of pins is initialized. y,z ` and the reward matrix R y,z `, where the initial Q y,z All elements in ` are 0. If the initial R y,z If the coordinate point corresponding to the element in ` is the end point, the value of the element is positive. If the coordinate point is a pin in a different group or is occupied by a trace in a different group, the value of the element is negative. Otherwise, the value of the element is a smaller negative value.

[0119] Specifically, all elements in the initial mass matrix are assigned a value of 0. In the initial reward matrix, the elements of the coordinates corresponding to the end point are assigned a large positive reward, which physically means that the trace is connected to the end point; the elements of the coordinates occupied by pins or traces from different groups are assigned a large negative reward, which physically means that the trace coincides with other potentials and is short-circuited; the elements of the remaining coordinates are assigned a small negative reward, which physically means that points will be deducted for each step. The longer the total path, that is, the longer the trace, the lower the total reward.

[0120] S3, based on the current coordinate point and the current mass matrix Q y,z `Select action, according to the initial R y,z `And reward rules determine its reward value, and update Q according to the reward value and Q-learning Q value calculation formula y,z `;The reward rules are: the total routing length is as short as possible, and the routing paths of different potential points cannot overlap.

[0121] Specifically, according to the current coordinate point and the current mass matrix Q y,z , based on the greedy mechanism to select the action, according to the current reward matrix R y,z `And reward rules determine its reward value, and update the Q value according to the reward value and the Q-value calculation formula of Q-learning; the reward rule is: the total routing length is as short as possible, and the routing paths of different potential points cannot overlap;

[0122] It can be understood that during the iterative update process of S3, since the coordinates of the end point (i.e., the zth pin and the z-1th group of routing) and the pins or routings in different groups remain unchanged, the value of the reward matrix remains unchanged and is always the initial reward matrix.

[0123] S4, repeat S3 until the preset reward value is obtained after the latest action is taken, indicating that the zth and z+1th pins of the yth group are connected, that is, a round of environment interaction and training of the agent is completed; the preset reward value is the end point reward value;

[0124] S5, repeat S2-S4 until the training stop condition is met (for example, when the agent training success requirement is met or the preset maximum number of training times is reached), and the current agent training is completed; the trained agent is used to obtain the optimal wiring result between the two pins in the group (i.e., the zth group of starting and ending points);

[0125] S6, repeating S2-S5 until all the pins of the yth group (y=1, 2...N, that is, all the pins) are connected, and finally obtaining the optimal wiring result of the circuit to be wired.

[0126] The routing algorithm follows similar principles and processes as the layout algorithm, with only differences in the modeling and algorithm design of specific problems. Specifically:

[0127] The wiring problem is converted into a maze-walking problem with the pin to be connected as the starting point, other pins or traces with the same potential as the end point, and pins or traces with different potentials as obstacles. The optimization goal is to minimize the path length from the starting point to the end point.

[0128] The length and width of the virtual PCB are still divided into m×n coordinate points, with the number of layers defined as l. The optional actions are moving to the "above," "below," "left," or "right" of layer i (j=1, 2…l), so the number of optional actions is 4×l. Based on the above definitions, the sizes of the mass matrix Q' and reward matrix R' are (l×m×n)×(4×l) and (l×m×n)×1, respectively. The (l×m×n) rows correspond to the (l×m×n) coordinate points on the virtual PCB, and (4×l) correspond to the optional actions at each coordinate point. Each element in the mass matrix Q' represents the quality of the action selected at each grid point, and each element in the reward matrix R' represents the reward received after the agent moves to that coordinate.

[0129] The routing action is the moving direction of the next coordinate point. The reward for the routing action is a small negative reward for each step, indicating that the shorter the total path length, the higher the reward. If the movement overlaps with a pin or trace at another potential, a large negative reward is given, indicating that such a situation is avoided. If the movement reaches a pin or trace at the same potential, a large positive reward is given, indicating that the end point has been found and the learning round is over.

[0130] The basic process of the routing method is as follows: first, all pins are grouped according to their potential, and two pins in the same group are selected as the starting and ending points for independent routing optimization. For each group of pins, a quality matrix Q' and a reward matrix R' are defined. Q' ...

[0131] Since the complexity of input and output is not high, the reinforcement learning agent chooses to use the simpler and faster Q-table to implement it, that is, the agent is built based on the Q-table, which is updated in the form of the Bellman optimal equation, as shown in formula (16). Where α is the learning rate, γ is the discount factor, and max a Q(S′, a) is the empirical Q value, which is the Q value given by the experience of taking actions in the state S′ and receiving rewards in the past. The larger the α, the faster the learning rate, and the less the Q table is considered when updating the Q table. The larger the γ, the more experience is considered.

[0132] Q(S,A)←(1-α)Q(S,A)+α[R(S,α)+γ·max a Q(S′,α)]

[0133] In addition to the problem modeling and algorithm selection, the solution process of the layout method is basically similar to the wiring method. The agent continuously acts to obtain the reward R t , and updates the Q-table. If the maximum number of training times is reached, the learning process ends, and the agent moves from the starting point to the end point completely according to the output of the Q-table. This path is the optimal route discovered by the agent.

[0134] The method provided by the present invention is further described in detail below with reference to a specific example.

[0135] In order to specifically introduce the hierarchical power electronic layout and wiring method based on reinforcement learning proposed in this paper, the typical boost-buck converter Cuk converter is discussed as an example. The Cuk converter is a single-transistor non-isolated DC converter proposed by Slobodan Cuk of California Institute of Technology in 1980 to improve the Buck-Boost converter. Figure 1As shown in Figure 1, the Cuk converter can be viewed as a series combination of a boost converter and a buck converter, incorporating a switching transistor. Its basic topology includes coupling capacitor C1, switching transistor Q1, boost inductor L1, buck inductor L2, freewheeling diode D1, and filter capacitor C2. The Cuk converter's most significant feature is the continuity of the input power supply current and output load current. Inductors are included in both the input and output stages, significantly reducing input and output current ripple. This results in minimal output voltage ripple and minimal impact on the input power supply.

[0136] like Figure 2 As shown in the diagram, according to the Cuk circuit, you can place the required components and connect them in Altium Designer to draw its schematic diagram. Note that in this figure, the input power supply U E The output load R is represented by terminals JP1 and JP2, respectively, and the remaining components correspond to each other. Two engineering netlist files can be exported from the Altium Designer schematic file. One is the component table (Components), which is used to store information about each component in the schematic, including component number, component package type, and component model. The other is the connection network table (Nets), which is used to store all electrical connection networks in the schematic, including network names, node information in the network, network connection descriptions, etc. Based on these two engineering netlists, all information about the circuit to be laid out required by the program of the present invention can be obtained.

[0137] The large language model driven reinforcement learning hierarchical power electronic circuit layout and wiring method proposed in this invention is based on Python language and uses tkinter library to build a window GUI interface. Figure 3 As shown in the figure, the buttons in the GUI window can be clicked to access corresponding functions. Based on this operation interface, the following detailed steps for automatic placement and routing of Cuk circuits are demonstrated.

[0138] Step (1): Click the "Browse" button on the operation interface, select the txt files of the two engineering network tables of the Cuk circuit Components and Nets exported from Altium Designer and the pre-defined component sequence, and import the Cuk circuit information into the program.

[0139] Step (2): Perform modal analysis on the Cuk circuit to obtain the power flow path.

[0140] (A) When the switch tube Q1 is turned on, the original freewheeling diode D1 is reverse biased and cut off. The input circulating current I1 makes the inductor L1 store energy. The coupling capacitor C1 discharges the current I2 to store energy in L2 and supply power to the load I3. The sum of the input and output currents in the switch tube I Q =I1+I2, such as Figure 4 shown.

[0141] (B) When switch Q1 is turned off, the freewheeling diode D1 is forward biased and conducts. The energy storage current I1 from input port JP1 (usually the power supply) and inductor L1 charges the coupled inductor C1. Simultaneously, the energy release current I2 from inductor L2 acts to maintain uninterrupted load current. At this point, the current flowing through freewheeling diode D1 is the sum of the input and output currents, I Q =I1+I2, such as Figure 5 shown.

[0142] (C) According to the above analysis, following the direction of power flow, the sequence of component placement order can be defined as JP1→L1→C1→Q1→D1→L2→C2→JP2.

[0143] (D) LLM replaces the manual analysis process described above. Simply input the Components and Nets engineering netlists of the Cuk circuit exported from Altium Designer and provide relevant prompts, such as the circuit input port JP1, the output port JP2, and the components of the commutation circuit. LLM then outputs a series of optimal component sequences based on its understanding. It then extracts relevant information from the Components and Nets, establishes a Library and Component, and, based on the information in the tables, constructs the agent's layout state matrix, coordinate state matrix, and calculates rewards.

[0144] Step (3): The component sequence output by LLM is input into the verification and post-processing program. The program compares the project netlist and the component sequence, automatically eliminating obviously unreasonable sequences. If there are sequence components missing, the missing components are added to the sequence according to the random principle. All remaining sequences will be sent to the subsequent layout optimization process.

[0145] Step (4): Click the "Set Boundary" button on the operation interface and use the left mouse button to draw the desired boundary. After drawing, click "Set Boundary" again to enter the boundary information into the system. Click the "Preprocess" button to define the component forced lock flag and initial coordinates. Based on this, the state matrix can be initialized. For example:

[0146] (A) The layout characteristics of the virtual PCB are represented by the state matrix O t As shown in the following equation, since the virtual PCB is divided into a 20×20 grid, the initial size of O0 is also 20×20. The first row of the matrix has a value of 1, and the rest are 0. This means that during initialization, the first row is set as a layout-prohibited area, while the remaining areas have not yet been placed with components.

[0147]

[0148] If the first action places the component JP1 at coordinates (18, 3), and the length and width of the component occupy two X coordinate points and three Y coordinate points, respectively, then O1 is updated to the following formula, where element 2 means that the area is occupied by the new component after the current action is taken.

[0149]

[0150] (B) The characteristics of the circuit and the components themselves are determined by the state matrix P t As shown in the following formula, since the Cuk circuit contains 8 elements in total, P0 has 8 rows, and each row corresponds to the information of one element.

[0151] And since no components are placed in the initial stage, all element values ​​are 0.

[0152]

[0153] After the first step places JP1 at coordinates (3, 18), P1 is updated to the following formula. The first two elements, 3 and 18, represent that the component is placed at coordinates 3 in the X direction and 18 in the Y direction, respectively. Based on the current placement angle, the component occupies 2 grid points in the X direction and 3 grid points in the Y direction.

[0154]

[0155] Step (5): Click “AI Training” and the program will start to execute the layout algorithm and convert the state matrix O t and P t Input deep neural network, output quality vector Q t The deep neural network has three parts: the convolutional neural network that extracts PCB layout features, the graph convolutional neural network that extracts circuit diagram features, and the multi-layer perceptron that integrates the first two and outputs the final result. t The element value in represents the quality of the action of selecting the placement position of the component, and the number of elements is the number of actions. For example:

[0156] (A) In the convolutional extraction layout feature link, as shown in the following formula, the input matrix observation is extracted through two convolutional layers, and the output result is encoded into a vector of dimension 1, recorded as CNN OUT. Define the function g to encode the multi-dimensional matrix into a one-dimensional vector. Define conv1 as the first convolution layer, its input channel is 1, its output channel is 4, and the activation function is tanh. Considering the minimum element size to be a unit length, the convolution step is set to 1. In order to make the output feature map size the same as the input matrix, the convolution kernel size is set to 2, and the padding size is set to 1. Define conv2 as the second convolution layer, its input channel is 4, its output channel is 4, the activation function is tanh, the convolution step is set to 1, the convolution kernel size is set to 2, and the padding size is set to 0.

[0157] CNN OUT =g(tanh(conv2(tanh(conv1(O t )))))

[0158] (B) In the process of extracting circuit diagram features through graph convolution, we first need to construct an adjacency matrix containing the connection information of the graph. First, we perform graph analysis on the Cuk converter. The Cuk converter consists of eight components and can be regarded as having eight nodes. From this, we can obtain the adjacency matrix Adj representing the structure of the graph as shown below. Its eight rows and eight columns represent components C1, C2, L1, L2, JP1, JP2, Q1, and D1 respectively. If there is an electrical connection between two components, then the element in the adjacency matrix is ​​1, otherwise it is 0. Taking component C1 as an example (the first row or first column of the adjacency matrix Adj), Figure 2 It can be seen that C1 is electrically connected to L1, L2, Q1, and D1, so the (0, 2), (0, 3), (0, 6), and (0, 7) of the adjacency matrix Adj are all 1, and because the eight rows and eight columns represent the same sequence of elements, the adjacency matrix Adj is a naturally symmetric matrix.

[0159]

[0160] In order to combine the connection information between components with the characteristics of the components without losing the characteristic information of the components themselves, it is necessary to add self-connection to the adjacency matrix Adj to obtain the matrix A as shown in the following formula. Because each component is connected to itself, it is only necessary to set all the main diagonals of the adjacency matrix Adj to 1 to obtain the matrix A containing self-connection information.

[0161]

[0162] Based on the matrix A containing self-connection information, we define the degree matrix D as shown below. In graph theory, the degree of a node is equal to the number of nodes connected to it. A degree matrix containing self-connection information needs to take its own connections into account. This results in a diagonal matrix D with values ​​only on the main diagonal, and all other positions zero.

[0163] D=Diag[5 6 4 5 6 6 7 7]

[0164] Consider matrix A and feature matrix P t The dimensions are different. In order to avoid the direct multiplication of the two to affect the distribution of the original features, resulting in unbalanced maximum and minimum values, thus inducing the gradient explosion problem in the network training process. Therefore, the formula is defined as Symmetric normalized Laplace matrix The mathematical principle is to perform the formula on matrix A The normalization process shown is performed to eliminate the adverse effects caused by different dimensions and obtain the final matrix containing the connection information of the graph

[0165] Use the normalized adjacency matrix Multiply by the feature matrix P t After the weight gcn_weights operation of the graph convolution layer, the output result can be encoded into a vector of dimension 1, recorded as GCN OUT , as shown below.

[0166]

[0167] (C) Vector CNN OUT With GCN OUT Perform head-to-tail splicing encoding and input into the multi-layer perceptron operation to obtain the quality vector Q t .

[0168] (D) Since the reinforcement learning algorithm uses the DQN algorithm, the action space of the algorithm is discrete, so the component placement action should be defined in a discrete way. In this example, the optional actions are defined as placing the component "above", "below", "left", and "right" of the previous component at a rotation angle of 0°, 90°, 180°, or 270°. Therefore, the output layer of the deep neural network has 4×4 nodes, that is, the vector Q t The size is 1×16. The larger the element value, the better the quality of the selected action.

[0169] Step (6): The output of the deep neural network is the quality vector Q t , Q t Each element value q (ai) Indicates the quality of the action of selecting the component placement. At this time, the ε-greedy algorithm is applied to select action a based on the highest element value with a certain probability ε. t , randomly select an action with probability 1-ε.

[0170] The relationship between the number of learning iterations m and the probability ε is as follows:

[0171]

[0172] Where ε0 is the initial probability, 0<=η<=1 is the variable that adjusts the rate of change. In this example, η=1, m is the number of reinforcement learning iterations, and m max is the upper limit of m.

[0173] Step (7): After getting the next component placement action, call the virtual PCB board update program, add the new component according to the action, and get the reward R t For example, the evaluation indicators are as follows:

[0174] 1) The overlapping area of ​​components;

[0175] 2) The overlapping area between the regions enclosed by different potential points;

[0176] 3) Total trace length.

[0177] Considering that different issues in circuit layout require different levels of consideration and that mathematical calculations are performed on different scales, we assign a weight of 15 to the most severe component overlap metric of the three evaluation criteria, which has a different scale, to highlight its importance in learning. The potential overlap area, which has the same scale as the overlap area, is assigned a weight of 10. A weight that is too small will make it difficult to distinguish from the line length reward, while a weight that is too large will result in a significant disparity in reward changes between different operations, potentially leading to gradient explosion or vanishing during training. The weight of the line length metric remains at 1.

[0178] Step (8): Run the hierarchical power electronic circuit layout and wiring system based on reinforcement learning proposed by the present invention. In step t, the state vector O of the current layout is t and P t , input deep neural network, deep neural network operation output Q t , thus selecting action a t , get the new state vector O t+1 and P t+1 , and receive reward R according to the evaluation index t . Store the four sets of values ​​of step t (O t ,P t ), a t , R t ,(O t+1 ,P t+1 ).

[0179] Step (9): After completing a certain number of layout processes, randomly extract (O t ,P t ), a t , R t ,(O t+1 ,Pt+1 ) sample data, a round of training and learning is carried out through the reinforcement learning algorithm to adjust the internal parameters of the deep neural network Perform an update. The specific training and learning process is as follows:

[0180] Through a round of training and learning by reinforcement learning algorithm, the internal parameters of the deep neural network are Perform an update. The specific training and learning process is as follows:

[0181] (A) According to the following formula, the R of the sample data is calculated respectively. t and (O t+1 ,P t+1 ) for processing

[0182]

[0183] R t Indicates the reward after each step, max(·) indicates the selection of the maximum element; k (i.e. a t ) represents the output quality vector Q of the deep neural network t The kth element in .

[0184] (B) Due to the real q k Given by the deep neural network, it is necessary to adjust the weight and bias parameters of the deep neural network Make q k accessible The loss function Loss is defined as the above formula (11).

[0185] (C) Calculation parameters For the gradient of Loss, it can be adjusted by the following formula

[0186]

[0187] 0.001 is the learning rate of the deep neural network. After training, the loss becomes smaller, indicating that the deep neural network has updated its design strategy.

[0188] The corresponding process of steps (5) to (9) is as follows: Figure 6 As shown. The overall summary can be as follows: (1) the state matrix is ​​input into the deep neural network; (2) according to the output quality vector Q t , the ε-greedy algorithm takes action a t ; (3) After updating the layout strategy, the action can be evaluated and a reward R is given t ; (4) Update q according to formula (27) * ; (5) Fit q ​​through deep neural network *, so the loss function equation (11) is established, and the NN parameters are adjusted according to equation (28).

[0189] Step (10): When all 8 components are laid out, one round of operation ends and the next round should be restarted from the initial state (O0, P0).

[0190] Step (11): When the number of training times reaches the maximum value m max , or the maximum training time is reached, the training is considered complete. For example, the average time taken to obtain the optimal layout result of the Cuk converter using the parallel training method proposed in the present invention is only 526 seconds. The design results are as follows Figure 7 shown.

[0191] Step (12): Click “Route” and the program will execute the routing algorithm. The basic principle and implementation steps of the routing algorithm are similar to those of the layout algorithm. The only difference is the specific problem modeling and the use of the relatively simpler and faster Q-table in the reinforcement learning method. Therefore, I will not go into details. Based on the optimal layout result, the routing result is as follows Figure 8 shown.

[0192] An embodiment of the present invention provides an electronic device, comprising: a computer-readable storage medium and a processor;

[0193] The computer-readable storage medium is used to store executable instructions;

[0194] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method described in any one of the above embodiments.

[0195] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method described in any of the above embodiments.

[0196] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hierarchical power electronic circuit layout method based on reinforcement learning, characterized in that: include: S1, builds a deep neural network as an intelligent agent; The input of the intelligent agent is the layout state matrix O of each grid point after gridding the PCB board and the coordinate state matrix P of each component of the circuit to be laid out. The layout state of each grid point includes no component and within the layout range, outside the layout range, occupied by a component placed in the past action, and occupied by a component placed in the current action. The coordinate state of each component includes the X and Y coordinates of each component and the number of grid points occupied by the component in the X and Y directions when the coordinates are used as the reference point. The output is the quality vector Q of all possible layout actions of the circuit. The layout action is the direction of the next component placement, and the quality is the probability of obtaining the optimal layout result when selecting each layout action. The deep neural network includes CNN, GCN and MLP. S2, the layout state matrix O of step t t and the coordinate state matrix P t Input to the agent, CNN from O t Extract the layout features of the PCB plane, GCN from P t Extract circuit diagram features from the MLP, and generate Q according to the layout and circuit diagram features. t ; Select an action a from t , get the layout state matrix O of step t+1 t+1 and the coordinate state matrix P t+1 , and evaluate it through the reward rule to get the reward R t ; O t 、P t 、a t 、R t , O t+1 、P t+1 As a set of data, it is stored in the memory pool; where t = 0, 1, 2, ... K, where K is the number of components in the circuit to be laid out; the reward rule is: the placement positions of the components do not conflict and the total routing length is as short as possible; S3, repeating S2 until all components are placed in a preset order; randomly extracting a batch of data from the memory pool to train the agent; wherein the preset order is consistent with the power flow direction of the circuit; S4, repeating S2-S3 until the training stop condition is met, and using the trained intelligent agent to obtain the optimal layout result of the circuit to be laid out; Based on the reward rule, set the reward function r = r overlap +r overlap_net_area +r hpwl ; Among them, r overlap is the component overlap area bonus, len i (c_list) represents the number of components on each grid point of the i-th PCB board, a>1, n×m is the total number of grid points on the PCB board; r overlap_net_area The reward is the overlapping area between any two groups of pins with different potentials after they are connected. N is the number of categories of pin potentials in the circuit to be laid out, net i 、net j are pins of groups i and j respectively, where i>j; r hpwl is the total trace length bonus, Δx i , Δy i They are the maximum horizontal and vertical spacings between the pins in the pin group of the i-th potential.

2. The method according to claim 1, wherein The preset order of all components is obtained by inputting information of each component of the circuit and the connection relationship of each component into the LLM.

3. The method according to claim 2, wherein The LLM further comprises a verification and post-processing module for checking whether the number of elements in the generated element sequence is correct, and if not, inserting the missing elements into the element sequence.

4. The method according to claim 1, wherein When training the agent, the weight parameters of the deep neural network are updated in the GPU, and the memory pool is updated in the CPU.

5. The method according to claim 4, wherein When laying out multiple circuits to be laid out, deep neural networks of multiple agents are trained simultaneously in the GPU; the multiple agents correspond one-to-one to the multiple circuits to be laid out, and are respectively used to obtain the optimal layout results of the corresponding circuits to be laid out.

6. A hierarchical power electronic circuit layout and routing method based on reinforcement learning, characterized in that: The method according to any one of claims 1 to 5 is used to obtain an optimal layout result of the circuit to be laid out, that is, the position of each pin of each component of the circuit to be laid out, so as to perform wiring. The wiring method includes: S1, build an intelligent agent based on Q-table; where the state is l×n×m coordinate points on the virtual PCB board, and the action is to move from the current coordinate point to "above", "below", "left", or "right" of the x-th layer; where x = 1, 2…l; define the quality matrix Q' as the quality of each action for all coordinate points on each layer, and define the action reward value for all coordinate points on each layer as the reward matrix R'; S2: Divide the components into N groups according to their pin potentials. In the yth group of pins, take the zth pin and the z-1th group of traces as the end points, and the z+1th pin as the starting point to form the zth group of starting and ending points of the yth group of pins, and initialize its mass matrix Q. y,z ` and the reward matrix R y,z `; where y = 1, 2…N, z = 1, 2…M y -1,M y is the total number of pins in the yth group, and the z-1th group of traces is the line between the zth pin and the z-1th pin; the initial Q y,z All elements in ` are 0; if the initial R y,z If the coordinate point corresponding to the element in ` is the end point, the value of the element is positive. If the coordinate point is a pin in a different group or is occupied by a trace in a different group, the value of the element is negative. Otherwise, the value of the element is a smaller negative value. S3, based on the current coordinate point and the current mass matrix Q y,z `Select action, according to the initial R y,z `And reward rules determine its reward value, and update Q according to the reward value and Q-learning Q value calculation formula y,z `; The reward rules are: the total routing length is as short as possible, and the routing paths of different potential points cannot overlap; S4, repeating S3 until a preset reward value is obtained after taking the latest action; the preset reward value is the end point reward value; S5, repeat S2-S4 until the training stop condition is met; use the trained agent to obtain the optimal routing result of the zth group of starting and end point combinations; S6, repeating S2-S5 until all pins are connected, and finally obtaining the optimal wiring result of the circuit to be wired.

7. An electronic device, characterized in that: include: Computer-readable storage medium and processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 6.

Images