Hierarchical power electronic circuit layout method based on reinforcement learning

Abstract

The invention discloses a hierarchical power electronic circuit layout method based on reinforcement learning, which belongs to the field of integrated circuits and mainly comprises three levels of element sequence generation, layout optimization and wiring optimization. The method comprises the following steps: firstly, analyzing the power flow direction of a power electronic circuit, and reasonably arranging an element layout sequence, so as to realize basic optimization from a top layer; then, on the basis of the element sequence, a convolutional neural network is used for extracting layout features, a graph convolutional neural network is used for extracting graph features of a circuit connection relation, specific design rules of the power electronic circuit are embedded into a reinforcement learning reward system, and a deep Q network (DQN) is used for achieving layout of elements; through reinforcement learning, the intelligent agent can summarize experience in trial and error without human intervention, and finally a reward maximized layout result is generated.

Description

Technical Field

[0001] The present invention belongs to the field of integrated circuits, and more specifically, relates to a hierarchical power electronic circuit layout method based on reinforcement learning. Background Art

[0002] With the continuous development of power electronics technology, the requirements for the efficiency, power density and reliability of power electronic circuits are constantly increasing, so the factors to be considered in the design are becoming more and more complex. For example, the design of power electronic circuits needs to separate the power routing and signal routing, the "commutation loop" in the power loop needs to be minimized, different voltage levels need to be reliably isolated, and local heating devices need to have sufficient heat dissipation capabilities. In order to ensure the performance of the designed power electronic circuits, engineers with rich product design experience and multidisciplinary knowledge are usually required to perform repeated iterative design and testing, which is very time-consuming, labor-intensive and costly. In order to improve design efficiency, various electronic design automation (EDA) tools are usually needed to assist designers in completing the tedious and mechanical content of the design process, checking and avoiding possible design risks, thereby reducing the number of modification iterations.

[0003] After years of research and development, general electronic automation design tools can now realize automated layout and routing, but they are basically only developed for electronic circuits, and only focus on signal flow and connectivity issues when solving related problems. Due to the particularity of power electronic circuit design, there are a large number of components and different packages. The layout layer, position and angle will affect the design results, so there is a very high design solution space. Connectivity is no longer the only optimization goal. Factors such as heat, electricity, electromagnetic interference (EMI) and power density will have a huge impact on the design results and need to be included in the evaluation indicators.

[0004] Therefore, the existing electronic automation design tools are not applicable in the field of power electronics. There is still a need to develop targeted intelligent layout methods that can meet various power electronics design requirements to assist power electronics designers in solving design problems efficiently and accurately. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a hierarchical power electronic circuit layout method based on reinforcement learning, thereby solving the technical problems that the existing power electronic circuit layout design is highly dependent on expert knowledge and experience, has a long design cycle, high cost, and a single optimization target.

[0006] To achieve the above object, according to a first aspect of the present invention, a method for hierarchical power electronic circuit layout based on reinforcement learning is provided, comprising:

[0007] S1, building a deep neural network as an agent;

[0008] The input of the agent is the layout state matrix O of each grid point after the PCB board is gridded and the coordinate state matrix P of each component of the circuit to be laid out. The layout state of each grid point includes no component and within the layout range, outside the layout range, occupied by components placed by past actions, and occupied by components placed by current actions. The coordinate state of each component includes the X and Y coordinates of each component, and the number of grid points in the X and Y directions occupied by the component when the coordinates are used as reference points. The output is the quality vector Q of all possible layout actions of the circuit. The layout action is the direction of the next component placement, and the quality is the possibility of obtaining the optimal layout result when selecting each layout action. The deep neural network includes CNN, GCN and MLP.

[0009] S2, the layout state matrix O of step t t and the coordinate state matrix P t Input to the agent, CNN from O t Extract the layout features of the PCB plane from P t The circuit diagram features are extracted from the MLP, and the Q t ; Select an action a from t , get the layout state matrix O of step t+1 t+1 and the coordinate state matrix P t+1 , and evaluate it through the reward rule to get the reward R t ; O t , P t 、a t , R t , O t+1 , P t+1 As a set of data, it is stored in the memory pool; wherein, t = 0, 1, 2, ... K, K is the number of components of the circuit to be laid out; the reward rule is: the placement positions of the components have no conflicts and the total routing length is as short as possible;

[0010] S3, repeat S2 until all components are placed in a preset order; randomly extract a batch of data from the memory pool to train the intelligent agent; wherein the preset order is consistent with the power flow direction of the circuit.

[0011] S4, repeat S2-S3 until the training stop condition is met, and use the trained intelligent agent to obtain the optimal layout result of the circuit to be laid out.

[0012] According to a second aspect of the present invention, a method for layout and wiring of a hierarchical power electronic circuit based on reinforcement learning is provided, wherein the method described in the first aspect is used to obtain the optimal layout result of the circuit to be laid out, that is, the position of each pin of each component of the circuit to be laid out, so as to perform wiring, and the wiring method comprises:

[0013] S1, build an intelligent agent based on Q-table; where the state is the l×n×m coordinate points of the virtual PCB board, and the action is to move from the current coordinate point to the "above", "below", "left", or "right" of the xth (x=1,2…l) layer; define the quality matrix Q` as the quality of each action of all coordinate points in each layer, and define the action reward value of all coordinate points in each layer as the reward matrix R`;

[0014] S2, divide the components into N groups according to the potential of each pin of each component. In the yth group of pins, take the zth pin and the z-1th group of routing as the end point, and the z+1th pin as the starting point to form the zth group of starting point and end point combination of the yth group of pins, and initialize its quality matrix Q y,z ` and the reward matrix R y,z `; where y = 1,2…N, z = 1,2…M y -1,M y is the total number of pins in the yth group, and the z-1th group of routing is the connection between the zth pin and the z-1th pin; the initial Q y,z All elements in ` are 0; if the initial R y,z The coordinate point corresponding to the element in is the end point, the value of the element is positive, if the coordinate point is a pin of a different group or is occupied by a routing line of a different group, the value of the element is negative, otherwise the value of the element is a smaller negative value;

[0015] S3, based on the current coordinate point and the current mass matrix Q y,z `Select an action, based on the initial R y,z ` and reward rules to determine its reward value, and update Q according to the reward value and Q-learning Q value calculation formula y,z `; The reward rules are: the total routing length is as short as possible, and the routing paths of different potential points cannot overlap;

[0016] S4, repeating S3 until a preset reward value is obtained after taking the latest action; the preset reward value is the end point reward value;

[0017] S5, repeat S2-S4 until the training stop condition is met; use the trained intelligent agent to obtain the optimal wiring result of the zth group of starting point and end point combinations.

[0018] S6, repeat S2-S5 until all pins are connected, and finally obtain the optimal wiring result of the circuit to be wired.

[0019] According to a third aspect of the present invention, there is provided an electronic device, comprising: a computer-readable storage medium and a processor;

[0020] The computer-readable storage medium is used to store executable instructions;

[0021] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect or the second aspect.

[0022] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a processor to execute the method described in the first aspect or the second aspect.

[0023] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0024] The hierarchical power electronic circuit layout method based on reinforcement learning provided by the present invention is mainly divided into three levels: component sequence generation, layout optimization and wiring optimization. First, the power flow analysis for the power electronic circuit is used to reasonably arrange the component layout order, so as to achieve basic optimization from the top level; then, based on the component sequence, the convolutional neural network is used to extract the layout features, the graph convolutional neural network is used to extract the graph features of the circuit connection relationship, and the unique design rules of the power electronic circuit are embedded in the reward system of reinforcement learning, and the deep Q network (Deep Q Network, DQN) is used to realize the layout of components; through reinforcement learning, the intelligent agent can summarize experience in trial and error without human intervention, and finally generate a layout result with maximum reward; the power flow of the power electronic circuit obtained based on the circuit modal analysis is used as a reference for formulating the component layout sequence, and the physical laws and expert experience are used as the prior knowledge of deep neural network learning to improve the algorithm optimization efficiency. In addition, the hierarchical power electronic circuit wiring method based on reinforcement learning provided by the present invention, based on the optimal layout result obtained by the above-mentioned layout method, adopts a reinforcement learning method based on Q-table to quickly complete the wiring. In summary, the layout and wiring method provided by the present invention converts the layout and wiring problem of power electronic circuits into a problem similar to "playing chess" or "walking a maze", uses deep reinforcement learning to continuously try various layout and wiring structures, and efficiently searches for the optimal layout and wiring results in the process of trial and error and learning, avoiding complex manual design and iterative processes, and improving design efficiency; converts the conflict rules and optimization goals of layout and wiring into reinforcement learning rewards, thereby converting the layout and wiring problem into a reinforcement learning problem of finding the maximum sum of rewards, and the deep neural network can automatically explore and learn the layout and wiring results that can maximize the rewards. In particular, a simplified method is used to evaluate the line length index in the layout optimization process, which realizes the decoupling of the layout process and the wiring process and reduces the time complexity.

[0025] Furthermore, when training the intelligent agent, the present invention preferably adopts a method of making full use of the CPU and GPU, splits the program and distributes it to the CPU or GPU for calculation according to the calculation requirements, and adopts a programming method of multi-threaded synchronous calculation, which realizes the simultaneous evaluation of component layout optimization based on different sequences and shortens the program running time; on this basis, a deep neural network "equivalent parallel training" model is adopted to simultaneously train multiple DQNs on a single-core GPU, which can simultaneously learn and layout and route multiple component sequences, significantly accelerating the entire design process and further improving the program running efficiency.

[0026] Furthermore, the present invention utilizes the world knowledge of LLM to obtain the generation of the component sequence of the circuit to be laid out, and fully utilizes the LLM's powerful understanding ability and learning ability of relevant knowledge and experience of power electronics, and uses it for global planning and detail assistance of layout and routing optimization problems, completing tasks such as component sequence generation, user language and algorithm language conversion, and hierarchical design process interaction; fully combining the high flexibility but uncertain output of LLM and the autonomous learning and trial-and-error characteristics of reinforcement learning, it provides a new idea for solving layout and routing problems, reduces dependence on artificial expert experience, and improves design efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of a method for layout of hierarchical power electronic circuits based on reinforcement learning provided by an embodiment of the present invention;

[0028] Figure 2 A topological diagram of a Cuk converter provided in an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a Cuk converter drawn in Altium Designer software according to an embodiment of the present invention;

[0030] Figure 4 The operating interface designed based on Python language provided by the embodiment of the present invention;

[0031] Figure 5 A current loop when a switch of a Cuk converter provided by an embodiment of the present invention is turned on;

[0032] Figure 6 A current loop when the switch of the Cuk converter provided by an embodiment of the present invention is disconnected;

[0033] Figure 7 The optimal layout result of the Cuk converter obtained by using the layout method provided by the embodiment of the present invention;

[0034] Figure 8 The optimal wiring result of the Cuk converter is obtained by using the layout method provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0036] The traditional layout and wiring process of power electronic circuits is cumbersome and complicated. It requires experienced professionals to go through multiple rounds of iterations and modifications to complete the design. It is time-consuming, labor-intensive and costly, so it is urgently needed to be assisted by new technologies and new ideas. The layout and wiring problem of power electronic circuits is essentially to find the optimal solution in a huge solution space. To address this problem, artificial intelligence (AI) is an effective method. Among the various AI methods, reinforcement learning (RL) allows continuous exploration in the search space and continuous learning from experience in the trial and error process, and ultimately efficiently finds the optimal solution. This process does not require human intervention, so it can significantly reduce the design burden on designers. Based on this, an embodiment of the present invention provides a hierarchical power electronic circuit layout method based on reinforcement learning, such as Figure 1 As shown, including:

[0037] S1, building a deep neural network as an agent;

[0038] The input of the agent is the layout state matrix O of each grid point after the PCB board is gridded and the coordinate state matrix P of each component of the circuit to be laid out. The layout state of each grid point includes no component and is within the layout range, occupied by components placed by past actions, outside the layout range, and occupied by components placed by current actions. The coordinate state of each component includes the X and Y coordinates of each component, and the number of grid points in the X and Y directions occupied by the component when the coordinates are used as reference points. The output is the quality vector Q of all possible layout actions of the circuit. The layout action is the orientation of the next component placement, and the quality is the possibility of obtaining the optimal layout result when selecting each layout action. The deep neural network includes a convolutional neural network, a graph convolutional neural network, and a multi-layer perceptron.

[0039] S2, the layout state matrix O of step t t and the coordinate state matrix P t Input to the agent, the convolutional neural network is from O t The graph convolutional neural network extracts the layout features of the PCB plane from P t The multi-layer perceptron generates Q according to the layout and circuit diagram features. t ; Select an action a from t , get the layout state matrix O of step t+1 t+1 and the coordinate state matrix P t+1 , and evaluate it through the reward rule to get the reward R t ; O t , P t 、a t , R t , Ot+1 , P t+1 As a set of data, it is stored in the memory pool; wherein, t = 0, 1, 2, ... K, K is the number of components of the circuit to be laid out; the reward rule is: the placement positions of the components have no conflicts and the total routing length is as short as possible;

[0040] S3, repeat S2 until all components are placed in a preset order, randomly extract a batch of data from the memory pool, and use a reinforcement learning algorithm to train the intelligent agent; wherein the preset order is consistent with the power flow direction of the circuit.

[0041] S4, repeating S2-S3 until a preset condition is met, and using the trained intelligent agent to obtain the optimal layout result of the circuit to be laid out.

[0042] Specifically, the method provided by the present invention comprises the following steps:

[0043] (1) Define a two-dimensional state matrix O t and P t , O t It is used to describe the coordinate occupancy information in the virtual PCB layout, indicating whether there are components placed at the coordinate point; t Used to describe the characteristics of each component, including the current coordinates and placement angle.

[0044] Among them, the layout state matrix O t It represents the current layout characteristics of the virtual PCB board, which is an n×m matrix. In essence, the virtual PCB board is divided into n×m grids (it can be understood that when the layout area of ​​the PCB board is a square, n=m). If there is no component on the grid point and it is within the allowable layout range, the matrix element value of the point is 0; if a component is placed on the grid point (that is, it has been occupied by a component placed by a past action) or it is outside the allowable layout range, the matrix element value of the point is 1; to highlight the impact of the "current action", if the grid point is occupied by a component placed by the current operation, the matrix element value of the point is 2.

[0045] Coordinate state matrix P t The characteristics of the circuit to be laid out are represented by a K×4 matrix, where K is the number of components in the circuit to be laid out. Each row of the matrix represents a component, and the four elements in a row represent the X and Y coordinates where the component is placed and the number of grid points in the X and Y directions occupied by the component with the coordinates as the reference point.

[0046] Among them, the placement order of each component in the circuit to be laid out can be obtained by manually analyzing the power flow direction of the circuit to be laid out.

[0047] Considering that the newly emerging large language model provides a new perspective for solving many problems with its rich knowledge and powerful natural language understanding capabilities, but because LLM is essentially a probabilistic generation model, the output is uncertain. If the flexibility of the large language model and the trial-and-error characteristics of reinforcement learning can be combined together, it will inevitably bring new breakthroughs in solving layout problems. Based on this, in order to improve the efficiency of obtaining the placement order, as a further optimization of the present invention, LLM is used to replace the above-mentioned manual analysis process, that is, the preset order of all the components is obtained by inputting the information of each component of the circuit and the connection relationship of each component into LLM.

[0048] Specifically, the netlist of a power electronic circuit is input, and LLM uses its extensive knowledge and understanding capabilities to generate a component sequence that meets the power flow requirements by analyzing the netlist and specific prompts.

[0049] Using circuit design software such as Altium Designer, a netlist containing component information and connection relationship information can be directly exported from the circuit schematic. The netlist is input into LLM with corresponding prompts, which include requirements for LLM, supplementary descriptions of the circuit, and input and output examples. The requirements are that LLM is expected to output a set of component sequences after reading the netlist information; the supplementary description includes a description of the basic information and characteristics of the input circuit, such as the topology type of the circuit, the input, output, and commutation loop information. These supplements will be used by LLM first, thereby increasing the possibility of successful optimization; the input and output examples are a combination of other input netlists and output component sequences, which are used to provide reference and optimization direction for LLM. After receiving this netlist input, LLM can reconstruct it into a power electronic circuit topology and perform operating modal analysis to generate a component sequence that meets the power flow requirements.

[0050] Since the sequence generated by LLM may not be completely correct or meet specific design requirements, in order to ensure the correctness and practicality of the generated component sequence, as a further preference of the present invention, the LLM also includes a verification and post-processing module for checking whether the number of components in the generated component sequence is correct. If not, the missing components are inserted into the component sequence.

[0051] Specifically, a verification and post-processing procedure is added to the LLM to check whether the number of components in the generated component sequence is correct. If missing components are found, a random placement strategy can be adopted to insert them into the component sequence.

[0052] If it is not possible to directly determine the most reasonable sequence from the generated series of component sequences, each sequence can be used as input to train the corresponding intelligent agent and then select the optimal layout result.

[0053] Define two lists, Library and Component, to convert the component model package and layout status (including position coordinates, placement angle, potential label, component name and special attribute flag) into digital expression, which is used to store component information and node connection network.

[0054] The expression of the component package table Library is as follows:

[0055] Library={Footprint,Size,N1,N2,…,N T}

[0056] Where Footprint is a string representing the package name of the component. Size is a 1×N vector. The size of N is related to the shape of the package and is the number of parameters required to uniquely describe the package shape. For example, if the package is a rectangle, then the value of N is 2. The first element in Size is the length of the component in the X direction on the virtual PCB board, and the second element is the length of the component in the Y direction on the virtual PCB board. N j (j=1,2…T) represents the position of each pin of the component in the package, and T is the total number of pins contained in the component. j It is a 1×2 vector that describes the coordinates of the pin relative to the package reference point (the point in the upper left corner). The package name Footprint can be used to accurately index the required package information, and then the package size Size and pin coordinate N can be used to accurately index the required package information. j (j=1,2…T) can define the space occupied by a component on the virtual PCB board, laying the foundation for subsequent layout and routing operations.

[0057] The expression of the component information table Component is as follows:

[0058] Component={Footprint,Angle,P,N,Name,Flag1,Flag2}

[0059] Among them, Footprint is a string, which represents the package name of the component. Angle is also a string, which represents the placement angle of the component, that is, the component is rotated 0°, 90°, 180° or 270° relative to the definition of the default package. P represents the coordinates of the component placed on the virtual PCB board, which is a 1×2 vector, in which the first element represents the coordinate in the X direction and the second element represents the coordinate in the Y direction. N represents the potential of each pin of the component (pins with the same potential need to be connected), which is a 1×T vector, T is the total number of pins contained in the component, and each element in N corresponds to a pin one by one, defining the potential number of each pin, and there is an electrical connection between pins with the same potential number. Name is a string, which represents the name of each component and is used to distinguish different components. Flag1 and Flag2 are both scalars with values ​​of 0 or 1, representing whether the position of the component is fixed and cannot be moved and whether the component has been placed on the virtual PCB board.

[0060] (2) Define a multi-input and multi-output deep neural network as the agent, with the state matrix O at step t t and P t As input to the agent (i.e., deep neural network), the agent outputs the quality vector Q of all possible layout actions for the circuit to be laid out. t The layout action refers to the position of the next component placement, and the quality refers to the possibility of selecting each action pair to obtain the optimal layout result. The higher the action quality, the more likely it is to obtain the optimal layout result in the end.

[0061] Among them, the deep neural network has two input channels, namely the Convolutional Neural Network (CNN) channel and the Graph Convolution Network (GCN) channel. The Convolutional Neural Network is used to extract the layout features of the PCB plane, and the Graph Convolutional Neural Network is used to extract the circuit diagram features. The output results of the two channels are then concatenated and encoded, and finally the encoded results are decoded by the Multilayer Perceptron (MLP) to obtain the final action output, which is as follows:

[0062] 1) State matrix O t Sent into the CNN input channel, through the convolutional neural network, the two-dimensional matrix O t is mapped to a vector CNN of size 1×c OUT :

[0063]

[0064] represents a convolutional neural network, whose structure needs to be appropriately selected according to the complexity of the design. Represents the trainable parameters in the convolutional neural network, namely weights and biases. g(·) means encoding a multidimensional matrix into a one-dimensional vector. This operation is to subsequently concatenate the outputs of deep neural networks of different dimensions together for subsequent calculations.

[0065] 2) State matrix P t Sent into the GCN input channel, through the graph convolutional neural network, the two-dimensional matrix P t is mapped to a vector GCN of size 1×g OUT :

[0066]

[0067] represents a graph convolutional neural network, Represents the trainable parameters in a graph convolutional neural network.

[0068] 3) Output the convolutional neural network to CNN OUT And graph convolutional neural network output GCN OUT The head and tail are concatenated and then sent to the multi-layer perceptron for decoding, and finally a vector Q of size 1×s is obtained t :

[0069]

[0070] represents a multi-layer perceptron, represents the trainable parameters in the multi-layer perceptron, and v(·) represents the CNN OUT and GCN OUT Concatenate the ends into a 1×(c+g) vector.

[0071] 4) Quality vector Q t As shown below:

[0072] Q t ={q(a1),q(a2),…,q(a t )}

[0073] Among them, a i (i=1,2…s) is the action of placing the element, and s is the number of optional actions. Each element value Indicates "select component placement action a i "The quality of the action. The vector Q t The size is 1×s.

[0074] 5) All possible actions are the optional orientations for placing components. The quality of an action is a numerical value, and the higher the value, the better the action. Selecting this action is more likely to generate an optimal PCB layout.

[0075] (3) According to all the element values in vector Q t select an action a from the action space using the ε-greedy algorithm t , that is, an orientation for placing a component, to obtain the state matrices O t+1 and P t+1 at the (t + 1)-th step, specifically including:

[0076] 1) Apply the ε-greedy algorithm, that is, with a certain probability ε, select the action a according to the maximum element value in Q t , and randomly select an action with a probability of 1 - ε. The action a t is a value, 1 < a t < s, representing the orientation for placing the component corresponding to the a t -th element in vector Q t . Different values of a t represent different orientations for placing components. t

[0077] 2) After selecting the orientation for placing the component, place the new component on the virtual PCB board by selecting this action, and then the state matrices O t and P t at the (t + 1)-th step can be obtained from the current state matrices O t+1 and P t+1 .

[0078] (4) Define a reward rule to evaluate the state matrices O t+1 and P t+1 (that is, the new virtual PCB board layout) to obtain the reward R t . It can be understood that the reward R t is used to train the deep neural network. The higher the R t , the greater the possibility of taking the corresponding action when encountering the same input O t and P t in the future; conversely, the smaller the R t , the smaller the possibility of taking the corresponding action when encountering the same input O t and P t in the future.

[0079] The said reward rule is: The placement positions of each component have no conflicts and the total trace length is as short as possible. Based on this reward rule, the reward can be measured comprehensively by multiple indicators, such as:

[0080] 1) The overlapping area of components;

[0081] 2) The overlapping area between the regions enclosed by different potential points;

[0082] 3) Total trace length.

[0083] Among them, the component overlap area reward r overlap is the number of overlapping coordinate points raised to the power of a (a>1), as shown in the following formula. i (c_list) represents the number of components at the i-th coordinate point (the range beyond the boundary is considered to be always distributed with a large component). If there is no component distributed at this coordinate point, it will not participate in the calculation. overlap The physical meaning is that as the overlapping area increases, its impact on the layout becomes more and more serious.

[0084]

[0085] The overlap reward r between the areas surrounded by different potential points overlap_net_area It is the overlapping area between the areas enclosed by any two groups of pins with different potentials after they are connected, as shown in the following formula, where S represents the intersection of the area enclosed by the ith potential and the area enclosed by the jth potential. Since its impact on the layout is less than the problem of component overlap, it is not squared.

[0086]

[0087] Considering that the layout algorithm needs to constantly evaluate the line length, efficiency is an important factor, so the line length reward r hpwl The Half-Perimeter Wire Length (HPWL) model is used. HPWL is the most commonly used routing mode for automatic layout and routing algorithms. It only cares about the lengths of the top, bottom, left, and right sides, as shown in the following formula, where Δx i , Δy i They are the maximum spacing between the pins in the pin group of the i-th potential in the horizontal X direction and the vertical Y direction, Δx i For example, Δx i It is the difference between the x-coordinate of the rightmost pin and the x-coordinate of the leftmost pin in the horizontal X direction under the i-th group of potential distribution. This formula means the half perimeter of the rectangle surrounded by the two pins on the diagonal line. If there are N potentials in the circuit, the reward is r hpwl It is the accumulation of the sum of the maximum differences in the horizontal X direction and the vertical Y direction under each potential distribution. The main features of HPWL are simplicity, fast evaluation, and an actual wiring length error of about 10%.

[0088]

[0089] Each time the agent takes an action, it places a component on the virtual PCB and calculates the current reward. Once all components are placed, the current round of learning stops and the next round begins.

[0090] It can be understood that the potential information of each pin required for calculating the above-mentioned reward is obtained from the component information table Component.

[0091] It should be noted that the indicators for measuring rewards are not fixed, and can be further adjusted, refined or added with more evaluation indicators according to the operating indicators of the circuit to be laid out and wired. For example, factors such as electromagnetic interference, loss and temperature rise of the circuit can also be considered, and "the routing with high frequency and high current is as short as possible" and "the peak temperature of the virtual PCB board is as low as possible" can be added as evaluation indicators to meet the design requirements. The above-mentioned reward indicators and the calculation formulas of each indicator are only used as examples, and the embodiments of the present invention do not limit them to uniqueness.

[0092] (5) Use reinforcement learning algorithms to use rewards to modify the weights and biases of the agent (i.e., deep neural network). t ,P t ), a t , R t , (O t+1 ,P t+1 ) is stored in the memory pool as a set of data, and a batch of data is randomly extracted from the memory pool to train the agent (i.e., deep neural network) through the reinforcement learning algorithm to update the weights and biases of the deep neural network.

[0093] The memory pool contains a large number of (O t , P t ), a t , R t , (O t+1 , P t+1 ) is a group of sample data. It has a maximum capacity. When the sample data reaches the maximum capacity, the first-in-first-out principle is adopted to eliminate the sample data that first enters the memory pool. Randomly extracting a batch of sample data is to randomly select a specified number of sample data from all the data in the memory pool.

[0094] A batch of data is randomly extracted from the memory pool to train the deep neural network through the reinforcement learning algorithm, and the weights and biases of the deep neural network are updated. The specific process is as follows:

[0095] 1) Use the DQN algorithm to train a deep neural network. The training goal is to obtain a layout strategy with the highest possible reward. The formula of the DQN algorithm is:

[0096]

[0097] Rt represents the reward after each step, max(·) represents the selection of the maximum element; k represents the output vector Q of the deep neural network t The kth one in the above corresponds to the selected component placement action, i.e., a t ; Represents a complete deep neural network consisting of a convolutional neural network, a graph convolutional neural network, and a multilayer perceptron.

[0098] 2) Due to the real q k Given by the deep neural network, it is necessary to adjust the weights and bias parameters of the deep neural network Make q k Can approach The loss function is defined as:

[0099]

[0100] 3) By calculating parameters For the gradient of Loss, it can be adjusted by the following algorithm

[0101]

[0102] Where α is the learning rate of the deep neural network. After training, the loss becomes smaller, indicating that the deep neural network has updated its design strategy.

[0103] (6) Define the “step” of the above strategy as: according to O t and P t , the deep neural network outputs Q t , and then choose action a t , get O t+1 ,P t+1 and R t The "one working cycle" of the above strategy is defined as: repeating S1 and S3, so that the layout state of the virtual PCB board goes through a round from the initial state. When one round is completed, the next round is restarted. The sign of stopping the deep neural network training is defined as: when the requirements for successful deep neural network training or the preset maximum number of training times are met, the operation stops.

[0104] It can be understood that in the initial state, the state matrix O t The element value corresponding to the allowed layout area is 0, and the element value corresponding to the area outside the preset layout range is 1; the state matrix P t Then all elements are 0.

[0105] Each time you run a step, you get an action a t , that is, place the component at the specified position, and then update the state matrix to O t+1 and Pt+1 , and receive a reward R t When all components are placed, the circuit layout is complete and the current round of learning stops. When the preset maximum number of training times is reached or the final reward accumulation value reaches a certain threshold, the training stops.

[0106] In order to improve the learning efficiency of the system, as a further optimization of the present invention, when training the intelligent agent, the weight parameter update of the deep neural network is completed in the GPU, and the update of the memory pool is completed in the CPU.

[0107] Specifically, the training process uses a training method that combines CPU and GPU to improve the learning efficiency of the system. By adopting a multi-process programming method, the program that takes a long time to "train the neural network, update the weight parameters" is placed in the GPU with a faster matrix operation speed and runs as the main process; the programs that take a short time such as "placing components", "updating simulated PCB boards", and "calculating rewards" (collectively referred to as updating memory pool programs) are placed in the CPU and run as sub-processes. In this way, the CPU and GPU can divide the work and cooperate with each other. The main process is always training the neural network, and the sub-process is always updating the memory pool, so that the program duration is always determined by the program duration of training the neural network. In addition, the sub-process can make full use of time to simultaneously update the memory pool under multiple component sequences, thereby evaluating the impact of different sequences on the layout results.

[0108] When laying out multiple circuits to be laid out, in order to further improve the learning efficiency of the system, as a further optimization of the present invention, deep neural networks of multiple agents are trained simultaneously in the GPU; the multiple agents correspond one-to-one to the multiple circuits to be laid out, and are respectively used to obtain the optimal layout results of the corresponding circuits to be laid out.

[0109] When training a neural network on a GPU, the inputs of several neural networks are concatenated into a large matrix X = [X1...X i ... X n ], the weights are also concatenated into a large weight matrix W = [W1 ... W i ... W n ], and the same is true for bias. By computing these large matrices in parallel, it is possible to perform operations on multiple neural networks at the same time (i.e., several agents perform parallel layout and routing of multiple circuits to be laid out and routed at the same time).

[0110] On this basis, by summing and averaging the loss functions and performing reverse gradient propagation on the total loss function, we can achieve the effect of parallel training of multiple neural networks on a single-core GPU. Therefore, the loss function should be changed to:

[0111]

[0112] The essence of this training method is not to train multiple different neural networks separately at the same time, but to calculate the loss functions of multiple neural networks in sequence and then perform backpropagation together as a large neural network model. The calculation of the loss function is still performed in series, but the backpropagation calculation process achieves a parallel effect. Therefore, this model is called the "equivalent parallel training" model.

[0113] (7) After the layout agent training is completed, the layout state matrix and coordinate state matrix of the circuit to be laid out are input into the trained layout agent to obtain the optimal layout result of the circuit to be laid out. The optimal layout result is the placement position of each component of the circuit to be laid out, which is represented by the pin coordinates of each component.

[0114] It is understandable that during the training phase of the layout agent, the selection of actions is random, and during the application phase of the layout agent, that is, when the trained layout agent is used to obtain the optimal layout result of the circuit to be laid out, the selection of actions is based only on the quality of the actions.

[0115] The embodiment of the present invention provides a method for layout and wiring of a hierarchical power electronic circuit based on reinforcement learning, which adopts the layout method described in the above embodiment to obtain the optimal layout result of the circuit, based on the current optimal layout result, and takes the layout result as the initial input, and starts to run the wiring algorithm in accordance with the principle and process similar to the layout algorithm, and finally obtains the optimal layout and wiring result, including:

[0116] S1, build an intelligent agent based on Q-table; where the state is the l×n×m coordinate points of the virtual PCB board, and the action is to move from the current coordinate point to the "above", "below", "left", or "right" of the xth (x=1,2…l) layer; define the quality matrix Q` as the quality of each action of all coordinate points in each layer, and define the action reward value of all coordinate points in each layer as the reward matrix R`.

[0117] Specifically, the wiring problem is divided into a reinforcement learning optimization problem of finite states and actions. The state is the l×n×m coordinate points of the virtual PCB board, and the action is to move from the current coordinate point to the "above", "below", "left", or "right" of the xth (x=1,2…l) layer. For any state, any action corresponds to an action quality Q value, and such a (state, action)-Q key-value pair is defined as the quality matrix Q`; for any state, any action will obtain a reward R value for evaluating the current action, and such a (state, action)-R key-value pair is defined as the reward matrix R`.

[0118] S2, divide the components into N groups according to their pin potentials, and select the pins of the zth (z=1,2…M) group from the pins of the yth (y=1,2…N) group.y -1,M y The total number of pins in the yth group) and the z-1th group of routing are used as the end point, the z+1th pin is used as the starting point, and the zth group of starting and ending points of the yth group of pins are combined, and the mass matrix Q of the zth group of starting and ending points of the yth group of pins is initialized. y,z ` and the reward matrix R y,z `, where the initial Q y,z All elements in ` are 0. If the initial R y,z If the coordinate point corresponding to the element in ` is the end point, the value of the element is a positive value. If the coordinate point is a pin of a different group or is occupied by a routing line of a different group, the value of the element is a negative value. Otherwise, the value of the element is a smaller negative value.

[0119] Specifically, all elements in the initial quality matrix are assigned a value of 0. In the initial reward matrix, a large positive reward is assigned to the element corresponding to the coordinate of the end point, which physically means that the trace is connected to the end point. A large negative reward is assigned to the element of the coordinate occupied by pins or traces in different groups, which physically means that the trace overlaps with other potentials and short-circuits. A small negative reward is assigned to the elements of the remaining coordinates, which physically means that points will be deducted for each step taken. The longer the total path, that is, the longer the trace, the lower the total reward.

[0120] S3, based on the current coordinate point and the current mass matrix Q y,z `Select an action, based on the initial R y,z ` and reward rules to determine its reward value, and update Q according to the reward value and Q-learning Q value calculation formula y,z `; The reward rules are: the total routing length is as short as possible and the routing paths of different potential points cannot overlap.

[0121] Specifically, according to the current coordinate point and the current mass matrix Q y,z , based on the greedy mechanism to select the action, according to the current reward matrix R y,z `The reward value is determined by the reward rule, and the Q value is updated according to the reward value and the Q value calculation formula of Q-learning; the reward rule is: the total routing length is as short as possible, and the routing paths of different potential points cannot overlap;

[0122] It can be understood that during the iterative update process of S3, since the coordinates of the end point (i.e., the zth pin and z-1 group of routing) and the pins or routings in different groups remain unchanged, the value of the reward matrix remains unchanged and is always the initial reward matrix.

[0123] S4, repeat S3 until a preset reward value is obtained after taking the latest action, indicating that the zth and z+1th pins of the yth group are connected, that is, a round of environment interaction and training of the agent is completed; the preset reward value is the end point reward value;

[0124] S5, repeat S2-S4 until the training stop condition is met (for example, when the requirement for successful agent training is met or the preset maximum number of training times is reached, the training of the current agent is completed; the trained agent is used to obtain the optimal wiring result between the two pins in the group (i.e., the combination of the starting point and the end point of the zth group);

[0125] S6, repeat S2-S5 until all the pins of the yth group (y=1, 2...N, ie, all the pins) are connected, and finally the optimal wiring result of the circuit to be wired is obtained.

[0126] The routing algorithm follows similar principles and processes as the layout algorithm, with only differences in modeling and algorithm design for specific problems, specifically:

[0127] The wiring problem is converted into a maze problem with the pin to be connected as the starting point, other pins or traces with the same potential as the end point, and pins or traces with different potentials as obstacles. The optimization goal is to shorten the path length from the starting point to the end point.

[0128] The length and width of the virtual PCB board are still divided into m×n coordinate points, the number of layers is defined as l, and the optional actions are moving to the "above", "below", "left", and "right" of the i-th (j=1,2…l) layer, so the number of optional actions is 4×l. Based on the above definition, the sizes of the mass matrix Q' and the reward matrix R' are (l×m×n)×(4×l) and (l×m×n)×1, respectively. The (l×m×n) rows correspond to the (l×m×n) coordinate points of the virtual PCB board, and (4×l) correspond to the optional actions at each coordinate point. Each element in the mass matrix Q' is the quality of selecting a certain action at each grid point, and each element in the reward matrix R' is the reward obtained after the agent moves to the coordinate.

[0129] The routing action is the moving direction of the next coordinate point. The reward for the routing action is a smaller negative reward for each step, which means that the shorter the total path length, the higher the reward. If the route moves to coincide with a pin or trace at another potential, a large negative reward is given, indicating that the route hopes to avoid such a situation. If the route moves to a pin or trace at the same potential, a large positive reward is given, indicating that the end point has been found and the round of learning is over.

[0130] The basic process of the wiring method is: first, all pins are grouped according to their potentials, and two pins in the same group are selected as the starting point and the end point for independent wiring optimization. For each group of pins, a set of quality matrix Q' and reward matrix R' are defined respectively, where the initial value of Q' is 0, and it is updated in real time according to R' and algorithm update rules during the training process; the initial value of R' is obtained according to the current wiring state: if there are other pins or traces with different potentials occupying the coordinate point, the element value of the point is a large negative value; if there are pins or traces with the same potential occupying the coordinate point, the element value of the point is a large positive value; if the point is not occupied, the element value of the point is a small negative value; if a new trace is added during the wiring process, the reward matrix is ​​updated according to the same rules.

[0131] Since the complexity of input and output is not high, the reinforcement learning agent chooses to use a simpler and faster Q-table to implement it, that is, to build an agent based on the Q-table, which is updated in the form of the Bellman optimal equation, as shown in formula (16). Where α is the learning rate, γ is the discount factor, and max a Q(S', a) is the empirical Q value, that is, the Q value given based on the experience of taking actions in the S' state in the past to obtain rewards. The larger α is, the greater the learning rate is, and the less consideration is given to previous results when the Q table is updated. The larger γ is, the greater the consideration is given to experience.

[0132] Q(S,A)←(1-α)Q(S,A)+α[R(S,α)+γ·max a Q(S′,α)]

[0133] In addition to the modeling of the problem and the choice of algorithm, the solution process of the layout method is basically similar to the wiring method. The agent keeps moving and obtains the reward R t , and update the Q-table. If the maximum number of training times is reached, it means that the learning process is over, and the agent moves from the starting point to the end point completely according to the output of the Q-table. This path is the optimal route result explored and learned by the agent.

[0134] The method provided by the present invention is further described in detail below with a specific example.

[0135] In order to specifically introduce the hierarchical power electronics layout and wiring method based on reinforcement learning proposed in this paper, the typical boost-buck converter Cuk converter is used as an example for discussion. The Cuk converter is a single-tube non-isolated DC converter proposed by Slobodan Cuk of California Institute of Technology in 1980 to improve the Buck-Boost converter. Figure 1As shown in the figure, the Cuk converter can be regarded as a series connection of the Boost converter and the Buck converter, with the switch tube merged. The main components in its basic topology include coupling capacitor C1, switch tube Q1, Boost inductor L1, Buck inductor L2, freewheeling diode D1, and filter capacitor C2. The biggest feature of the Cuk converter is that the input power current and the output load current are continuous, and it has inductors in both the input and output sections, which can significantly reduce the ripple of the input and output currents, thus having the advantages of small output voltage pulsation and small impact on the input power supply.

[0136] like Figure 2 As shown in the figure, according to the Cuk circuit schematic, you can place the required components and connect them in Altium Designer to draw its schematic diagram. Note that in this figure, the input power supply U E The output load R is represented by terminals JP1 and JP2 respectively, and the remaining components correspond to each other. Two engineering network table files can be exported from the schematic file of Altium Designer, one is the component table (Components), which is used to store the information of each component in the schematic, including component number, component packaging form and component model. The other is the wiring network table (Nets), which is used to store all electrical connection networks in the schematic, including network name, node information in the network, network connection description, etc. Based on these two engineering network tables, all information about the circuit to be laid out required by the program of the present invention can be obtained.

[0137] The large language model driven reinforcement learning hierarchical power electronic circuit layout and wiring method proposed in the present invention is based on Python language and uses tkinter library to build a window GUI interface. Figure 3 As shown. The buttons in the GUI window can be clicked to enter the corresponding functions. Based on this operation interface, the following shows the detailed steps of automatic layout and routing of Cuk circuits.

[0138] Step (1): Click the "Browse" button on the operation interface, select the txt files of the two engineering network tables of the Cuk circuit Components and Nets exported from Altium Designer and the pre-defined component sequence, and import the Cuk circuit information into the program.

[0139] Step (2): Perform modal analysis on the Cuk circuit to obtain the power flow path.

[0140] (A) When the switch tube Q1 is turned on, the original freewheeling diode D1 is reverse biased and cut off, the input circulating current I1 makes the inductor L1 store energy, the coupling capacitor C1 discharges the current I2 to make L2 store energy and supply power I3 to the load. The sum of the input and output currents in the switch tube I Q =I1+I2, such as Figure 4 shown.

[0141] (B) When the switch tube Q1 is turned off, the original freewheeling diode D1 is forward biased and turned on. The energy storage current I1 of the input port JP1 (usually the power supply) and the inductor L1 charges the coupled inductor C1. At the same time, the energy release current I2 of the inductor L2 acts to maintain the uninterrupted load current. At this time, the current flowing through the freewheeling diode D1 is the sum of the input and output currents I Q =I1+I2, such as Figure 5 shown.

[0142] (C) According to the above analysis, following the direction of power flow, the sequence of component placement order can be defined as JP1→L1→C1→Q1→D1→L2→C2→JP2.

[0143] (D) LLM is used to replace the above manual analysis process. It only needs to input the two engineering network tables of Components and Nets of the Cuk circuit exported from Altium Designer, and provide relevant prompt words, such as the circuit input port is JP1, the output port is JP2, and which components are included in the commutation circuit, etc. LLM can output a series of optimal component sequences under its understanding. Then, relevant information is extracted from Components and Nets, and Library and Component are established. According to the information in the table, the layout state matrix and coordinate state matrix of the intelligent agent are constructed, and the reward calculation is completed.

[0144] Step (3): Input the component sequence output by LLM into the verification and post-processing program. The program will compare the engineering network table and the component sequence, automatically eliminate the obviously unreasonable sequence, and if there are sequence components missing, add the missing components to the sequence according to the random principle. All remaining sequences will be sent to the subsequent layout optimization process.

[0145] Step (4): Click the "Set Boundary" button on the operation interface, use the left mouse button to draw the desired boundary, and click "Set Boundary" again after drawing to input the boundary information into the system. Click the "Preprocessing" button to define the component forced lock flag and initial coordinates. Based on this, the initialization of the state matrix can be completed. For example:

[0146] (A) The layout characteristics of the virtual PCB are represented by the state matrix O t As shown in the following formula, since the virtual PCB is divided into a 20×20 grid, the initial size of O0 is also 20×20. The first row of the matrix has a value of 1, and the rest are 0, which means that during the initialization setting, the area in the first row is set as a layout-unallowed area, and components have not yet been placed in other areas.

[0147]

[0148] If the first step places the component JP1 at the coordinate position (18, 3), and the length and width of the component occupy 2 X coordinate points and 3 Y coordinate points respectively, then O1 is updated as follows, where element 2 means that the area is occupied by the new component after the current action is taken.

[0149]

[0150] (B) The characteristics of the circuit and the components themselves are given by the state matrix P t As shown in the following formula, since the Cuk circuit contains 8 components in total, P0 has 8 rows, and each row corresponds to the information of one component.

[0151] And since no components are placed in the initial stage, all element values ​​are 0.

[0152]

[0153] After the first step places the component JP1 at the coordinate position (3,18), P1 is updated to the following formula. The first two elements 3 and 18 represent that the component is placed at the position with coordinates 3 in the X direction and 18 in the Y direction, respectively. According to the current placement angle, the component occupies 2 grid points in the X direction and 3 grid points in the Y direction.

[0154]

[0155] Step (5): Click “AI Training” and the program starts to execute the layout algorithm and convert the state matrix O t and P t Input deep neural network, output quality vector Q t The deep neural network has three parts: the convolutional neural network that extracts PCB layout features, the graph convolutional neural network that extracts circuit diagram features, and the multi-layer perceptron that integrates the first two and outputs the final result. t The element value in represents the quality of the action of selecting the position of the component to be placed, and the number of elements is the number of actions. For example:

[0156] (A) In the convolutional feature extraction process, as shown in the following formula, the input matrix observation is extracted through two convolutional layers, and the output result is encoded into a vector with a dimension of 1, denoted as CNN. OUT. Define the function g to encode a multidimensional matrix into a one-dimensional vector. Define conv1 as the first convolution layer, its input channel is 1, its output channel is 4, and the activation function is tanh. Considering the minimum element size to be a unit length, the convolution step is set to 1. In order to make the output feature map size the same as the input matrix, the convolution kernel size is set to 2, and the padding size is set to 1. Define conv2 as the second convolution layer, its input channel is 4, its output channel is 4, the activation function is tanh, the convolution step is set to 1, the convolution kernel size is set to 2, and the padding size is set to 0.

[0157] CNN OUT =g(tanh(conv2(tanh(conv1(O t )))))

[0158] (B) In the process of extracting circuit diagram features through graph convolution, we first need to construct an adjacency matrix containing the connection information of the graph. First, we perform graph analysis on the Cuk converter. The Cuk converter consists of eight components, which can be regarded as having eight nodes. From this, we can obtain the adjacency matrix Adj representing the structure of the graph as shown below. Its eight rows and eight columns represent components C1, C2, L1, L2, JP1, JP2, Q1, and D1 respectively. If there is an electrical connection between two components, then the element in the adjacency matrix is ​​1, otherwise it is 0. Taking component C1 as an example (the first row or first column of the adjacency matrix Adj), Figure 2 It can be seen that C1 is electrically connected to L1, L2, Q1, and D1, so (0, 2), (0, 3), (0, 6), and (0, 7) of the adjacency matrix Adj are all 1, and because the sequence of elements represented by the eight rows and eight columns is the same, the adjacency matrix Adj is a natural symmetric matrix.

[0159]

[0160] In order to combine the connection information between components with the characteristics of the components without losing the characteristic information of the components themselves, it is necessary to add self-connection on the basis of the adjacency matrix Adj to obtain the matrix A as shown in the following formula. Because each component is connected to itself, it is only necessary to set all the main diagonals of the adjacency matrix Adj to 1 to obtain the matrix A containing self-connection information.

[0161]

[0162] Based on the matrix A containing self-connection information, the degree matrix D is defined as shown in the following formula. In graph theory, the "degree" of a node is equal to the number of nodes connected to it. The degree matrix containing self-connection information needs to take into account the connection with itself. Thus, a diagonal matrix D can be obtained in which only the main diagonal has values, and the rest of the positions are all 0.

[0163] D=Diag[5 6 4 5 6 6 7 7]

[0164] Consider the matrix A and the feature matrix P t The dimensions are different. In order to avoid the direct multiplication of the two to affect the distribution of the original features, resulting in unbalanced large and small values, thus inducing the gradient explosion problem in the network training process. Therefore, the formula is defined as Symmetric normalized Laplacian matrix The mathematical principle is to perform the formula on the matrix A The normalization process shown in Figure 2 is used to eliminate the adverse effects caused by different dimensions and obtain the final matrix containing the connection information of the graph.

[0165] Using a normalized adjacency matrix Multiply by the feature matrix P t After the weight gcn_weights of the graph convolutional layer is calculated, the output result can be encoded into a vector with a dimension of 1, denoted as GCN OUT , as shown below.

[0166]

[0167] (C) Vector CNN OUT With GCN OUT Perform the head-to-tail concatenation encoding and input the multi-layer perceptron operation to obtain the quality vector Q t .

[0168] (D) Since the reinforcement learning algorithm uses the DQN algorithm, the action space of the algorithm is discrete, so the component placement action should be defined in a discrete way. In this example, the optional actions are defined as placing the component "above", "below", "left", and "right" of the previous component at a rotation angle of 0°, 90°, 180°, or 270°. Therefore, the output layer of the deep neural network has 4×4 nodes, that is, the vector Q t The size is 1×16. The larger the element value, the better the quality of the selected action.

[0169] Step (6): The output of the deep neural network is the quality vector Q t , Q t Each element value q (ai) Indicates the quality of the action of selecting the component placement position. At this time, the ε-greedy algorithm is applied to select action a according to the highest element value with a certain probability ε. t , randomly select an action with probability 1-ε.

[0170] The relationship between the number of learning iterations m and the probability ε is as follows:

[0171]

[0172] Where ε0 is the initial probability, 0<=η<=1 is the variable that adjusts the rate of change. In this example, η=1, m is the number of reinforcement learning iterations, and m max is the upper limit of m.

[0173] Step (7): After getting the next component placement action, call the virtual PCB board update program, add the new component according to the action, and get the reward R t . For example, the evaluation indicators are as follows:

[0174] 1) The overlapping area of ​​components;

[0175] 2) The overlapping area between the regions enclosed by different potential points;

[0176] 3) Total trace length.

[0177] Considering that different issues have different degrees of consideration in circuit layout and the dimensions of mathematical calculations are different, the most serious component overlap index with different dimensions among the three evaluation criteria is given a weight of 15 to highlight its importance in learning, and the overlap area of ​​the potential enclosed area with the same dimension as the overlap area is given a weight of 10. If the weight is too small, it will be difficult to distinguish from the line length reward. If it is too large, there will be a problem of too large a gap in the reward changes between different operations, inducing gradient explosion or disappearance in the training process. The weight of the line length indicator is kept at 1.

[0178] Step (8): Run the hierarchical power electronic circuit layout and wiring system based on reinforcement learning proposed by the present invention. In step t, the state vector O of the current layout is t and P t , input deep neural network, deep neural network operation output Q t , so we choose action a t , and get the new state vector O t+1 and P t+1 , and get reward R according to the evaluation index t . Store the four sets of values ​​of step t (O t ,P t ), a t , R t , (O t+1 ,P t+1 ).

[0179] Step (9): After a certain number of rounds of layout process are completed, randomly select (O t ,P t ), a t , R t , (O t+1 ,Pt+1 ) sample data, a round of training and learning is performed through the reinforcement learning algorithm to adjust the internal parameters of the deep neural network Perform an update. The specific training and learning process is as follows:

[0180] Through a round of training and learning by reinforcement learning algorithm, the internal parameters of the deep neural network are Perform an update. The specific training and learning process is as follows:

[0181] (A) According to the following formula, the R of the sample data is t and (O t+1 ,P t+1 ) for processing

[0182]

[0183] R t represents the reward after each step, max(·) represents the selection of the maximum element; k (i.e. a t ) represents the quality vector Q of the deep neural network output t The kth element in .

[0184] (B) Since the real q k Given by the deep neural network, it is necessary to adjust the weights and bias parameters of the deep neural network Make q k Can approach The loss function Loss is defined as the above formula (11).

[0185] (C) Calculation parameters For the gradient of Loss, it can be adjusted by the following formula

[0186]

[0187] Among them, 0.001 is the learning rate of the deep neural network. After training, the loss becomes smaller, indicating that the deep neural network has updated its design strategy.

[0188] The process corresponding to the above steps (5) to (9) is as follows: Figure 6 As shown. The overall summary can be as follows: (1) the state matrix is ​​input into the deep neural network; (2) according to the output quality vector Q t , the ε-greedy algorithm takes action a t ; (3) After updating the layout strategy, the action can be evaluated and a reward R is given t ; (4) Update q according to formula (27) * ; (5) Fit q ​​through deep neural network *, so the loss function equation (11) is established, and the NN parameters are adjusted according to equation (28).

[0189] Step (10): When all eight components are laid out, one round of operation ends and the next round should be restarted from the initial state (O0, P0).

[0190] Step (11): When the number of training times reaches the maximum value m max , or the maximum training time is reached, the training is considered complete. For example, the average time required to obtain the optimal layout result of the Cuk converter using the parallel training method proposed in the present invention is only 526 seconds. The design results are as follows Figure 7 shown.

[0191] Step (12): Click "Route" and the program will execute the routing algorithm. The basic principle and implementation steps of the routing algorithm are similar to those of the layout algorithm. The only difference is the specific problem modeling and the use of the relatively simpler and faster Q-table in the reinforcement learning method. Therefore, I will not repeat it here. Based on the optimal layout result, the routing result is as follows Figure 8 shown.

[0192] An embodiment of the present invention provides an electronic device, including: a computer-readable storage medium and a processor;

[0193] The computer-readable storage medium is used to store executable instructions;

[0194] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method described in any of the above embodiments.

[0195] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method described in any of the above embodiments.

[0196] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hierarchical power electronic circuit layout method based on reinforcement learning, characterized in that: include: S1, building a deep neural network as an agent; The input of the agent is the layout state matrix O of each grid point after the PCB board is gridded and the coordinate state matrix P of each component of the circuit to be laid out. The layout state of each grid point includes no component and within the layout range, outside the layout range, occupied by components placed by past actions, and occupied by components placed by current actions. The coordinate state of each component includes the X and Y coordinates of each component, and the number of grid points in the X and Y directions occupied by the component when the coordinates are used as reference points. The output is the quality vector Q of all possible layout actions of the circuit. The layout action is the direction of the next component placement, and the quality is the possibility of obtaining the optimal layout result when selecting each layout action. The deep neural network includes CNN, GCN and MLP. S2, the layout state matrix O of step t t and the coordinate state matrix P t Input to the agent, CNN from O t Extract the layout features of the PCB plane from P t The circuit diagram features are extracted from the MLP, and the Q t ; Select an action a from t , get the layout state matrix O of step t+1 t+1 and the coordinate state matrix P t+1 , and evaluate it through the reward rule to get the reward R t ; O t , P t 、a t , R t , O t+1 , P t+1 As a set of data, it is stored in the memory pool; wherein, t = 0, 1, 2, ... K, K is the number of components of the circuit to be laid out; the reward rule is: the placement positions of the components have no conflicts and the total routing length is as short as possible; S3, repeating S2 until all components are placed in a preset order; randomly extracting a batch of data from the memory pool to train the agent; wherein the preset order is consistent with the power flow direction of the circuit; S4, repeating S2-S3 until the training stop condition is met, and using the trained intelligent agent to obtain the optimal layout result of the circuit to be laid out.

2. The method according to claim 1, characterized in that The preset order of all components is obtained by inputting the information of each component of the circuit and the connection relationship of each component into the LLM.

3. The method according to claim 2, characterized in that The LLM also includes a verification and post-processing module for checking whether the number of components in the generated component sequence is correct. If not, the missing components are inserted into the component sequence.

4. The method according to claim 1, characterized in that Based on the reward rule, set the reward function r = r overlap +r overlap_net_area +r hpwl ; Among them, r overlap For component overlap area bonus, len i (c_list) represents the number of components on each grid point of the i-th PCB board, a>1, n×m is the total number of grid points on the PCB board; r overlap_net_area The reward is the overlapping area between any two groups of pins with different potentials after they are connected. N is the number of categories of pin potentials in the circuit to be laid out, net i 、net j are the i-th and j-th groups of pins, respectively, where i>j; r hpwl is the total trace length bonus, Δx i , Δy i They are the maximum spacing between the pins in the pin group of the i-th potential in the horizontal and vertical directions respectively.

5. The method according to claim 1, characterized in that When training the agent, the weight parameters of the deep neural network are updated in the GPU, and the memory pool is updated in the CPU.

6. The method according to claim 5, characterized in that When laying out multiple circuits to be laid out, deep neural networks of multiple agents are trained simultaneously in the GPU; the multiple agents correspond one-to-one to the multiple circuits to be laid out, and are respectively used to obtain the optimal layout results of the corresponding circuits to be laid out.

7. A hierarchical power electronic circuit layout and wiring method based on reinforcement learning, characterized in that: The method according to any one of claims 1 to 6 is used to obtain the optimal layout result of the circuit to be laid out, that is, the pin positions of each component of the circuit to be laid out, so as to perform wiring, and the wiring method comprises: S1, build an intelligent agent based on Q-table; where the state is the l×n×m coordinate points of the virtual PCB board, and the action is to move from the current coordinate point to the "above", "below", "left", and "right" of the xth (x=1,2…l)th layer; define the quality matrix Q` as the quality of each action of all coordinate points in each layer, and define the action reward value of all coordinate points in each layer as the reward matrix R`; S2, divide the components into N groups according to the potential of each pin of each component. In the yth group of pins, take the zth pin and the z-1th group of routing as the end point, and the z+1th pin as the starting point to form the zth group of starting point and end point combination of the yth group of pins, and initialize its quality matrix Q y,z ` and the reward matrix R y,z `; where y = 1,2…N, z = 1,2…M y -1,M y is the total number of pins in the yth group, and the z-1th group of routing is the connection between the zth pin and the z-1th pin; the initial Q y,z All elements in ` are 0; if the initial R y,z The coordinate point corresponding to the element in is the end point, the value of the element is positive, if the coordinate point is a pin of a different group or is occupied by a routing line of a different group, the value of the element is negative, otherwise the value of the element is a smaller negative value; S3, based on the current coordinate point and the current mass matrix Q y,z `Select an action, based on the initial R y,z ` and reward rules to determine its reward value, and update Q according to the reward value and Q-learning Q value calculation formula y,z `; The reward rules are: the total routing length is as short as possible, and the routing paths of different potential points cannot overlap; S4, repeating S3 until a preset reward value is obtained after taking the latest action; the preset reward value is the end point reward value; S5, repeat S2-S4 until the training stop condition is met; use the trained intelligent agent to obtain the optimal wiring result of the zth group of starting point and end point combinations; S6, repeat S2-S5 until all pins are connected, and finally obtain the optimal wiring result of the circuit to be wired.

8. An electronic device, characterized in that: include: A computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 7.

Images