Multi-agent joint layout model training method and device, equipment and medium

Abstract

The invention discloses a multi-agent joint layout model training method and device, equipment and a medium, and relates to the field of printed circuit board design. The method comprises the steps that parameter information related to components to be arranged and grouping information generated based on the types of the components and aiming at the components to be arranged are acquired, and a corresponding intelligent agent is configured for each group of components; according to the parameter information and by adopting a distributed mode, carrying out layout on respective corresponding components through each intelligent agent; obtaining decision data generated by the policy network in the layout process of each agent, and evaluating the decision data by using the value network to obtain learning experience data for each component; and according to the learning experience data, determining update parameters of the strategy network and the value network of each agent by using the objective function, updating the corresponding agent by using the update parameters, and obtaining a multi-agent joint layout model based on all the updated agents. And the layout capability of the printed circuit board is improved.

Description

Technical Field

[0001] The present invention relates to the field of printed circuit board design, and in particular to a multi-agent joint layout model training method, device, equipment and storage medium. Background Art

[0002] Printed circuit boards, also known as PCBs, are important electronic components that are widely used in the production and manufacturing of various electronic devices. PCB layout is an important process in the entire layout design process, which directly affects the wiring design, which is the most labor-intensive part in the later stage, and the final PCB board performance. The industry usually uses electronic design automation (EDA) software to complete the entire process from schematic drawing to PCB layout design. In related technologies, heuristic algorithms are widely used in PCB automatic layout design. Under this algorithm, the combination of the arrangement of all electronic components on the PCB board is defined as the solution space, and the optimal solution is obtained through rules or searches. Although this method can improve computing power by distributed search, the heuristic algorithm has a slow convergence speed and is difficult to parallelize, and is prone to fall into local optimality, and cannot be applied to the layout design of large and complex PCB boards. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a multi-agent joint layout model training method, device, equipment and medium, which can improve the printed circuit board layout capability and support large-scale and complex printed circuit board layout tasks. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a multi-agent joint layout model training method, comprising:

[0005] Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent includes a corresponding strategy network and value network;

[0006] According to the parameter information and in a distributed manner, each of the intelligent agents performs layout for the components corresponding to each other, until all components are laid out to obtain a complete layout of the printed circuit board;

[0007] Obtaining decision data generated by the strategy network during the layout process of each agent, and evaluating the decision data using the value network to obtain learning experience data for each component;

[0008] According to the learning experience data, the update parameters of the strategy network and the value network of each agent are determined using the objective function, the corresponding agent is updated using the update parameters, and a multi-agent joint layout model is obtained based on all the updated agents.

[0009] Optionally, the performing layout for the components corresponding to each of the intelligent agents in a distributed manner according to the parameter information includes:

[0010] By each of the intelligent agents performing layout sorting on the components in the corresponding component group according to the target sorting rule;

[0011] According to the parameter information and the layout sorting, each of the intelligent agents performs layout for the components corresponding to each other in a distributed manner.

[0012] Optionally, the step of performing layout for the components corresponding to the respective components by each of the intelligent agents in a distributed manner according to the parameter information and the layout sorting includes:

[0013] Obtaining the state space corresponding to each component by splicing the parameter information and the occupancy of the printed circuit board;

[0014] According to the order of the layout sorting, the state space corresponding to each component is input into the corresponding strategy network in turn, and the position coordinates of the component are determined according to the output layout action; the strategy network is trained based on the deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output.

[0015] Optionally, determining the position coordinates of components according to the outputted layout action includes:

[0016] Select the layout action with the highest probability from all the output layout actions corresponding to the current state space;

[0017] The horizontal coordinate and the vertical coordinate corresponding to the component are determined according to the layout action with the highest probability.

[0018] Optionally, after performing layout for the components corresponding to the respective intelligent agents in a distributed manner according to the parameter information, the method further includes:

[0019] Determine whether the current layout of components conforms to the pre-configured component layout rules;

[0020] If it meets the requirements, the layout of the next component will be executed. If it does not meet the requirements, the layout of the current component will be re-executed.

[0021] Optionally, the obtaining of parameter information related to the components to be laid out includes:

[0022] Obtaining parameter information of components to be laid out according to circuit-related files; the circuit-related files include one or more of circuit schematics, circuit board files, and netlist files; the parameter information includes component parameters and connection information between components;

[0023] The parameter information is preprocessed to obtain parameter information that supports reading by the intelligent agent.

[0024] Optionally, the preprocessing the parameter information includes:

[0025] The parameter information is input into a graph convolutional neural network to generate a graph embedding including adjacency matrix features, component width, height and type features, and component connection edge features.

[0026] Optionally, before obtaining parameter information related to the components to be laid out, the method further includes:

[0027] The effective area on the printed circuit board is evenly divided into square grids, and the coordinate position of each grid point is defined sequentially, and a corresponding occupancy status mark is added to each grid point.

[0028] Optionally, the step of performing layout for the components corresponding to the components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board, further includes:

[0029] Determining a reward value using a target reward mechanism according to the complete layout of the printed circuit board, and optimizing the agent according to the reward value;

[0030] The target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

[0031] Optionally, determining the reward value by using a target reward mechanism according to the complete layout of the printed circuit board includes:

[0032] If the target reward mechanism is a reward method based on rule constraints, the corresponding reward value is determined by estimating the wiring length and wiring congestion according to the complete layout of the printed circuit board.

[0033] Optionally, determining the reward value by using a target reward mechanism according to the complete layout of the printed circuit board includes:

[0034] If the target reward mechanism is a reward method based on human expert evaluation, the complete layout of the printed circuit board and the circuit-related files corresponding to the complete layout of the printed circuit board are input into a reward function prediction model based on human electronic engineer scoring to obtain a reward value output by the reward function prediction model;

[0035] The reward function prediction model is obtained after training using sample pairs, wherein the sample pairs include circuit-related files, printed circuit boards corresponding to the circuit-related files generated using a deep reinforcement learning model, and reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to a preset reward value scoring standard.

[0036] Optionally, obtaining the decision data generated by the policy network during the layout of each agent includes:

[0037] Generate corresponding decision data according to the current decision network of each agent; the decision data includes the state space of the components corresponding to the agent, layout actions, state space transfer after layout, and the reward value corresponding to the agent.

[0038] Optionally, the using a value network to evaluate the decision data to obtain learning experience data for each component includes:

[0039] The global action value function is calculated using the value network based on the decision data; the value network is constructed based on the action value, with the local state space and layout action as input and the action value as output;

[0040] Based on the decision data, a global target action value function is calculated using a target value network; the network structure of the target value network is the same as that of the value network, and the model parameter update frequency is lower than that of the value network;

[0041] Determine a reference value corresponding to each intelligent agent, and obtain an advantage function corresponding to the intelligent agent according to the difference between the global target action value and the reference value;

[0042] Based on the decision data, the global action value function, the global target action value function and the advantage function, the learning experience data corresponding to the component is obtained.

[0043] Optionally, determining the reference value corresponding to each agent includes:

[0044] Add the joint actions of the remaining agents to the action value corresponding to the current agent, and use the value network to obtain the target global action value;

[0045] Based on the target global action value and the policy network, a benchmark value corresponding to the agent is calculated.

[0046] Optionally, the calculation formula of the benchmark value is:

[0047]

[0048] in, is the joint layout action of the agents except agent i, t is the tth component, is the target global action value, a it is the layout action of agent i, S t is the global status information, is the policy network of agent i, O it is the local environment state space of agent i.

[0049] Optionally, calculating a global action value function based on the decision data using the value network includes:

[0050] The value network is used to calculate the local action value corresponding to each agent respectively, and the global action value function is approximated according to all action values;

[0051] The calculating of the global target action value function based on the decision data using the target value network includes:

[0052] The target value network is used to calculate the local target action value corresponding to each agent, and the global target action value function is approximated based on all target action values.

[0053] Optionally, determining update parameters of a strategy network and a value network for each of the agents using an objective function according to the learning experience data includes:

[0054] Determine a first update parameter for each of the strategy networks according to the learning experience data and the optimization objective function;

[0055] A second update parameter for each of the value networks is determined according to the learning experience data and the loss function.

[0056] Optionally, the objective optimization function is constructed based on the action advantage function; and the loss function is constructed based on the action value function and the target action value function.

[0057] Optionally, the updating of the corresponding agent using the update parameter includes:

[0058] The network parameters of the policy network are updated using the first update parameters, and the network parameters of the value network are updated using the second update parameters.

[0059] In a second aspect, the present application discloses a printed circuit board automatic layout method, which is applied to the above-mentioned multi-agent joint layout model, comprising:

[0060] Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and assign agents in the multi-agent joint layout model to each group of components;

[0061] The parameter information associated with each group of components is input into the corresponding agent so that the multi-agent joint layout model can perform automatic layout of the printed circuit board.

[0062] In a third aspect, the present application discloses a multi-agent joint layout model training device, comprising:

[0063] A grouping module, used to obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent contains its own corresponding strategy network and value network;

[0064] A layout module, used to layout the corresponding components through each of the intelligent agents in a distributed manner according to the parameter information, until all components are laid out to obtain a complete layout of the printed circuit board;

[0065] An evaluation module, used to obtain the decision data generated by the strategy network during the layout process of each intelligent agent, and evaluate the decision data using the value network to obtain learning experience data for each component;

[0066] A network update module is used for a module to determine the update parameters of the strategy network and value network for each of the agents according to the learning experience data using the objective function, update the corresponding agents using the update parameters, and obtain a multi-agent joint layout model based on all the updated agents.

[0067] In a fourth aspect, the present application discloses an electronic device, including:

[0068] Memory, used to store computer programs;

[0069] A processor is used to execute the computer program to implement the aforementioned multi-agent joint layout model training method.

[0070] In a fifth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned multi-agent joint layout model training method.

[0071] In the present application, parameter information related to components to be laid out and grouping information for the components to be laid out generated based on component types are obtained, and corresponding intelligent agents are configured for each group of components; each intelligent agent contains its own corresponding strategy network and value network; layout is performed for the corresponding components by each intelligent agent in a distributed manner according to the parameter information until the layout of all components is completed to obtain a complete layout of the printed circuit board; decision data generated by the strategy network during the layout process of each intelligent agent is obtained, and the decision data is evaluated using the value network to obtain learning experience data for each component; based on the learning experience data, the update parameters of the strategy network and the value network for each intelligent agent are determined using the objective function, the corresponding intelligent agent is updated using the updated parameters, and a multi-agent joint layout model is obtained based on all updated intelligent agents.

[0072] It can be seen that by distributing different intelligent agents to execute layout operations on different components, the layout of printed circuit boards can be realized by multi-intelligence joint efforts, the influence between multiple intelligent agents can be coordinated, the scale of printed circuit board layout can be expanded, and the intelligent agents can be updated through the evaluation of the value network, so as to improve the layout ability of the intelligent agents through reinforcement learning, support large-scale and complex printed circuit board layout tasks, and improve the printed circuit board layout capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0074] Figure 1 A flow chart of a multi-agent joint layout model training method provided in this application;

[0075] Figure 2 A specific value network evaluation method flow chart provided for this application;

[0076] Figure 3 A specific multi-agent joint layout model training method flow chart provided in this application;

[0077] Figure 4 A schematic diagram of the structure of a multi-agent joint layout model training device provided in this application;

[0078] Figure 5 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] In the related art, heuristic algorithms are widely used in PCB automatic layout design. Under this algorithm, the combination of the arrangement of all electronic components on the PCB board is defined as the solution space, and the optimal solution is obtained through rules or searches. Although this method can improve computing power by distributed search, the convergence speed of the heuristic algorithm is slow and difficult to parallelize, and it is easy to fall into local optimality and cannot be applied to the layout design of large and complex PCB boards. In order to overcome the above technical problems, this application proposes a multi-agent joint layout model training method, which can improve the layout capability of printed circuit boards and support large-scale and complex printed circuit board layout tasks.

[0081] The present application embodiment discloses a multi-agent joint layout model training method, see Figure 1 As shown, the method may include the following steps:

[0082] Step S11: Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent contains its own corresponding strategy network and value network.

[0083] In this embodiment, parameter information related to the components to be laid out is obtained, such as component parameters and connection information between components. Wherein, the obtaining of parameter information related to the components to be laid out may include: obtaining parameter information of the components to be laid out according to circuit-related files; the circuit-related files include one or more of circuit schematics, circuit board files and netlist files; the parameter information includes component parameters and connection information between components; and preprocessing the parameter information to obtain parameter information that supports intelligent reading. That is, according to the circuit schematics, PCB circuit board files or netlists generated in the previous stage of the PCB design process, the number and type of components to be laid out on the PCB board, the parameters of each component, such as component size, number of pins, and distance to the ground or other pins, as well as the connection type and resistance value between components are extracted. After obtaining the parameter information, preprocessing is performed to obtain data that supports intelligent reading. The understandable parameter information is usually heterogeneous and can be preprocessed so that the intelligent agent can easily obtain these data and structures.

[0084] In this embodiment, the preprocessing of the parameter information may include: inputting the parameter information into a graph convolutional neural network to generate a graph embedding containing adjacency matrix features, component width and height and type features, and component connection edge features. That is, taking the input circuit-related file as a netlist file as an example, the parameter information is input into a graph convolutional neural network (GCN, Graph Convolutional Network), which can generate a graph embedding (GraphEmbedding) containing information such as netlist graph adjacency matrix features, component width and height and type features, and component connection edge features. Alternatively, it is also possible to obtain parameter information supporting the intelligent agent by retrieval by constructing a data structure such as a database or a knowledge graph.

[0085] In this embodiment, before obtaining the parameter information related to the components to be laid out, the following may also be included: evenly dividing the effective area on the printed circuit board into square grids, sequentially defining the coordinate position of each grid point, and adding a corresponding occupation status mark to each grid point. That is, before building the model, the PCB board coordinate division and grid point status definition are performed, including clarifying the actual size and shape of the PCB board, determining the area where components can be placed and the prohibited area, and evenly dividing the available area into square grids with intervals, sequentially defining the coordinate position x of each grid point i ,y i , that is, finally, the grid points are used as the center points of the components for actual layout. The thickness of the grid can be determined according to the complexity of the layout problem and the layout training time or computing resource scale that can be tolerated. In addition, it is necessary to define the occupation status mark, for example, the status of the occupied grid points is marked as 0, that is, the grid point is unavailable, and the status of the unoccupied grid points is marked as 1, that is, the grid point is available. In addition, it should be noted that the grid points covered by the area of ​​the components should all be in an unavailable state.

[0086] In this way, in each decision step of the agent, the agent will identify which grid points on the PCB have already been placed with components, thereby adaptively identifying the layout positions. The state of the grid points on the PCB board, that is, the occupancy of the printed circuit board, can be determined by extracting the image features of the components that have been laid out on the PCB board using a convolutional neural network, or by directly marking and stacking the occupancy state of the grid points in a vectorized manner for recording.

[0087] In this embodiment, after obtaining the parameter information of the components to be laid out, the components to be laid out are grouped according to the device type, that is, different types of electronic components are grouped, such as resistors, capacitors, transistors, sensors, etc. Then, corresponding agents are configured for each group of components, that is, the designated components are laid out and designed by different agents respectively, wherein a corresponding strategy network and value network are constructed for each agent, wherein the strategy network is mainly used for decision-making layout, and the value network is mainly used for calculating the value of the decision action, so that the value updates the model parameters for reinforcement learning.

[0088] In this embodiment, each intelligent agent corresponds to a unique strategy network, that is, the input of the strategy network is the state space obtained by splicing the parameter information of the components corresponding to the intelligent agent and the current occupancy of the printed circuit board, that is, the local state space. The local state spaces corresponding to all intelligent agents are combined to form the global state space. The output of the strategy network is the layout action obtained based on the local state space observation; the value network takes the local state space and the corresponding layout action as input, and outputs the action value.

[0089] Specifically, the construction of the strategy network and value network can use the MAPPO (Multi-agent PPO, multi-agent proximal strategy optimization) algorithm framework. According to the types of electronic components that need to be laid out on the PCB board, set the corresponding number of agents and establish a strategy network for each agent. and value network Among them, i is the local environment state space of each agent, and the global environment state information of the entire PCB layout design system is S = {o1, o2, ..., o N}; a i is the layout action of each agent based on its own local observation, and the joint action of the system is A = {a1, a2, ..., a N}, i represents the i-th agent. Where N is the number of electronic component groups (i.e., the number of agents) contained in the system, θ i ,ω i are the policy network and value network parameters. Considering the lack of immediate action reward guidance in the PCB layout decision process, this solution does not use the state value V(s) of the classic PPO algorithm, but uses the action value Q(s,a). After completing the network settings, initialize all network parameters θ i ,ω i , i∈[1,N].

[0090] Step S12: Layout the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board.

[0091] In this embodiment, according to the parameter information and in a distributed manner, each agent performs layout for the components corresponding to each other, that is, utilizing the distributed execution characteristics between multiple agents, distributed parallel execution layout, improving layout efficiency, and different agents can coordinate layout positions through communication until the layout of the components to be laid out is completed to obtain a complete layout of the printed circuit board. It can be understood that distribution refers to dividing a huge computing problem into many small parts, and then assigning these parts to multiple agents for processing, each agent is responsible for different computing tasks, and different agents cooperate with each other, that is, a system composed of multiple agents collaboratively controls the layout of the circuit board, and finally combines the calculation results of each agent to obtain the final result. That is, each agent completes the layout of the electronic components in the group in a distributed manner, and collaboratively completes the layout design of the electronic components on the entire PCB board.

[0092] In this embodiment, the layout of the components corresponding to each of the intelligent agents in a distributed manner according to the parameter information may include the following steps:

[0093] S121: Layout and sort components in the corresponding component group by each of the intelligent agents according to the target sorting rule.

[0094] The target sorting rule is a sorting rule based on component priority, wherein components related to power supply, components occupying a large area, and components with special interface requirements have a high priority. That is, in each group, components related to power supply, components occupying a large area, and components with special interface requirements are given priority, and the layout of each group of components is sorted separately.

[0095] S122: According to the parameter information and the layout sorting, each of the intelligent agents performs layout for the corresponding components in a distributed manner.

[0096] Among them, according to the parameter information and the layout sorting, each of the intelligent agents performs layout for the corresponding components in a distributed manner, including: obtaining the state space corresponding to each component by splicing the parameter information and the occupancy of the printed circuit board; inputting the state space corresponding to each component into the corresponding strategy network in turn according to the order of the layout sorting, and determining the position coordinates of the component according to the output layout action; the strategy network is obtained by training based on a deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output.

[0097] It can be understood that in each group, the component ID to be currently laid out, the PCB grid occupancy, the characteristics of the components, and the graph embedding of the connection relationship are simply spliced ​​or fully connected to form the state space of the intelligent agent. Subsequently, the layout strategy network with the component layout action as the output will be trained using the PPO algorithm for continuous action space. The layout action is the layout horizontal coordinate x and vertical coordinate y. Among them, considering that the components are placed on the grid points in the actual layout, if the horizontal and vertical coordinate values ​​output by the decision network are inside the grid, the grid point coordinate x is taken downward. i and i , as the layout action output of the current i-th component in this group.

[0098] In this embodiment, determining the position coordinates of components according to the output layout actions includes: selecting the layout action with the highest probability from all the output layout actions corresponding to the current state space; and determining the horizontal and vertical coordinates corresponding to the components according to the layout action with the highest probability.

[0099] That is, for each agent i, the t-step decision, that is, the decision for component t, is input into the strategy network according to the state space O composed of the component ID to be laid out in each group, the PCB grid occupancy, and the netlist embedding In the above example, we obtain the probability distribution of the horizontal coordinate x and vertical coordinate y of the components on the PCB. In order to simplify the decision-making process, we select the layout action a according to the greedy algorithm each time. it0 , that is, choose the layout action with the highest probability At the same time, take the nearest grid point coordinate x it and it , as the layout action output a of the tth component that each agent is responsible for it , and the joint action a t ={a 1t , a 2t , ..., a Nt}.

[0100] In this embodiment, after the layout of the corresponding components is performed by each of the intelligent agents in a distributed manner according to the parameter information, it may also include: judging whether the layout of the current components meets the pre-configured component layout rules; if it meets, the layout of the next component is executed, and if it does not meet, the layout of the current component is re-executed. It can be understood that there are some rules for component layout, including ensuring that the spacing, direction and connectivity of the components meet the actual needs, as well as the blocking avoidance and via design between components. Therefore, after making a decision, it is necessary to judge whether it is feasible in combination with the actual situation. If it is identified that the decision conflicts with the previously placed components, for example, the position relationship is overlapped according to the size of each component, then the layout of this component is abandoned. Next, you can directly enter the next layout decision for the current component, that is, re-execute the decision step; or take the grid point of the current decision as the origin, select the grid points within the surrounding target range, and perform component layout and validity judgment in sequence, for example, perform component layout and validity judgment on the surrounding [-k, k]*[-k, k] grid points in sequence. In order to ensure the consistency of the layout positions of each component, the main consideration is to complete the layout of all electronic components through deep reinforcement learning, that is, to directly proceed to the next decision after determining that the current decision is invalid.

[0101] By repeating the above layout decisions, each agent uses its own strategy network according to the local observations of each decision step. The decision is made sequentially, the layout of electronic components within the group is completed in a distributed manner, and finally the layout design of electronic components on the entire PCB board is completed collaboratively.

[0102] Step S13: Obtain the decision data generated by the strategy network during the layout process of each intelligent agent, and use the value network to evaluate the decision data to obtain learning experience data for each component.

[0103] In this embodiment, by obtaining the decision data generated by each decision of the strategy network during the agent layout process, the value of each decision action is evaluated using the value network to obtain learning experience data containing value evaluation.

[0104] In this embodiment, the acquisition of decision data generated by the strategy network during the layout process of each agent may include: generating corresponding decision data according to the current decision network of each agent; the decision data includes the state space of the components corresponding to the agent, the layout action, the state space transition after layout, and the reward value corresponding to the agent. In this embodiment, after each agent executes the layout action, the state space transition o′ is observed. it , the global state information is transferred to s t ={o′ 1t , o′ 2t , ..., o′Nt At the same time, after each decision, each agent generates a decision data according to the decision of the current strategy network, τ i ={o it , a it , o′ it , r t} t＝1～T , where T is the total number of steps in a complete interactive layout, that is, a decision data is recorded for each decision. In order to save storage space, the global state and action information is not directly stored in the process, that is, the decision data contains local state space and layout actions based on the agent.

[0105] Step S14: Determine the update parameters of the strategy network and value network for each of the agents using the objective function according to the learning experience data, update the corresponding agents using the update parameters, and obtain a multi-agent joint layout model based on all updated agents.

[0106] In this embodiment, the update parameters of the strategy network and value network for each agent are determined by using the objective function according to the learning experience data. The objective function can be a target optimization function or a loss function, that is, by analyzing the gap with the expectation based on the obtained learning experience data containing value evaluation, the agent is continuously fine-tuned, the layout ability of the agent is continuously improved, and a multi-agent joint layout model is obtained based on all updated agents. It can be understood that each time the strategy network corresponding to each agent executes a decision step, that is, a component is laid out once, a set of decision data is generated, and each set of decision data is evaluated once using the value network, or, after all decisions are completed, each set of decision data is evaluated in turn using the value network, wherein the instant rewards in the decision-making process are all 0. After each agent completes the layout decision for all components in the component group for which it is responsible, it continues to evaluate the decision value, generates and accumulates learning experience, and after the overall layout of the final printed circuit board is completed, a layout reward is calculated for the entire layout diagram. According to all evaluations, instant rewards and final layout rewards in the decision-making process, the agent is updated. The multi-agent joint layout model decomposes new PCB design parameters and tasks, and each agent layout strategy network distributes the layout of each electronic component group to achieve the overall joint layout of the PCB board.

[0107] In this embodiment, parameter information related to components to be laid out and grouping information for the components to be laid out generated based on component types are obtained, and corresponding intelligent agents are configured for each group of components; each intelligent agent contains a corresponding strategy network and value network; layout is performed for the corresponding components through each intelligent agent in a distributed manner according to the parameter information until the layout of all components is completed to obtain a complete layout of the printed circuit board; decision data generated by the strategy network during the layout process of each intelligent agent is obtained, and the decision data is evaluated using the value network to obtain learning experience data for each component; based on the learning experience data, the update parameters of the strategy network and value network for each intelligent agent are determined using the objective function, the corresponding intelligent agent is updated using the updated parameters, and a multi-agent joint layout model is obtained based on all updated intelligent agents.

[0108] It can be seen that by distributing different intelligent agents to execute layout operations on different components, the layout of printed circuit boards can be realized by multi-intelligence joint efforts, the influence between multiple intelligent agents can be coordinated, the scale of printed circuit board layout can be expanded, and the intelligent agents can be updated through the evaluation of the value network, so as to improve the layout ability of the intelligent agents through reinforcement learning, which can be applicable to large-scale and complex printed circuit board layout tasks and improve the printed circuit board layout capabilities.

[0109] Based on the above embodiments, see Figure 2 As shown, the method of evaluating the decision data using the value network to obtain learning experience data for each component may include the following specific steps:

[0110] Step S21: Calculate the global action value function using the value network based on the decision data; the value network is constructed based on action value, with local state space and layout action as input and action value as output.

[0111] In this embodiment, the value network in the agent is used to calculate the global action value function, that is, according to the decision data τ obtained for each agent i , using the value network Calculate the value of each decision action in the decision-making process of each agent.

[0112] In this embodiment, the calculation of the global action value function based on the decision data using the value network may include: using the value network to calculate the local action value corresponding to each agent respectively, and approximating the global action value function based on all action values. In this embodiment, considering the large state space in the PCB layout design problem, the designed strategy network and value network are both based on the local observation of the agent. i and a single decision action a i, so the output of the value network is the partial action value, and the global action value is obtained by approximation. The global action value is approximately:

[0113]

[0114] Step S22: Calculate the global target action value function using the target value network based on the decision data; the network structure of the target value network is the same as that of the value network, and the model parameter update frequency is lower than that of the value network.

[0115] In this embodiment, the global target action value function is calculated based on the decision data using the target value network. The network structure of the target value network is the same as that of the value network, and the model parameter update frequency is lower than that of the value network. It can be understood that in order to break the sample correlation between consecutive decisions and improve the stability of the model, a set of target value networks with the same structure as the value network but with a lower model parameter update frequency is set. Specially used to calculate the target action value.

[0116] The method of calculating the global target action value function based on the decision data using the target value network may include: calculating the local target action value corresponding to each agent using the target value network, and approximating the global target action value function based on all target action values. i and a single decision action a i , so the target value network output is the partial action value, and the global target action value is obtained by approximation. The global target action value is approximately:

[0117]

[0118] Step S23: Determine the benchmark value corresponding to each agent, and obtain the advantage function corresponding to the agent based on the difference between the global target action value and the benchmark value.

[0119] In this embodiment, the global state information and joint action information of the PCB layout design system are integrated to calculate the action advantage function of each agent in the multi-agent distributed decision-making process. The mutual influence in the multi-agent collaborative task is specifically taken into consideration, and a benchmark value is introduced in the calculation process of the action advantage function. The benchmark value reflects the influence of the action decision of other agents on the current agent in the multi-agent collaborative task. The value obtained by subtracting the benchmark value from the global target action value is used as the advantage function, which makes the relative contribution evaluation of each agent more accurate.

[0120] In this embodiment, the determination of the reference value may include: adding the joint action of the remaining agents to the action value corresponding to the current agent, and using the value network to obtain the target global action value; based on the target global action value and the strategy network, calculating the reference value corresponding to the agent. That is, the advantage function is:

[0121]

[0122] in, Represents the target action value of the system as a whole at the tth layout decision step, s′ t and a′ t is the global state space and joint layout action at the next moment, γ is the discount factor, which is usually taken as a constant between 0 and 1. is the benchmark value, i.e., the counterfactual benchmark. For each agent, the benchmark value in the above formula is replaced respectively to calculate their respective action advantage functions.

[0123] The calculation formula of the benchmark value is:

[0124]

[0125] in, is the joint layout action of the agents except agent i, t is the tth component, is the target global action value, a it is the layout action of agent i, S t is the global status information, is the policy network of agent i, O it is the local environment state space of agent i.

[0126] Step S24: Based on the decision data, the global action value function, the global target action value function and the advantage function, the learning experience data corresponding to the component is obtained.

[0127] In this embodiment, the learning experience data corresponding to the components is finally obtained based on the decision data, the global action value function, the global target action value function and the advantage function:

[0128]

[0129] The learning experience is stored in the playback buffer D, where each learning experience is marked as c it .

[0130] Accordingly, in this embodiment, the update parameters of the strategy network and the value network of each agent are determined by using the objective function according to the learning experience data. Specifically, the data in the playback buffer D can be randomly sorted and numbered, and grouped into M learning records. In actual training, M is often taken as 32 or 64 randomly selected and composed of B small batch sample sets:

[0131] D b ={{c ij} i＝1～N} j＝1～M , b=1~B.

[0132] Different mini-batch sample sets D b Substitute the objective functions separately to determine the update parameters of the policy network and value network for each agent.

[0133] Wherein, the method of determining the update parameters of the policy network and the value network of each of the intelligent agents using the objective function according to the learning experience data may include: determining the first update parameter for each of the policy networks according to the learning experience data and the optimization objective function; determining the second update parameter for each of the value networks according to the learning experience data and the loss function. That is, the policy network is fine-tuned using the optimization objective function, and the value network is fine-tuned using the loss function. In this embodiment, the method of updating the corresponding intelligent agent using the update parameters includes: updating the network parameters of the policy network using the first update parameters, and updating the network parameters of the value network using the second update parameters.

[0134] In this embodiment, the objective optimization function is constructed based on the action advantage function, and the strategy network optimization objective of agent i is as follows:

[0135]

[0136] Among them, θ ib represents the policy network parameters of agent i after b iterations, θ i Indicates the network parameters currently to be updated. ib (s j , a j ), the target value network of each agent realizes the collaboration among agents centrally and indirectly through the approximate summation operation. clip(·) represents the clipping operation to prevent θ i With θ ib The difference is too large, so the restriction parameter is taken as ε∈(0,1), which is specifically expressed as:

[0137]

[0138] By maximizing the objective function JCLIP (θ i ), the policy network parameters θ of each agent i to update.

[0139] In this embodiment, the loss function is constructed based on the action value function and the target action value function. Each agent explicitly shares the same loss function L, centrally and indirectly realizing the collaboration among agents. The overall value function Q(s) of the PCB layout design system at time t is obtained by calculation. j , a j ) and the target value function Then the value loss function L of all M learning experiences in each small batch sample set is:

[0140]

[0141] By minimizing the loss function L, the value network parameter ω of each agent i The target value network has the same structure as the value network, but the model parameter update frequency is lower. Then, at intervals of a certain number of steps, such as 1000 training steps, the parameters of the online learning evaluation network are assigned to the target evaluation network to achieve the target network parameter ω′ i Updates.

[0142] It can be seen that the value function of each agent is used to approximate the global value function, and each agent shares the same loss function. At the same time, the reference value is introduced to use the mutual influence between agents to calculate the advantage function, and the training and update of the value network and the policy network are realized in a centralized manner. When all B small batch sample sets are iteratively calculated, the training process for a PCB layout design cycle ends and enters the layout interaction stage until the training converges or reaches the set maximum number of training steps. At this time, the training of the target policy model ends, and the policy parameters θ of each agent are retained. i Therefore, by implicitly integrating the global observation information of each agent and jointly deploying actions, the policy network and value network model parameter updates of each agent are completed centrally.

[0143] Therefore, the sequential placement of electronic components is modeled as a reinforcement learning sequential decision problem, and electronic components with different characteristics are grouped. The global automatic layout of the printed circuit board is completed by implicit centralized training combined with explicit distributed execution, and the influence between multiple agents is coordinated to expand the layout scale of the printed circuit board. Through explicit distributed execution, multiple agents are supported to layout components with different performances respectively, which improves execution efficiency and expands the scale of the layout of printed circuit boards. Combined with implicit centralized training, the overall value function and shared loss function of the PCB layout are approximately calculated, and the benchmark value is introduced to reflect the influence between agents, so as to realize indirect and implicit collaboration of multiple agents.

[0144] The present application embodiment discloses a specific multi-agent joint layout model training method, see Figure 3 As shown, the method may include the following steps:

[0145] Step S31: obtaining parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and configuring corresponding intelligent agents for each group of components; each intelligent agent includes its own corresponding strategy network and value network.

[0146] Step S32: Layout the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board.

[0147] Step S33: Determine a reward value using a target reward mechanism according to the complete layout of the printed circuit board, and optimize the agent according to the reward value.

[0148] In this embodiment, after each agent completes the layout of each group of components and the overall layout of the PCB, feedback learning is performed through a preset reward mechanism, thereby optimizing the agent. Each agent i performs the tth electronic component layout action a in the group it , only when the layout is completed, the overall calculation gets a reward value r T , the immediate rewards in the process are all 0, that is, the reward value is given based on the complete layout result, so the reward value is calculated after all layouts are completed.

[0149] Among them, the target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

[0150] In the first case, that is, if the target reward mechanism is a reward method based on rule constraints, the corresponding reward value is determined by estimating the wiring length and wiring congestion according to the complete layout of the printed circuit board. That is, for the complete PCB layout diagram, the wiring length and wiring congestion are estimated, and a reward R_T is given. The action reward in the process is 0. In the field of EDA, it is required to ensure that the wire length is as short and compact as possible, with small circuit delay and electromagnetic interference. At the same time, the wiring congestion ratio is limited to avoid too dense wiring in certain areas, which will have an adverse effect on the performance and work efficiency of the entire circuit. The reward value calculation process is as follows:

[0151] HPWL(i)=(max b∈i {x b}-min b∈i {x b}+1)+(max b∈i {y b}-min b∈i {y b}+1)

[0152]

[0153]

[0154] R p，g =-HPWL(netlist)-λ*Congestion(p,g)

[0155] Among them, HPWL(i) is the semi-circumference line length of grid point i, which is used to approximate the semi-circumference of the winding of each electronic component. HPWL(nestlist) represents the winding wire length of all components on the PCB. q(i) is a regularization factor used to improve the accuracy of line length estimation. In addition, Congestion refers to wiring congestion, A refers to the total area of ​​the PCB wiring area, and W max Refers to the widest line width on the PCB board. p，g It means the cumulative reward that can be obtained by using layout strategy p for layout of netlist g. That is, the winding half-circumference of each electronic component is calculated, the winding wire length of all components is determined according to the winding half-circumference of each electronic component, the wiring congestion is determined according to the winding wire length of the component, and finally the reward value is determined based on the wiring congestion and the winding wire length of the component.

[0156] In the second case, that is, if the target reward mechanism is a reward method based on human expert evaluation, the complete layout of the printed circuit board and the circuit-related files corresponding to the complete layout of the printed circuit board are input into the reward function prediction model based on the scoring of human electronic engineers to obtain the reward value output by the reward function prediction model; the reward function prediction model is obtained after training with sample pairs, and the sample pairs include circuit-related files, and the printed circuit boards corresponding to the circuit-related files generated by the deep reinforcement learning model, and the reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to the preset reward value scoring criteria. At this time, the interactive training mode mainly for deep reinforcement learning requires many iterative steps and slow model convergence speed. The training of the model is accelerated by introducing the design experience of human electronic engineers. For example, select multiple PCB netlist files used in actual projects, use different pre-trained deep reinforcement learning models to generate corresponding PCB layout diagrams for each file, and experienced electronic engineers score the quality of the PCB layout diagrams generated by different networks. Considering that different engineers have different preferences for PCB design styles, the scoring interval for each PCB layout is set to [1, 10], the increment is set to 1, and the quality scores of PCB layout designs of different engineers for the same netlist and the same model are averaged, which is the reward value of PCB design. Thus, a large number of sample pairs are accumulated to form a data set, and each sample pair is [x = [netlist, PCB layout], y = reward value]. The reward function prediction model is trained by supervised learning. During training, the model is continuously updated based on the scoring results of human feedback, so that the model is gradually optimized to be able to better utilize human expert experience to make decisions.

[0157] In the third case, that is, if the target reward mechanism is a reward method based on rule constraints and a reward method based on human expert evaluation, then in the early stage of agent training, the reward function prediction model based on the score of human electronic engineers is used to determine the reward value, and in the later stage of agent training, the reward value is determined by estimating the wiring length and wiring congestion according to the complete layout of the printed circuit board. It can be understood that in the interactive PCB joint layout design problem based on multi-agent reinforcement learning, the above evaluation method combined with human scoring is helpful for adjusting the model optimization direction and accelerating the training in the early stage of layout agent training, but in the later stage of design, in order to encourage each agent to break away from the knowledge limitations of human electronic engineers and explore the layout methods of different types of electronic components, it is necessary to adjust the reward back to the form based on wiring length and wiring congestion constraints, and guide the parameter update of the decision-making agent with a universal optimization goal, so as to break through the limitations of human knowledge. Therefore, by combining different reward methods and adopting different reward methods for different stages, the accuracy of the reward is further improved, thereby accelerating the promotion of the layout ability of the agent.

[0158] Step S34: Obtain the decision data generated by the strategy network during the layout process of each intelligent agent, and use the value network to evaluate the decision data to obtain learning experience data for each component.

[0159] Step S35: Determine the update parameters of the strategy network and value network for each of the intelligent agents using the objective function according to the learning experience data, and update the corresponding intelligent agents using the update parameters.

[0160] Step S36: Obtain a multi-agent joint layout model based on all updated agents.

[0161] In this embodiment, the value network is continuously used for updating during the layout process, and the reward mechanism is used for optimization after the layout is completed. After multiple iterations of training, a trained agent is obtained, and then a multi-agent joint layout model is obtained based on all the agents, and then the multi-agent joint layout model is used to perform actual automatic layout of the printed circuit board.

[0162] Among them, the specific processes of the above steps S31, S32, S34, and S35 can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0163] As can be seen from the above, in this embodiment, the reward value is determined by using the target reward mechanism according to the complete layout of the printed circuit board, and the agent is optimized according to the reward value. Therefore, the value network is continuously used for updating during the layout process, and the reward mechanism is used for optimization after the layout is completed. After multiple iterations of training, the trained agent is obtained, and then a multi-agent joint layout model is obtained based on all agents, and then the multi-agent joint layout model is used to perform the actual automatic layout of the printed circuit board, which improves the layout capability of the printed circuit board and can be applied to large-scale and complex printed circuit board layout tasks.

[0164] Correspondingly, the embodiment of the present application also discloses a method for automatic layout of printed circuit boards, which is applied to the above-mentioned multi-agent joint layout model, including: obtaining parameter information related to the components to be laid out, and grouping information for the components to be laid out generated based on the component type, and assigning the agents in the multi-agent joint layout model to each group of components; inputting the parameter information associated with each group of components into the corresponding agent, so that the multi-agent joint layout model can automatically layout the printed circuit board. Through the distribution of different agents to perform layout operations on different components, the layout of the printed circuit board by multi-agent joint is realized, the influence between the multi-agents is coordinated, the scale of the printed circuit board layout is expanded, and the multi-agent joint layout model is obtained through the continuous update of the evaluation of the value network. After reinforcement learning, the layout ability of the agent is improved, which can be applied to large-scale and complex printed circuit board layout tasks, and the printed circuit board layout ability is improved.

[0165] Correspondingly, the present application embodiment also discloses a multi-agent joint layout model training device, see Figure 4 As shown, the device comprises:

[0166] The grouping module 11 is used to obtain parameter information related to the components to be laid out, as well as grouping information for the components to be laid out generated based on the component types, and configure corresponding intelligent agents for each group of components; each intelligent agent includes a corresponding strategy network and value network;

[0167] A layout module 12, configured to perform layout for the components corresponding to each of the intelligent agents in a distributed manner according to the parameter information, until all components are laid out to obtain a complete layout of the printed circuit board;

[0168] An evaluation module 13, used to obtain the decision data generated by the strategy network during the layout process of each agent, and evaluate the decision data using the value network to obtain learning experience data for each component;

[0169] The network update module 14 is used as a module to determine the update parameters of the strategy network and value network for each of the agents using the objective function according to the learning experience data, update the corresponding agents using the update parameters, and obtain a multi-agent joint layout model based on all the updated agents.

[0170] As can be seen from the above, in this embodiment, parameter information related to components to be laid out and grouping information for the components to be laid out generated based on component types are obtained, and corresponding intelligent agents are configured for each group of components; each intelligent agent contains its own corresponding strategy network and value network; according to the parameter information and in a distributed manner, layout is performed for the corresponding components through each of the intelligent agents until the layout of all components is completed to obtain a complete layout of the printed circuit board; the decision data generated by the strategy network during the layout process of each intelligent agent is obtained, and the decision data is evaluated using the value network to obtain learning experience data for each component; according to the learning experience data, the update parameters of the strategy network and value network for each of the intelligent agents are determined using the objective function, the corresponding agents are updated using the updated parameters, and a multi-agent joint layout model is obtained based on all updated agents.

[0171] It can be seen that by distributing different intelligent agents to execute layout operations on different components, the layout of printed circuit boards can be realized by multi-intelligence joint efforts, the influence between multiple intelligent agents can be coordinated, the scale of printed circuit board layout can be expanded, and the intelligent agents can be updated through the evaluation of the value network, so as to improve the layout ability of the intelligent agents through reinforcement learning, which can be applicable to large-scale and complex printed circuit board layout tasks and improve the printed circuit board layout capabilities.

[0172] In some specific embodiments, the layout module 12 may specifically include:

[0173] A sorting unit, used to sort the layout of components in the corresponding component groups by each of the intelligent agents according to the target sorting rule;

[0174] A layout unit is used to layout the corresponding components through each of the intelligent agents in a distributed manner according to the parameter information and the layout sorting.

[0175] In some specific embodiments, the layout unit may specifically include:

[0176] A state space generating unit, used for obtaining a state space corresponding to each component by combining the parameter information and the occupancy of the printed circuit board;

[0177] A layout action acquisition unit is used to input the state space corresponding to each component into the corresponding strategy network in sequence according to the order of the layout sorting, and determine the position coordinates of the component according to the output layout action; the strategy network is obtained by training based on a deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output.

[0178] In some specific embodiments, the layout action acquisition unit may specifically include:

[0179] A selection unit, used to select the layout action with the highest probability from all the output layout actions corresponding to the current state space;

[0180] The coordinate determination unit is used to determine the horizontal coordinate and the vertical coordinate corresponding to the component according to the layout action with the highest probability.

[0181] In some specific embodiments, the multi-agent joint layout model training device may specifically include:

[0182] A judging unit, configured to judge whether the layout of the current components conforms to the pre-configured component layout rules after each of the intelligent agents performs layout for the corresponding components according to the parameter information in a distributed manner;

[0183] The layout unit is used to execute the layout of the next component if it meets the requirements, and to re-execute the layout of the current component if it does not meet the requirements.

[0184] In some specific embodiments, the grouping module 11 may specifically include:

[0185] A parameter information acquisition unit, used to acquire parameter information of components to be laid out according to circuit-related files; the circuit-related files include one or more of circuit schematics, circuit board files and netlist files; the parameter information includes component parameters and connection information between components;

[0186] The preprocessing unit is used to preprocess the parameter information to obtain parameter information that supports reading by the intelligent agent.

[0187] In some specific embodiments, the pre-processing unit may specifically include:

[0188] The graph embedding generation unit is used to input the parameter information into the graph convolutional neural network to generate a graph embedding including adjacency matrix features, component width, height and type features, and component connection edge features.

[0189] In some specific embodiments, the multi-agent joint layout model training device may specifically include:

[0190] The grid division unit is used to evenly divide the effective area on the printed circuit board into square grids, and sequentially define the coordinate position of each grid point, and add a corresponding occupancy status mark to each grid point.

[0191] In some specific embodiments, the multi-agent joint layout model training device may specifically include:

[0192] A reward unit is used to layout the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board, determine a reward value according to the complete layout of the printed circuit board using a target reward mechanism, and optimize the intelligent agent according to the reward value; the target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

[0193] In some specific embodiments, the reward unit may specifically include:

[0194] The first reward value calculation unit is used to determine the corresponding reward value by estimating the wiring length and wiring congestion according to the complete layout of the printed circuit board if the target reward mechanism is a reward method based on rule constraints.

[0195] In some specific embodiments, the reward unit may specifically include:

[0196] A second reward value calculation unit is used for inputting the complete layout of the printed circuit board and the circuit-related files corresponding to the complete layout of the printed circuit board into a reward function prediction model based on the scoring of human electronic engineers to obtain a reward value output by the reward function prediction model if the target reward mechanism is a reward method based on human expert evaluation;

[0197] The reward function prediction model is obtained after training using sample pairs, wherein the sample pairs include circuit-related files, printed circuit boards corresponding to the circuit-related files generated using a deep reinforcement learning model, and reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to a preset reward value scoring standard.

[0198] In some specific embodiments, the evaluation module 13 may specifically include:

[0199] The decision data determination unit is used to generate corresponding decision data according to the current decision network of each agent; the decision data includes the state space of the components corresponding to the agent, the layout action, the state space transfer after layout, and the reward value corresponding to the agent.

[0200] In some specific embodiments, the evaluation module 13 may specifically include:

[0201] A global action value function determination unit, configured to calculate a global action value function based on the decision data using the value network; the value network is constructed based on action value, with local state space and layout action as input and action value as output;

[0202] A global target action value function determination unit, configured to calculate a global target action value function based on the decision data using a target value network; the target value network has the same network structure as the value network and a model parameter update frequency lower than that of the value network;

[0203] An advantage function determination unit, used to determine a reference value, and obtain an advantage function corresponding to the agent according to a difference between the global target action value and the reference value;

[0204] The learning experience data determining unit is used to obtain the learning experience data corresponding to the component based on the decision data, the global action value function, the global target action value function and the advantage function.

[0205] In some specific embodiments, the advantage function determination unit may specifically include:

[0206] The target global action value determination unit is used to add the joint action of the remaining agents to the action value corresponding to the current agent, and obtain the target global action value using the value network;

[0207] A reference value determination unit, configured to calculate a reference value corresponding to the agent based on the target global action value and the strategy network.

[0208] In some specific embodiments, the calculation formula of the reference value may be:

[0209]

[0210] in, is the joint layout action of the agents except agent i, t is the tth component, is the target global action value, a it is the layout action of agent i, S t is the global status information, is the policy network of agent i, O it is the local environment state space of agent i.

[0211] In some specific embodiments, the global action value function determination unit may specifically include:

[0212] A first approximation unit is used to use the value network to calculate the local action value corresponding to each agent respectively, and approximate the global action value function according to all action values;

[0213] In some specific embodiments, the global target action value function determination unit may specifically include:

[0214] The second approximation unit is used to calculate the local target action value corresponding to each agent using the target value network, and approximate the global target action value function based on all target action values.

[0215] In some specific embodiments, the network update module 14 may specifically include:

[0216] A first update parameter determination unit, configured to determine a first update parameter for each of the strategy networks according to the learning experience data and the optimization objective function;

[0217] The second update parameter determination unit is used to determine a second update parameter for each of the value networks according to the learning experience data and the loss function.

[0218] In some specific embodiments, the objective optimization function may be specifically constructed based on the action advantage function; the loss function may be specifically constructed based on the action value function and the target action value function.

[0219] In some specific embodiments, the network update module 14 may specifically include:

[0220] The updating unit is used to update the network parameters of the strategy network using the first updating parameter and to update the network parameters of the value network using the second updating parameter.

[0221] Furthermore, the present application also discloses an electronic device, see Figure 5 As shown, the contents in the drawings should not be considered as any limitation on the scope of use of the present application.

[0222] Figure 5 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multi-agent joint layout model training method disclosed in any of the aforementioned embodiments.

[0223] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0224] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including parameter information, etc. The storage method can be temporary storage or permanent storage.

[0225] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the multi-agent joint layout model training method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0226] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the multi-agent joint layout model training method disclosed in any of the aforementioned embodiments are implemented.

[0227] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0228] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0229] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0230] The above is a detailed introduction to the multi-agent joint layout model training method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-agent joint layout model training method, characterized in that: include: Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent includes a corresponding strategy network and value network; According to the parameter information and in a distributed manner, each of the intelligent agents performs layout for the components corresponding to each other, until all components are laid out to obtain a complete layout of the printed circuit board; Obtaining decision data generated by the strategy network during the layout process of each agent, and evaluating the decision data using the value network to obtain learning experience data for each component; According to the learning experience data, the update parameters of the strategy network and the value network of each agent are determined using the objective function, the corresponding agent is updated using the update parameters, and a multi-agent joint layout model is obtained based on all the updated agents.

2. The multi-agent joint layout model training method according to claim 1 is characterized in that: The step of performing layout for the components corresponding to the respective components by each of the intelligent agents in a distributed manner according to the parameter information includes: By each of the intelligent agents performing layout sorting on the components in the corresponding component group according to the target sorting rule; According to the parameter information and the layout sorting, each of the intelligent agents performs layout for the components corresponding to each other in a distributed manner.

3. The multi-agent joint layout model training method according to claim 2 is characterized in that: The step of performing layout for the components corresponding to the respective components by each of the intelligent agents in a distributed manner according to the parameter information and the layout sorting includes: Obtaining the state space corresponding to each component by splicing the parameter information and the occupancy of the printed circuit board; According to the order of the layout sorting, the state space corresponding to each component is input into the corresponding strategy network in turn, and the position coordinates of the component are determined according to the output layout action; the strategy network is trained based on the deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output.

4. The multi-agent joint layout model training method according to claim 3 is characterized in that: The step of determining the position coordinates of the components according to the outputted layout action comprises: Select the layout action with the highest probability from all the output layout actions corresponding to the current state space; The horizontal coordinate and the vertical coordinate corresponding to the component are determined according to the layout action with the highest probability.

5. The multi-agent joint layout model training method according to claim 1, characterized in that: After the layout of the components corresponding to each of the intelligent agents is performed in a distributed manner according to the parameter information, the method further includes: Determine whether the current layout of components conforms to the pre-configured component layout rules; If it meets the requirements, the layout of the next component will be executed. If it does not meet the requirements, the layout of the current component will be re-executed.

6. The multi-agent joint layout model training method according to claim 1, characterized in that: The obtaining of parameter information related to the components to be laid out includes: Obtaining parameter information of components to be laid out according to circuit-related files; the circuit-related files include one or more of circuit schematics, circuit board files, and netlist files; the parameter information includes component parameters and connection information between components; The parameter information is preprocessed to obtain parameter information that supports reading by the intelligent agent.

7. The multi-agent joint layout model training method according to claim 6, characterized in that: The preprocessing of the parameter information includes: The parameter information is input into a graph convolutional neural network to generate a graph embedding including adjacency matrix features, component width, height and type features, and component connection edge features.

8. The multi-agent joint layout model training method according to claim 1, characterized in that: Before obtaining parameter information related to the components to be laid out, the method further includes: The effective area on the printed circuit board is evenly divided into square grids, and the coordinate position of each grid point is defined sequentially, and a corresponding occupancy status mark is added to each grid point.

9. The multi-agent joint layout model training method according to claim 1, characterized in that: The method further includes: performing layout for the components corresponding to the components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board; Determining a reward value using a target reward mechanism according to the complete layout of the printed circuit board, and optimizing the agent according to the reward value; The target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

10. The multi-agent joint layout model training method according to claim 9, characterized in that: The step of determining the reward value by using a target reward mechanism according to the complete layout of the printed circuit board includes: If the target reward mechanism is a reward method based on rule constraints, the corresponding reward value is determined by estimating the wiring length and wiring congestion according to the complete layout of the printed circuit board.

11. The multi-agent joint layout model training method according to claim 9, characterized in that: The step of determining the reward value by using a target reward mechanism according to the complete layout of the printed circuit board includes: If the target reward mechanism is a reward method based on human expert evaluation, the complete layout of the printed circuit board and the circuit-related files corresponding to the complete layout of the printed circuit board are input into a reward function prediction model based on human electronic engineer scoring to obtain a reward value output by the reward function prediction model; The reward function prediction model is obtained after training using sample pairs, wherein the sample pairs include circuit-related files, printed circuit boards corresponding to the circuit-related files generated using a deep reinforcement learning model, and reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to a preset reward value scoring standard.

12. The multi-agent joint layout model training method according to claim 1, characterized in that: The acquisition of decision data generated by the strategy network during the layout of each agent includes: Generate corresponding decision data according to the current decision network of each agent; the decision data includes the state space of the components corresponding to the agent, layout actions, state space transfer after layout, and the reward value corresponding to the agent.

13. The multi-agent joint layout model training method according to claim 1, characterized in that: The use of a value network to evaluate the decision data to obtain learning experience data for each component includes: The global action value function is calculated using the value network based on the decision data; the value network is constructed based on the action value, with the local state space and layout action as input and the action value as output; Based on the decision data, a global target action value function is calculated using a target value network; the network structure of the target value network is the same as that of the value network, and the model parameter update frequency is lower than that of the value network; Determine a reference value corresponding to each intelligent agent, and obtain an advantage function corresponding to the intelligent agent according to the difference between the global target action value and the reference value; Based on the decision data, the global action value function, the global target action value function and the advantage function, the learning experience data corresponding to the component is obtained.

14. The multi-agent joint layout model training method according to claim 13, characterized in that: Determining the reference value includes: Add the joint actions of the remaining agents to the action value corresponding to the current agent, and use the value network to obtain the target global action value; Based on the target global action value and the policy network, a benchmark value corresponding to the agent is calculated.

15. The multi-agent joint layout model training method according to claim 14, characterized in that: The calculation formula of the reference value is: in, is the joint layout action of the agents except agent i, t is the tth component, is the target global action value, a it is the layout action of agent i, S t is the global status information, is the policy network of agent i, O it is the local environment state space of agent i.

16. The multi-agent joint layout model training method according to claim 13, characterized in that: The calculating of the global action value function by using the value network includes: The value network is used to calculate the local action value corresponding to each agent respectively, and the global action value function is approximated according to all action values; The calculating of the global target action value function based on the decision data using the target value network includes: The target value network is used to calculate the local target action value corresponding to each agent, and the global target action value function is approximated based on all target action values.

17. The multi-agent joint layout model training method according to claim 13, characterized in that: Determining update parameters of the strategy network and value network of each agent using an objective function according to the learning experience data includes: Determine a first update parameter for each of the strategy networks according to the learning experience data and the optimization objective function; A second update parameter for each of the value networks is determined according to the learning experience data and the loss function.

18. The multi-agent joint layout model training method according to claim 17, characterized in that: The target optimization function is constructed based on the action advantage function; the loss function is constructed based on the action value function and the target action value function.

19. The multi-agent joint layout model training method according to claim 17, characterized in that: The updating of the corresponding agent using the update parameter comprises: The network parameters of the policy network are updated using the first update parameters, and the network parameters of the value network are updated using the second update parameters.

20. A printed circuit board automatic layout method, characterized in that: The multi-agent joint layout model applied to any one of claims 1 to 19 comprises: Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and assign agents in the multi-agent joint layout model to each group of components; The parameter information associated with each group of components is input into the corresponding agent so that the multi-agent joint layout model can perform automatic layout of the printed circuit board.

21. A multi-agent joint layout model training device, characterized in that: include: A grouping module, used to obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent contains its own corresponding strategy network and value network; A layout module, used to layout the corresponding components through each of the intelligent agents in a distributed manner according to the parameter information, until all components are laid out to obtain a complete layout of the printed circuit board; An evaluation module, used to obtain the decision data generated by the strategy network during the layout process of each intelligent agent, and evaluate the decision data using the value network to obtain learning experience data for each component; A network update module is used for a module to determine the update parameters of the strategy network and value network for each of the intelligent agents using the objective function according to the learning experience data, update the corresponding intelligent agents using the update parameters, and obtain a multi-agent joint layout model based on all the updated intelligent agents.

22. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the multi-agent joint layout model training method as described in any one of claims 1 to 19.

23. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by a processor, the multi-agent joint layout model training method as described in any one of claims 1 to 19 is implemented.

Images