A multi-agent joint layout model training method, device, equipment and medium

Abstract

The present application discloses a multi-agent joint layout model training method, apparatus, equipment, and medium, relating to the field of printed circuit board design. The method comprises: obtaining parameter information related to components to be laid out, as well as grouping information for components to be laid out based on component type, configuring a corresponding agent for each group of components; laying out the corresponding components through each agent in a distributed manner according to the parameter information; obtaining the decision data generated by the strategy network during the layout process of each agent, and evaluating the decision data using the value network to obtain learning experience data for each component; determining the update parameters of the strategy network and value network for each agent based on the learning experience data using the objective function, updating the corresponding agent using the updated parameters, and obtaining a multi-agent joint layout model based on all updated agents. This improves the printed circuit board layout capability.

Description

Technical Field

[0001] The present invention relates to the field of printed circuit board design, and in particular to a multi-agent joint layout model training method, device, equipment and storage medium. Background Art

[0002] Printed circuit boards, also known as PCBs, are important electronic components widely used in the production and manufacturing of various electronic devices. PCB layout is a crucial step in the overall design process, directly impacting the wiring design, which is the most labor-intensive later stage, as well as the final PCB performance. The industry typically relies on Electronic Design Automation (EDA) software to complete the entire process, from schematic drawing to PCB layout design. Among related technologies, heuristic algorithms are widely used in automatic PCB layout design. In this algorithm, the combination of the arrangement of all electronic components on the PCB is defined as the solution space, and the optimal solution is obtained through rules or search. While this approach can achieve distributed search and improve computing power, heuristic algorithms converge slowly, are difficult to parallelize, and are prone to falling into local optimality, making them unsuitable for the layout design of large and complex PCBs. Summary of the Invention

[0003] In view of this, the present invention aims to provide a multi-agent joint layout model training method, apparatus, device, and medium that can improve printed circuit board layout capabilities and support large-scale and complex printed circuit board layout tasks. The specific solution is as follows:

[0004] In a first aspect, the present application discloses a multi-agent joint layout model training method, comprising:

[0005] Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent contains its own corresponding strategy network and value network;

[0006] According to the parameter information, each of the intelligent agents performs layout for the corresponding components in a distributed manner until all components are laid out to obtain a complete layout of the printed circuit board;

[0007] Obtaining the decision data generated by the strategy network during the layout process of each agent, and evaluating the decision data using the value network to obtain learning experience data for each component;

[0008] According to the learning experience data, the update parameters of the strategy network and value network of each agent are determined using the objective function, the corresponding agent is updated using the update parameters, and a multi-agent joint layout model is obtained based on all the updated agents.

[0009] Optionally, the performing layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information includes:

[0010] By each of the intelligent agents, the components in the corresponding component group are arranged and sorted according to the target sorting rule;

[0011] According to the parameter information and the layout sorting, each of the intelligent agents performs layout for the corresponding components in a distributed manner.

[0012] Optionally, the performing layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information and the layout sorting includes:

[0013] Obtaining the state space corresponding to each component by combining the parameter information and the occupancy status of the printed circuit board;

[0014] In the order of the layout sorting, the state space corresponding to each component is input into the corresponding strategy network in turn, and the position coordinates of the component are determined according to the output layout action; the strategy network is trained based on a deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output.

[0015] Optionally, determining the position coordinates of the components according to the output layout action includes:

[0016] Select the layout action with the highest probability from all the output layout actions corresponding to the current state space;

[0017] The horizontal coordinate and the vertical coordinate corresponding to the component are determined according to the layout action with the highest probability.

[0018] Optionally, after performing layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information, the method further includes:

[0019] Determine whether the current component layout complies with pre-configured component layout rules;

[0020] If it meets the requirements, the layout of the next component will be executed. If it does not meet the requirements, the layout of the current component will be re-executed.

[0021] Optionally, obtaining parameter information related to components to be laid out includes:

[0022] Obtaining parameter information of components to be laid out according to circuit-related files; the circuit-related files include one or more of a circuit schematic, a circuit board file, and a netlist file; the parameter information includes component parameters and connection information between components;

[0023] The parameter information is preprocessed to obtain parameter information that supports reading by the intelligent agent.

[0024] Optionally, the preprocessing the parameter information includes:

[0025] The parameter information is input into a graph convolutional neural network to generate a graph embedding including adjacency matrix features, component width, height and type features, and component connection edge features.

[0026] Optionally, before obtaining parameter information related to the components to be laid out, the method further includes:

[0027] The effective area on the printed circuit board is evenly divided into square grids, and the coordinate position of each grid point is defined sequentially, and a corresponding occupancy status mark is added to each grid point.

[0028] Optionally, the step of performing layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board further includes:

[0029] Determining a reward value using a target reward mechanism according to the complete layout of the printed circuit board, and optimizing the intelligent agent according to the reward value;

[0030] The target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

[0031] Optionally, determining a reward value by using a target reward mechanism according to the complete layout of the printed circuit board includes:

[0032] If the target reward mechanism is a reward method based on rule constraints, the corresponding reward value is determined by estimating the wiring length and wiring congestion according to the complete layout of the printed circuit board.

[0033] Optionally, determining a reward value by using a target reward mechanism according to the complete layout of the printed circuit board includes:

[0034] If the target reward mechanism is a reward method based on human expert evaluation, the complete printed circuit board layout and the circuit-related files corresponding to the complete printed circuit board layout are input into a reward function prediction model based on human electronic engineer scoring to obtain a reward value output by the reward function prediction model;

[0035] The reward function prediction model is obtained after training using sample pairs, where the sample pairs include circuit-related files, printed circuit boards corresponding to the circuit-related files generated using a deep reinforcement learning model, and reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to a preset reward value scoring standard.

[0036] Optionally, obtaining the decision data generated by the policy network during the layout of each agent includes:

[0037] Generate corresponding decision data based on the current decision network of each agent; the decision data includes the state space of the components corresponding to the agent, layout actions, state space transfer after layout, and the reward value corresponding to the agent.

[0038] Optionally, the use of a value network to evaluate the decision data to obtain learning experience data for each component includes:

[0039] Calculating a global action-value function using the value network based on the decision data; the value network is constructed based on action value, taking the local state space and layout action as input and the action value as output;

[0040] Calculating a global target action value function using a target value network based on the decision data; wherein the target value network has the same network structure as the value network and has a lower model parameter update frequency than the value network;

[0041] Determine a benchmark value corresponding to each agent, and obtain an advantage function corresponding to the agent based on the difference between the global target action value and the benchmark value;

[0042] Based on the decision data, the global action-value function, the global target action-value function and the advantage function, learning experience data corresponding to the component is obtained.

[0043] Optionally, determining the benchmark value corresponding to each agent includes:

[0044] Add the joint action of the remaining agents to the action value corresponding to the current agent, and use the value network to obtain the target global action value;

[0045] Based on the target global action value and the policy network, a benchmark value corresponding to the agent is calculated.

[0046] Optionally, the calculation formula of the benchmark value is:

[0047]

[0048] in, is the joint layout action of the agents except agent i, t is the t-th component, is the target global action value, a it is the layout action of agent i, S t is the global status information, is the policy network of agent i, O it is the local environment state space of agent i.

[0049] Optionally, calculating a global action-value function based on the decision data using the value network includes:

[0050] The value network is used to calculate the local action value corresponding to each agent, and the global action value function is approximated based on all action values;

[0051] The calculating of the global target action value function based on the decision data using the target value network includes:

[0052] The target value network is used to calculate the local target action value corresponding to each agent, and the global target action value function is approximated based on all target action values.

[0053] Optionally, determining update parameters of the strategy network and value network for each agent using an objective function according to the learning experience data includes:

[0054] Determine a first update parameter for each of the strategy networks according to the learning experience data and the optimization objective function;

[0055] A second update parameter for each of the value networks is determined according to the learning experience data and the loss function.

[0056] Optionally, the objective optimization function is constructed based on the action advantage function; and the loss function is constructed based on the action value function and the target action value function.

[0057] Optionally, the updating of the corresponding agent using the update parameter includes:

[0058] The network parameters of the policy network are updated using the first update parameter, and the network parameters of the value network are updated using the second update parameter.

[0059] In a second aspect, the present application discloses a method for automatic layout of printed circuit boards, which is applied to the above-mentioned multi-agent joint layout model, comprising:

[0060] Obtaining parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and assigning agents in the multi-agent joint layout model to each group of components;

[0061] The parameter information associated with each group of components is input into the corresponding agent so that the multi-agent joint layout model can automatically layout the printed circuit board.

[0062] In a third aspect, the present application discloses a multi-agent joint layout model training device, comprising:

[0063] A grouping module is used to obtain parameter information related to the components to be placed, as well as grouping information for the components to be placed based on the component types, and configure a corresponding intelligent agent for each group of components; each intelligent agent contains its own corresponding strategy network and value network;

[0064] a layout module, configured to perform layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information, until all components are laid out to obtain a complete layout of the printed circuit board;

[0065] An evaluation module is used to obtain the decision data generated by the strategy network during the layout process of each intelligent agent and evaluate the decision data using the value network to obtain learning experience data for each component;

[0066] A network update module is used for determining the update parameters of the strategy network and value network for each of the intelligent agents using the objective function according to the learning experience data, updating the corresponding intelligent agents using the updated parameters, and obtaining a multi-agent joint layout model based on all the updated intelligent agents.

[0067] In a fourth aspect, the present application discloses an electronic device, comprising:

[0068] Memory, used to store computer programs;

[0069] A processor is used to execute the computer program to implement the aforementioned multi-agent joint layout model training method.

[0070] In a fifth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned multi-agent joint layout model training method.

[0071] In the present application, parameter information related to components to be laid out, as well as grouping information for the components to be laid out based on the component types, are obtained, and corresponding intelligent agents are configured for each group of components; each intelligent agent contains its own corresponding strategy network and value network; layout is performed for the corresponding components according to the parameter information and in a distributed manner through each intelligent agent until the layout of all components is completed to obtain a complete layout of the printed circuit board; decision data generated by the strategy network during the layout process of each intelligent agent is obtained, and the decision data is evaluated using the value network to obtain learning experience data for each component; based on the learning experience data, the update parameters of the strategy network and value network for each intelligent agent are determined using the objective function, the corresponding intelligent agent is updated using the updated parameters, and a multi-agent joint layout model is obtained based on all updated intelligent agents.

[0072] It can be seen that by distributing different intelligent agents to execute layout operations on different components, the layout of printed circuit boards can be realized by multi-intelligence, the influence between multiple intelligent agents can be coordinated, the scale of printed circuit board layout can be expanded, and the intelligent agents can be updated through evaluation of the value network, so as to improve the layout ability of the intelligent agents through reinforcement learning, support large-scale and complex printed circuit board layout tasks, and improve the printed circuit board layout ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0074] Figure 1 A flowchart of a multi-agent joint layout model training method provided in this application;

[0075] Figure 2 A specific flow chart of the value network evaluation method provided for this application;

[0076] Figure 3 A flowchart of a specific multi-agent joint layout model training method provided in this application;

[0077] Figure 4 A schematic diagram of the structure of a multi-agent joint layout model training device provided in this application;

[0078] Figure 5 This is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0079] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0080] In related technologies, heuristic algorithms are widely used in PCB automatic layout design. Under this algorithm, the combination of the arrangement of all electronic components on the PCB board is defined as the solution space, and the optimal solution is obtained through rules or search. Although this method can improve computing power through distributed search, the heuristic algorithm converges slowly and is difficult to parallelize. It is also prone to falling into local optimality and cannot be applied to the layout design of large and complex PCB boards. To overcome the above technical problems, this application proposes a multi-agent joint layout model training method that can improve the layout capabilities of printed circuit boards and support large-scale and complex printed circuit board layout tasks.

[0081] The present application discloses a multi-agent joint layout model training method, see Figure 1 As shown, the method may include the following steps:

[0082] Step S11: Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent contains its own corresponding strategy network and value network.

[0083] In this embodiment, parameter information related to components to be laid out, such as component parameters and connection information between components, is obtained. Acquiring the parameter information related to components to be laid out can include: acquiring parameter information of components to be laid out based on circuit-related files; the circuit-related files include one or more of circuit schematics, circuit board files, and netlist files; the parameter information includes component parameters and connection information between components; and preprocessing the parameter information to obtain parameter information that supports intelligent agent reading. Specifically, based on files such as circuit schematics, PCB circuit board files, or netlists generated in a previous stage of the PCB design process, the number and type of components to be laid out on the PCB board, parameters of each component, such as component size, number of pins, and distance to ground or other pins, as well as connection types and resistance values ​​between components, are extracted. After obtaining the parameter information, preprocessing is performed to obtain data that supports intelligent agent reading. It is understood that the parameter information is often heterogeneous and can be preprocessed to enable the intelligent agent to conveniently access this data and structure.

[0084] In this embodiment, the preprocessing of the parameter information may include: inputting the parameter information into a graph convolutional neural network to generate a graph embedding containing adjacency matrix features, component width and height and type features, and component connection edge features. That is, taking the input circuit-related file as a netlist file as an example, the parameter information is input into a graph convolutional neural network (GCN, Graph Convolutional Network), which can generate a graph embedding (GraphEmbedding) containing information such as netlist graph adjacency matrix features, component width and height and type features, and component connection edge features. Alternatively, it is also possible to obtain parameter information supporting the intelligent agent by retrieval by constructing a data structure such as a database or a knowledge graph.

[0085] In this embodiment, before obtaining parameter information related to the components to be laid out, the following steps may be performed: evenly dividing the effective area on the printed circuit board into square grids, sequentially defining the coordinate position of each grid point, and adding a corresponding occupation status mark to each grid point. That is, before building the model, the PCB board coordinate division and grid point status definition are performed, including clarifying the actual size and shape of the PCB board, determining the area where components can be placed and the prohibited area, and evenly dividing the available area into square grids with intervals, sequentially defining the coordinate position x of each grid point. i ,y i , that is, the actual layout is finally performed using the grid points as the center points of the components. The thickness of the grid can be determined based on the complexity of the layout problem and the acceptable layout training time or computing resource scale. In addition, it is necessary to define the occupancy status mark. For example, the status of occupied grid points is marked as 0, indicating that the grid point is unavailable, and the status of unoccupied grid points is marked as 1, indicating that the grid point is available. It is also important to note that the grid points covered by the area of ​​the component should all be in the unavailable state.

[0086] In this way, at each decision step, the agent identifies which grid points on the PCB have already been placed with components, thereby adaptively identifying available placement locations. The grid point status on the PCB, or the occupancy status of the printed circuit board, can be determined by extracting image features of the placed components using a convolutional neural network, or by directly sequentially marking and stacking the grid point occupancy status using a vectorized approach.

[0087] In this embodiment, after obtaining parameter information for the components to be laid out, the components are grouped by device type, specifically grouping different types of electronic components, such as resistors, capacitors, transistors, and sensors. A corresponding agent is then assigned to each group of components, allowing different agents to design the layout of each component. A corresponding policy network and value network are constructed for each agent. The policy network is primarily used for layout decisions, while the value network is primarily used to calculate the value of decision actions, enabling value-based updates of model parameters for reinforcement learning.

[0088] In this embodiment, each intelligent agent corresponds to a unique strategy network, that is, the input of the strategy network is the state space obtained by splicing the parameter information of the components corresponding to the intelligent agent and the current occupancy of the printed circuit board, that is, the local state space. The local state spaces corresponding to all intelligent agents are merged to form the global state space. The output of the strategy network is the layout action obtained based on the local state space observation; the value network takes the local state space and the corresponding layout action as input, and outputs the action value.

[0089] Specifically, the construction of the strategy network and value network can use the MAPPO (Multi-agent PPO, multi-agent proximal strategy optimization) algorithm framework. According to the types of electronic components that need to be laid out on the PCB board, the corresponding number of agents is set, and a strategy network is established for each agent. and value network Among them, i is the local environment state space of each agent, and the global environment state information of the entire PCB layout design system is S = {o1, o2, ..., o N};a i is the layout action of each agent based on its own local observation, and the joint action of the system is A = {a1, a2, ..., a N}, i represents the i-th agent. Where N is the number of electronic component groups contained in the system (i.e. the number of agents), θ i 、ω i are the policy network and value network parameters. Considering the lack of immediate action reward guidance in the PCB layout decision process, this solution does not use the state value V(s) of the classic PPO algorithm, but uses the action value Q(s,a). After completing the network settings, initialize all network parameters θ i 、ω i , i∈[1,N].

[0090] Step S12: Layout the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board.

[0091] In this embodiment, each agent performs layout for its corresponding components based on the parameter information in a distributed manner. This utilizes the distributed execution characteristics of multiple agents, enabling distributed parallel execution of layout to improve layout efficiency. Furthermore, different agents can coordinate layout positions through communication until the layout of the components to be laid out is completed, resulting in a complete layout of the printed circuit board. It can be understood that the term "distributed" refers to dividing a large computational problem into many small parts, which are then assigned to multiple agents for processing. Each agent is responsible for a different computational task, while the different agents collaborate with each other. This means that a system composed of multiple agents collaboratively controls the layout of the circuit board, and finally, the computational results of each agent are combined to obtain the final result. This means that each agent distributes the layout of the electronic components within the group, collaboratively completing the layout design of the electronic components on the entire PCB.

[0092] In this embodiment, the layout of the components corresponding to each agent in a distributed manner according to the parameter information may include the following steps:

[0093] S121: Layout and sort components in the corresponding component group according to the target sorting rule by each of the intelligent agents.

[0094] The target sorting rule is based on component priority, with power-related components, components with large footprints, and components with special interface requirements being prioritized. That is, within each group, power-related components, components with large footprints, and components with special interface requirements are prioritized, and the layout of each group of components is sorted accordingly.

[0095] S122: Based on the parameter information and the layout sorting, each of the intelligent agents performs layout for the corresponding components in a distributed manner.

[0096] Among them, according to the parameter information and the layout sorting, each intelligent agent is used in a distributed manner to layout the corresponding components, including: obtaining the state space corresponding to each component by splicing the parameter information and the printed circuit board occupancy status; inputting the state space corresponding to each component into the corresponding strategy network in sequence according to the order of the layout sorting, and determining the position coordinates of the component according to the output layout action; the strategy network is obtained by training based on a deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output.

[0097] It can be understood that in each group, the graph embedding of the component ID to be currently laid out, the PCB grid occupancy, the characteristics of the components and the connection relationship is simply spliced ​​or fully connected to form the state space of the intelligent agent. Subsequently, the layout strategy network with the component layout action as the output will be trained using the PPO algorithm for continuous action space. The layout action is the layout horizontal coordinate x and vertical coordinate y. Among them, considering that the components are placed on the grid points in the actual layout, if the horizontal and vertical coordinate values ​​output by the decision network are inside the grid, the grid point coordinate x is taken downward. i and y i , as the layout action output of the current i-th component in this group.

[0098] In this embodiment, determining the position coordinates of the component based on the output layout action includes: selecting the layout action with the highest probability from all the output layout actions corresponding to the current state space; and determining the horizontal and vertical coordinates corresponding to the component based on the layout action with the highest probability.

[0099] That is, for each agent i, the t-step decision, that is, the decision for component t, is input into the strategy network according to the state space O composed of the component ID to be laid out in each group, the PCB grid occupancy, and the netlist graph embedding. In the above example, we obtain the probability distribution of the horizontal coordinate x and vertical coordinate y of the components on the PCB. In order to simplify the decision-making process, we select the layout action a according to the greedy algorithm each time. it0 , that is, choose the layout action with the highest probability At the same time, take the nearest grid point coordinate x it and y it , as the layout action output a of the tth component that each agent is responsible for it , and the joint action a t ={a 1t , a 2t ,...,a Nt}.

[0100] In this embodiment, after each agent performs layout for its corresponding component in a distributed manner based on the parameter information, the process may further include determining whether the current component layout complies with preconfigured component layout rules; if so, executing the layout for the next component; if not, re-executing the current component layout. It is understood that component layout has certain rules, including ensuring that component spacing, orientation, and connectivity meet actual requirements, as well as avoiding obstructions between components and designing vias. Therefore, after making a decision, it is necessary to determine whether it is feasible based on actual circumstances. If a conflict is identified between the decision and the previously placed component, such as an overlap in positional relationships determined based on the dimensions of the components, the placement of the component is abandoned. Next, the next placement decision for the current component can be directly executed, i.e., the decision step can be re-executed. Alternatively, the current decision grid point can be used as the origin, and the surrounding grid points within the target range can be selected to sequentially perform component layout and validity determination, such as sequentially performing component layout and validity determination on the surrounding [-k, k]*[-k, k] grid points. In order to ensure the consistency of the layout positions of each component, the main consideration is to complete the layout of all electronic components through deep reinforcement learning, that is, to directly proceed to the next decision after determining that the current decision is invalid.

[0101] By repeating the above layout decisions, each agent uses its own strategy network to The decision is made sequentially, the layout of electronic components within the group is completed in a distributed manner, and finally the layout design of electronic components on the entire PCB board is completed collaboratively.

[0102] Step S13: Obtain the decision data generated by the strategy network during the layout process of each intelligent agent, and use the value network to evaluate the decision data to obtain learning experience data for each component.

[0103] In this embodiment, by obtaining the decision data generated by each decision of the strategy network during the agent layout process, the value of each decision action is evaluated using the value network to obtain learning experience data including value evaluation.

[0104] In this embodiment, the acquisition of decision data generated by the strategy network during the layout process of each agent may include: generating corresponding decision data based on the current decision network of each agent; the decision data includes the state space of the components corresponding to the agent, the layout action, the state space transition after layout, and the reward value corresponding to the agent. In this embodiment, after each agent performs the layout action, the state space transition o′ is observed. it , the global state information is transferred to s t ={o′ 1t , o′ 2t ,...,o′Nt At the same time, after each decision, each agent generates a decision data according to the decision of the current strategy network, τ i ={o it , a it , o′ it , r t} t＝1～T , where T is the total number of steps in a complete interactive layout. This means that a piece of decision data is generated for each decision. To save storage space, the global state and action information is not directly stored during the process. Instead, the decision data contains the local state space and layout actions of the agent.

[0105] Step S14: Determine the update parameters of the strategy network and value network for each agent using the objective function according to the learning experience data, update the corresponding agent using the update parameters, and obtain a multi-agent joint layout model based on all updated agents.

[0106] In this embodiment, an objective function is used to determine the update parameters of each agent's policy network and value network based on learning experience data. This objective function can be a target optimization function or a loss function. Specifically, based on the learning experience data containing value evaluations, the objective function is used to analyze the gap with expectations, thereby continuously fine-tuning the agent and continuously improving its layout capabilities. A multi-agent joint layout model is then generated based on all updated agents. It is understood that each time the policy network corresponding to each agent makes a decision, i.e., places a component, a set of decision data is generated. Each set of decision data is evaluated using the value network. Alternatively, after all decisions are made, each set of decision data is evaluated sequentially using the value network. The immediate reward during the decision-making process is zero. After each agent completes layout decisions for all components in its component group, it continuously evaluates the value of its decisions, generates and accumulates learning experience, and after the final printed circuit board layout is completed, a layout reward is calculated for the entire layout diagram. Agent updates are performed based on all evaluations, immediate rewards, and final layout rewards during the decision-making process. The multi-agent joint layout model decomposes the new PCB design parameters and tasks, and the layout strategy network of each agent completes the layout of each electronic component group in a distributed manner to achieve the overall joint layout of the PCB board.

[0107] In this embodiment, parameter information related to the components to be laid out and grouping information for the components to be laid out based on the component type are obtained, and a corresponding intelligent agent is configured for each group of components; each intelligent agent contains its own corresponding strategy network and value network; according to the parameter information and in a distributed manner, layout is performed for the corresponding components through each intelligent agent until the layout of all components is completed to obtain a complete layout of the printed circuit board; the decision data generated by the strategy network during the layout process of each intelligent agent is obtained, and the decision data is evaluated using the value network to obtain learning experience data for each component; according to the learning experience data, the update parameters of the strategy network and value network for each intelligent agent are determined using the objective function, the corresponding intelligent agent is updated using the updated parameters, and a multi-agent joint layout model is obtained based on all updated intelligent agents.

[0108] It can be seen that by distributing different intelligent agents to execute layout operations on different components, the layout of printed circuit boards can be realized by multi-intelligence, the influence between multiple intelligent agents can be coordinated, the scale of printed circuit board layout can be expanded, and the intelligent agents can be updated through evaluation of the value network, so as to improve the layout ability of the intelligent agents through reinforcement learning. It can be applicable to large-scale and complex printed circuit board layout tasks, and improve the printed circuit board layout ability.

[0109] Based on the above embodiments, see Figure 2 As shown, the method of evaluating the decision data using the value network to obtain learning experience data for each component may include the following specific steps:

[0110] Step S21: Calculate the global action-value function using the value network based on the decision data; the value network is constructed based on action value, with the local state space and layout action as input and the action value as output.

[0111] In this embodiment, the value network within the agent is used to calculate the global action value function, that is, according to the decision data τ obtained for each agent, i , using the value network Calculate the value of each decision action in the decision-making process of each agent.

[0112] In this embodiment, the calculation of the global action value function based on the decision data using the value network may include: using the value network to calculate the local action value corresponding to each agent, and approximating the global action value function based on all action values. In this embodiment, considering the large state space in the PCB layout design problem, the designed strategy network and value network are both based on the local observation of the agent. i and a single decision action a i, so the output of the value network is the partial action value, and the global action value is obtained by approximation. The global action value is approximately:

[0113]

[0114] Step S22: Calculate the global target action value function using the target value network based on the decision data; the network structure of the target value network is the same as that of the value network, and the model parameter update frequency is lower than that of the value network.

[0115] In this embodiment, the global target action value function is calculated based on the decision data using the target value network. The network structure of the target value network is the same as that of the value network, and the model parameter update frequency is lower than that of the value network. It can be understood that in order to break the sample correlation between consecutive decisions and improve the stability of the model, a set of target value networks with the same structure as the value network but with a lower model parameter update frequency is set. Specifically used to calculate the target action value.

[0116] The calculation of the global target action value function based on the decision data using the target value network may include: calculating the local target action value corresponding to each agent using the target value network, and approximating the global target action value function based on all target action values. Similarly, the target value network is also based on the local observation of the agent. i and a single decision action a i , so the target value network output is the partial action value, and the global target action value is obtained by approximation. The global target action value is approximately:

[0117]

[0118] Step S23: Determine the benchmark value corresponding to each agent, and obtain the advantage function corresponding to the agent based on the difference between the global target action value and the benchmark value.

[0119] In this embodiment, the global state information and joint action information of the PCB layout design system are integrated to calculate the action advantage function of each agent in the multi-agent distributed decision-making process. Specifically taking into account the mutual influence in the multi-agent collaborative task, a benchmark value is introduced in the calculation process of the action advantage function. The benchmark value reflects the influence of the action decision of other agents on the current agent in the multi-agent collaborative task. The value obtained by subtracting the benchmark value from the global target action value is used as the advantage function, so that the relative contribution evaluation of each agent is more accurate.

[0120] In this embodiment, determining the benchmark value may include: adding the combined actions of the remaining agents to the action value corresponding to the current agent, and using the value network to obtain a target global action value; and calculating the benchmark value corresponding to the agent based on the target global action value and the policy network. That is, the advantage function is:

[0121]

[0122] in, Represents the target action value of the system as a whole at the tth layout decision step, s′ t and a′ t is the global state space and joint layout action at the next moment, γ is the discount factor, which is usually a constant between 0 and 1. is the benchmark value, i.e., the counterfactual benchmark. For each agent, the benchmark value in the above formula is replaced respectively to calculate the respective action advantage function.

[0123] The calculation formula of the benchmark value is:

[0124]

[0125] in, is the joint layout action of the agents except agent i, t is the t-th component, is the target global action value, a it is the layout action of agent i, S t is the global status information, is the policy network of agent i, O it is the local environment state space of agent i.

[0126] Step S24: Based on the decision data, the global action-value function, the global target action-value function and the advantage function, obtain the learning experience data corresponding to the component.

[0127] In this embodiment, the learning experience data corresponding to the component is finally obtained based on the decision data, the global action value function, the global target action value function and the advantage function:

[0128]

[0129] The learning experience is stored in the playback buffer D, where each learning experience is marked as c it .

[0130] Accordingly, in this embodiment, the update parameters of the policy network and value network of each agent are determined by using the objective function based on the learning experience data. Specifically, the data in the playback buffer D can be randomly sorted and numbered, and grouped into M learning records. In actual training, M is often taken as 32 or 64 randomly selected and composed of B small batch sample sets:

[0131] D b ={{c ij} i＝1～N} j＝1～M , b=1~B.

[0132] Different mini-batch sample sets D b Substitute the objective functions separately to determine the update parameters of the policy network and value network for each agent.

[0133] Determining the update parameters for the policy network and value network of each agent using the objective function based on the learning experience data may include: determining a first update parameter for each policy network based on the learning experience data and the optimization objective function; and determining a second update parameter for each value network based on the learning experience data and the loss function. Specifically, the policy network is fine-tuned using the optimization objective function, and the value network is fine-tuned using the loss function. In this embodiment, updating the corresponding agent using the update parameters includes: updating the network parameters of the policy network using the first update parameter, and updating the network parameters of the value network using the second update parameter.

[0134] In this embodiment, the objective optimization function is constructed based on the action advantage function, and the strategy network optimization objective of agent i is as follows:

[0135]

[0136] Among them, θ ib represents the policy network parameters of agent i after b iterations, θ i Indicates the network parameters to be updated. ib (s j , a j ), the target value network of each agent realizes the collaboration between agents centrally and indirectly through the approximate summation operation. clip(·) represents the clipping operation to prevent θ i and θ ib The difference is too large, so the restriction parameter is taken as ε∈(0,1), which is specifically expressed as:

[0137]

[0138] By maximizing the objective function JCLIP (θ i ), the policy network parameters θ of each agent i to update.

[0139] In this embodiment, the loss function is constructed based on the action value function and the target action value function. All agents explicitly share the same loss function L, which centrally and indirectly realizes the collaboration between agents. The overall value function Q(s) of the PCB layout design system at time t is obtained by calculation. j , a j ) and the target value function Then the value loss function L of all M learning experiences in each small batch sample set is:

[0140]

[0141] By minimizing the loss function L, the value network parameter ω of each agent is i The target value network has the same structure as the value network, but the target decision network has a lower frequency of model parameter updates. Then, at intervals of a certain number of steps, such as 1000 training steps, the parameters of the online learning evaluation network are assigned to the target evaluation network to achieve the target network parameter ω′ i Updates.

[0142] It can be seen that the global value function is approximated by using the value function of each agent, and each agent shares the same loss function. At the same time, a benchmark value is introduced to use the mutual influence between agents to calculate the advantage function, and the training and update of the value network and the policy network are realized in a centralized manner. When all B small batch sample sets are iteratively calculated, the training process for a PCB layout design cycle ends and enters the layout interaction phase until the training converges or reaches the set maximum number of training steps. At this time, the target policy model training is completed, and the policy parameters θ of each agent are retained. i Therefore, by implicitly integrating the global observation information of each agent and jointly deploying actions, the policy network and value network model parameter updates of each agent are completed centrally.

[0143] Therefore, the sequential placement of electronic components is modeled as a reinforcement learning sequential decision-making problem. Electronic components with different characteristics are grouped, and global automatic layout of printed circuit boards is achieved through implicit centralized training combined with explicit distributed execution. This method leverages the influence of multiple agents and expands the scale of PCB layouts. Explicit distributed execution supports multiple agents in individually placing components with different performance characteristics, improving execution efficiency while also expanding the scale of PCB layouts. Combined with implicit centralized training, the system achieves indirect and implicit collaboration among multiple agents by approximating the overall value function and shared loss function of the PCB layout and introducing a benchmark value to reflect the influence between agents.

[0144] The present application embodiment discloses a specific multi-agent joint layout model training method, see Figure 3 As shown, the method may include the following steps:

[0145] Step S31: Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent contains its own corresponding strategy network and value network.

[0146] Step S32: Layout the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board.

[0147] Step S33: Determine a reward value using a target reward mechanism according to the complete layout of the printed circuit board, and optimize the intelligent agent according to the reward value.

[0148] In this embodiment, after each agent completes the layout of each group of components and the overall PCB layout, it is fed back through a preset reward mechanism to optimize the agent. Each agent i performs the tth electronic component layout action a in the group. it , only when the layout is completed can the overall calculation get a reward value r T , the immediate rewards in the process are all 0, that is, the reward value is given based on the complete layout result, so the reward value is calculated after all layouts are completed.

[0149] Among them, the target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

[0150] In the first case, that is, if the target reward mechanism is a rule-based reward method, the corresponding reward value is determined based on the complete layout of the printed circuit board by estimating the wiring length and wiring congestion. That is, for the complete PCB layout, the wiring length and wiring congestion are estimated, and a reward R_T is given. The action reward in the process is 0. In the field of EDA, it is required to ensure that the wire length is as short and compact as possible, the circuit delay is small, and the electromagnetic interference is small. At the same time, the wiring congestion ratio is limited to avoid overly dense wiring in certain areas, which will have an adverse effect on the performance and work efficiency of the entire circuit. The reward value calculation process is as follows:

[0151] HPWL(i)=(max b∈i {x b}-min b∈i {x b}+1)+(max b∈i {y b}-min b∈i {y b}+1)

[0152]

[0153]

[0154] R p，g =-HPWL(netlist)-λ*Congestion(p,g)

[0155] Among them, HPWL(i) is the semi-circumference length of the grid point i, which is used to approximate the semi-circumference length of the winding of each electronic component. HPWL(nestlist) represents the winding wire length of all components on the PCB. q(i) is a regularization factor used to improve the accuracy of wire length estimation. In addition, Congestion represents wiring congestion, A refers to the total area of ​​the PCB wiring area, and W max Refers to the widest line width on the PCB board. p，g This represents the cumulative reward that can be obtained by placing netlist g using placement strategy p. This involves calculating the semi-circumference of each electronic component's wires, determining the wire lengths of all components based on these semi-circumferences, determining routing congestion based on these component wire lengths, and finally determining the reward value based on the routing congestion and the wire lengths of the components.

[0156] In the second case, that is, if the target reward mechanism is a reward method based on human expert evaluation, the complete layout of the printed circuit board and the circuit-related files corresponding to the complete layout of the printed circuit board are input into a reward function prediction model based on the scoring of human electronic engineers to obtain the reward value output by the reward function prediction model; the reward function prediction model is obtained after training using sample pairs, and the sample pairs include circuit-related files, and printed circuit boards corresponding to the circuit-related files generated using a deep reinforcement learning model, as well as reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to a preset reward value scoring standard. At this time, the interactive training mode mainly for deep reinforcement learning requires many iterative steps and has a slow model convergence speed. The training of the model is accelerated by introducing the design experience of human electronic engineers. For example, multiple PCB netlist files used in actual projects are selected, and different pre-trained deep reinforcement learning models are used to generate corresponding PCB layout drawings for each file. Experienced electronic engineers score the quality of the PCB layout drawings generated by different networks. Considering that different engineers have different preferences for PCB design styles, the scoring interval for each PCB layout is set to [1, 10], the increment is set to 1, and the quality scores of the PCB layout designs of different engineers for the same netlist and the same model are averaged to obtain the reward value of the PCB design. In this way, a large number of sample pairs are accumulated to form a data set, and each sample pair is [x = [netlist, PCB layout], y = reward value]. The reward function prediction model is trained through supervised learning. During training, the model is continuously updated based on the scoring results of human feedback, so that the model is gradually optimized to be able to better utilize human expert experience to make decisions.

[0157] In the third case, that is, if the target reward mechanism is a reward method based on rule constraints and a reward method based on human expert evaluation, then in the early stage of agent training, a reward function prediction model based on the scoring of human electronic engineers is used to determine the reward value, and in the later stage of agent training, the reward value is determined by estimating the wiring length and wiring congestion based on the complete layout of the printed circuit board. It can be understood that in the interactive PCB joint layout design problem based on multi-agent reinforcement learning, the above evaluation method combined with human scoring is helpful for adjusting the model optimization direction and accelerating the training in the early stage of layout agent training. However, in the later stage of design, in order to encourage each agent to break away from the knowledge limitations of human electronic engineers and explore the layout methods of different types of electronic components, it is necessary to adjust the reward back to the form based on wiring length and wiring congestion constraints, and guide the parameter update of the decision-making agent with a universal optimization goal, so as to break through the limitations of human knowledge. Therefore, by combining different reward methods and adopting different reward methods for different stages, the accuracy of the reward is further improved, thereby accelerating the promotion of the layout ability of the agent.

[0158] Step S34: Obtain the decision data generated by the strategy network during the layout process of each intelligent agent, and use the value network to evaluate the decision data to obtain learning experience data for each component.

[0159] Step S35: Determine the update parameters of the strategy network and value network for each of the agents using the objective function according to the learning experience data, and update the corresponding agents using the update parameters.

[0160] Step S36: Obtain a multi-agent joint layout model based on all updated agents.

[0161] In this embodiment, the value network is continuously used for updating during the layout process, and the reward mechanism is used for optimization after the layout is completed. After multiple iterative trainings, a trained intelligent agent is obtained, and then a multi-agent joint layout model is obtained based on all the intelligent agents, and then the multi-agent joint layout model is used to perform the actual automatic layout of the printed circuit board.

[0162] Among them, the specific processes of the above steps S31, S32, S34, and S35 can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0163] As can be seen above, this embodiment utilizes a target reward mechanism to determine a reward value based on the complete PCB layout, and optimizes the agent based on this reward value. Thus, the value network is continuously updated during the layout process, and the reward mechanism is used for optimization after the layout is completed. Through multiple iterations of this repetitive training, a trained agent is obtained. A multi-agent joint layout model is then generated based on all agents, and this multi-agent joint layout model is then used to perform the actual automated PCB layout. This improves PCB layout capabilities and is applicable to large-scale, complex PCB layout tasks.

[0164] Correspondingly, the embodiment of the present application also discloses a method for automatic layout of printed circuit boards, which is applied to the above-mentioned multi-agent joint layout model, including: obtaining parameter information related to the components to be laid out, and grouping information for the components to be laid out based on the component type, and assigning the agents in the multi-agent joint layout model to each group of components; inputting the parameter information associated with each group of components into the corresponding agent, so that the multi-agent joint layout model can automatically layout the printed circuit board. By distributing layout operations on different components through different agents, the layout of the printed circuit board by multiple agents is realized, the influence between multiple agents is coordinated, the scale of the printed circuit board layout is expanded, and the multi-agent joint layout model is obtained through continuous updating of the evaluation of the value network. After reinforcement learning, the layout ability of the agent is improved, and it can be applied to large-scale and complex printed circuit board layout tasks, thereby improving the printed circuit board layout ability.

[0165] Correspondingly, the present application also discloses a multi-agent joint layout model training device, see Figure 4 As shown, the device includes:

[0166] The grouping module 11 is used to obtain parameter information related to the components to be placed, as well as grouping information for the components to be placed based on the component types, and configure corresponding intelligent agents for each group of components; each intelligent agent includes its own corresponding strategy network and value network;

[0167] A layout module 12 is configured to perform layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board;

[0168] Evaluation module 13, used to obtain the decision data generated by the strategy network during the layout process of each agent, and evaluate the decision data using the value network to obtain learning experience data for each component;

[0169] The network update module 14 is used as a module to determine the update parameters of the strategy network and value network for each of the intelligent agents using the objective function according to the learning experience data, update the corresponding intelligent agents using the updated parameters, and obtain a multi-agent joint layout model based on all the updated intelligent agents.

[0170] As can be seen from the above, in this embodiment, parameter information related to the components to be laid out and grouping information for the components to be laid out based on the component type are obtained, and corresponding intelligent agents are configured for each group of components; each intelligent agent contains its own corresponding strategy network and value network; according to the parameter information and in a distributed manner, each intelligent agent is used to lay out the corresponding components until the layout of all components is completed to obtain a complete layout of the printed circuit board; the decision data generated by the strategy network in the layout process of each intelligent agent is obtained, and the decision data is evaluated using the value network to obtain learning experience data for each component; according to the learning experience data, the objective function is used to determine the update parameters of the strategy network and value network for each intelligent agent, the corresponding intelligent agent is updated using the updated parameters, and a multi-agent joint layout model is obtained based on all updated intelligent agents.

[0171] It can be seen that by distributing different intelligent agents to execute layout operations on different components, the layout of printed circuit boards can be realized by multi-intelligence, the influence between multiple intelligent agents can be coordinated, the scale of printed circuit board layout can be expanded, and the intelligent agents can be updated through evaluation of the value network, so as to improve the layout ability of the intelligent agents through reinforcement learning. It can be applicable to large-scale and complex printed circuit board layout tasks, and improve the printed circuit board layout ability.

[0172] In some specific embodiments, the layout module 12 may specifically include:

[0173] A sorting unit, configured to sort the layout of components in the corresponding component group by each of the intelligent agents according to a target sorting rule;

[0174] A layout unit is used to layout the corresponding components through each of the intelligent agents in a distributed manner according to the parameter information and the layout sorting.

[0175] In some specific embodiments, the layout unit may specifically include:

[0176] A state space generating unit, configured to obtain a state space corresponding to each component by combining the parameter information and the occupancy status of the printed circuit board;

[0177] A layout action acquisition unit is used to input the state space corresponding to each component into the corresponding strategy network in the order of the layout sorting, and determine the position coordinates of the component according to the output layout action; the strategy network is trained based on a deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output.

[0178] In some specific embodiments, the layout action acquisition unit may specifically include:

[0179] A selection unit is used to select the layout action with the highest probability from all the output layout actions corresponding to the current state space;

[0180] The coordinate determining unit is used to determine the horizontal coordinate and the vertical coordinate corresponding to the component according to the layout action with the highest probability.

[0181] In some specific embodiments, the multi-agent joint layout model training device may specifically include:

[0182] a judging unit, configured to judge whether the current layout of components complies with preconfigured component layout rules after each of the intelligent agents performs layout for the corresponding components in a distributed manner according to the parameter information;

[0183] The layout unit is used to execute the layout of the next component if it meets the requirements, and to re-execute the layout of the current component if it does not meet the requirements.

[0184] In some specific embodiments, the grouping module 11 may specifically include:

[0185] A parameter information acquisition unit, configured to acquire parameter information of components to be laid out based on circuit-related files; the circuit-related files may include one or more of a circuit schematic, a circuit board file, and a netlist file; the parameter information may include component parameters and connection information between components;

[0186] The preprocessing unit is used to preprocess the parameter information to obtain parameter information that supports reading by the intelligent agent.

[0187] In some specific embodiments, the pre-processing unit may specifically include:

[0188] A graph embedding generation unit is used to input the parameter information into a graph convolutional neural network to generate a graph embedding including adjacency matrix features, component width, height and type features, and component connection edge features.

[0189] In some specific embodiments, the multi-agent joint layout model training device may specifically include:

[0190] The grid division unit is used to evenly divide the effective area on the printed circuit board into square grids, sequentially define the coordinate position of each grid point, and add a corresponding occupancy status mark to each grid point.

[0191] In some specific embodiments, the multi-agent joint layout model training device may specifically include:

[0192] A reward unit is configured to, based on the parameter information and in a distributed manner, layout the corresponding components by each of the intelligent agents until all components are laid out to obtain a complete printed circuit board layout, determine a reward value based on the complete printed circuit board layout using a target reward mechanism, and optimize the intelligent agent based on the reward value; the target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

[0193] In some specific embodiments, the reward unit may specifically include:

[0194] The first reward value calculation unit is configured to determine a corresponding reward value by estimating wiring length and wiring congestion according to the complete layout of the printed circuit board if the target reward mechanism is a reward method based on rule constraints.

[0195] In some specific embodiments, the reward unit may specifically include:

[0196] a second reward value calculation unit, configured to input the complete printed circuit board layout and the circuit-related files corresponding to the complete printed circuit board layout into a reward function prediction model based on human electronic engineer scoring, and obtain a reward value output by the reward function prediction model, if the target reward mechanism is a reward method based on human expert evaluation;

[0197] The reward function prediction model is obtained after training using sample pairs, where the sample pairs include circuit-related files, printed circuit boards corresponding to the circuit-related files generated using a deep reinforcement learning model, and reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to a preset reward value scoring standard.

[0198] In some specific embodiments, the evaluation module 13 may specifically include:

[0199] The decision data determination unit is used to generate corresponding decision data based on the current decision network of each intelligent agent; the decision data includes the state space of the components corresponding to the intelligent agent, layout actions, state space transfer after layout, and the reward value corresponding to the intelligent agent.

[0200] In some specific embodiments, the evaluation module 13 may specifically include:

[0201] a global action-value function determining unit, configured to calculate a global action-value function based on the decision data using the value network; the value network is constructed based on action values, takes the local state space and layout actions as inputs, and outputs action values;

[0202] a global target action value function determination unit, configured to calculate a global target action value function based on the decision data using a target value network; the target value network has the same network structure as the value network and a lower model parameter update frequency than the value network;

[0203] an advantage function determination unit, configured to determine a reference value and obtain an advantage function corresponding to the agent based on a difference between the global target action value and the reference value;

[0204] The learning experience data determining unit is used to obtain the learning experience data corresponding to the component based on the decision data, the global action value function, the global target action value function and the advantage function.

[0205] In some specific embodiments, the advantage function determination unit may specifically include:

[0206] The target global action value determination unit is used to add the joint action of the remaining agents to the action value corresponding to the current agent, and use the value network to obtain the target global action value;

[0207] A reference value determination unit is used to calculate the reference value corresponding to the agent based on the target global action value and the strategy network.

[0208] In some specific embodiments, the calculation formula of the reference value may be:

[0209]

[0210] in, is the joint layout action of the agents except agent i, t is the t-th component, is the target global action value, a it is the layout action of agent i, S t is the global status information, is the policy network of agent i, O it is the local environment state space of agent i.

[0211] In some specific embodiments, the global action-value function determining unit may specifically include:

[0212] A first approximation unit is used to calculate the local action value corresponding to each agent using the value network, and approximate a global action value function based on all action values;

[0213] In some specific embodiments, the global target action value function determination unit may specifically include:

[0214] The second approximation unit is used to calculate the local target action value corresponding to each agent using the target value network, and approximate the global target action value function based on all target action values.

[0215] In some specific embodiments, the network update module 14 may specifically include:

[0216] A first update parameter determination unit, configured to determine a first update parameter for each of the strategy networks according to the learning experience data and the optimization objective function;

[0217] A second update parameter determination unit is used to determine a second update parameter for each value network according to the learning experience data and the loss function.

[0218] In some specific embodiments, the objective optimization function can be specifically constructed based on the action advantage function; the loss function can be specifically constructed based on the action value function and the target action value function.

[0219] In some specific embodiments, the network update module 14 may specifically include:

[0220] An updating unit is configured to update the network parameters of the policy network using the first updating parameter and to update the network parameters of the value network using the second updating parameter.

[0221] Furthermore, the present application also discloses an electronic device, see Figure 5 The contents in the drawings should not be considered as any limitation on the scope of use of the present application.

[0222] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the multi-agent joint layout model training method disclosed in any of the aforementioned embodiments.

[0223] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0224] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223 including parameter information, etc. The storage method can be temporary storage or permanent storage.

[0225] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, so as to enable the processor 21 to calculate and process the massive data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the multi-agent joint layout model training method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0226] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the multi-agent joint layout model training method disclosed in any of the aforementioned embodiments are implemented.

[0227] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0228] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0229] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0230] The above is a detailed introduction to the multi-agent joint layout model training method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-agent joint layout model training method, characterized in that: include: Obtain parameter information related to components to be laid out, as well as grouping information for the components to be laid out based on component types, and configure corresponding intelligent agents for each group of components; each intelligent agent contains its own corresponding strategy network and value network; According to the parameter information, each of the intelligent agents performs layout for the corresponding components in a distributed manner until all components are laid out to obtain a complete layout of the printed circuit board; Obtaining the decision data generated by the strategy network during the layout process of each agent, and evaluating the decision data using the value network to obtain learning experience data for each component; Determining update parameters for the strategy network and the value network of each agent using an objective function according to the learning experience data, updating the corresponding agent using the update parameters, and obtaining a multi-agent joint layout model based on all updated agents; The step of performing layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information includes: By each of the intelligent agents, the components in the corresponding component group are arranged and sorted according to the target sorting rule; According to the parameter information and the layout sorting, each of the intelligent agents performs layout for the corresponding components in a distributed manner; The step of performing layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information and the layout sorting includes: Obtaining the state space corresponding to each component by combining the parameter information and the occupancy status of the printed circuit board; In the order of the layout sorting, the state space corresponding to each component is input into the corresponding policy network, and the position coordinates of the component are determined based on the output layout action; the policy network is trained based on a deep reinforcement algorithm, and the policy network takes the local state space as input and outputs the layout action obtained based on the local state space observation; The step of determining the position coordinates of the components according to the outputted layout action includes: Select the layout action with the highest probability from all the output layout actions corresponding to the current state space; The horizontal coordinate and the vertical coordinate corresponding to the component are determined according to the layout action with the highest probability.

2. The multi-agent joint layout model training method according to claim 1, characterized in that: After each of the intelligent agents performs layout for the corresponding components according to the parameter information in a distributed manner, the method further includes: Determine whether the current component layout complies with pre-configured component layout rules; If it meets the requirements, the layout of the next component will be executed. If it does not meet the requirements, the layout of the current component will be re-executed.

3. The multi-agent joint layout model training method according to claim 1, characterized in that: The obtaining of parameter information related to the components to be laid out includes: Obtaining parameter information of components to be laid out according to circuit-related files; the circuit-related files include one or more of a circuit schematic, a circuit board file, and a netlist file; the parameter information includes component parameters and connection information between components; The parameter information is preprocessed to obtain parameter information that supports reading by the intelligent agent.

4. The multi-agent joint layout model training method according to claim 3, characterized in that: The preprocessing of the parameter information includes: The parameter information is input into a graph convolutional neural network to generate a graph embedding including adjacency matrix features, component width, height and type features, and component connection edge features.

5. The multi-agent joint layout model training method according to claim 1, characterized in that: Before obtaining parameter information related to the components to be laid out, the method further includes: The effective area on the printed circuit board is evenly divided into square grids, and the coordinate position of each grid point is defined sequentially, and a corresponding occupancy status mark is added to each grid point.

6. The multi-agent joint layout model training method according to claim 1, characterized in that: After the layout of the components corresponding to the components is performed by each intelligent agent in a distributed manner according to the parameter information until all components are laid out to obtain a complete layout of the printed circuit board, the method further includes: Determining a reward value using a target reward mechanism according to the complete layout of the printed circuit board, and optimizing the intelligent agent according to the reward value; The target reward mechanism includes a reward method based on rule constraints and / or a reward method based on human expert evaluation.

7. The multi-agent joint layout model training method according to claim 6, characterized in that: Determining the reward value by utilizing a target reward mechanism according to the complete layout of the printed circuit board includes: If the target reward mechanism is a reward method based on rule constraints, the corresponding reward value is determined by estimating the wiring length and wiring congestion according to the complete layout of the printed circuit board.

8. The multi-agent joint layout model training method according to claim 6, characterized in that: Determining the reward value by utilizing a target reward mechanism according to the complete layout of the printed circuit board includes: If the target reward mechanism is a reward method based on human expert evaluation, the complete printed circuit board layout and the circuit-related files corresponding to the complete printed circuit board layout are input into a reward function prediction model based on human electronic engineer scoring to obtain a reward value output by the reward function prediction model; The reward function prediction model is obtained after training using sample pairs, where the sample pairs include circuit-related files, printed circuit boards corresponding to the circuit-related files generated using a deep reinforcement learning model, and reward values ​​obtained by different engineers scoring the complete layout of the printed circuit board according to a preset reward value scoring standard.

9. The multi-agent joint layout model training method according to claim 1, characterized in that: The acquisition of decision data generated by the policy network during the layout of each agent includes: Generate corresponding decision data based on the current decision network of each agent; the decision data includes the state space of the components corresponding to the agent, layout actions, state space transfer after layout, and the reward value corresponding to the agent.

10. The multi-agent joint layout model training method according to claim 1, characterized in that: The use of a value network to evaluate the decision data to obtain learning experience data for each component includes: Calculating a global action-value function using the value network based on the decision data; the value network is constructed based on action value, taking the local state space and layout action as input and the action value as output; Calculating a global target action value function using a target value network based on the decision data; wherein the target value network has the same network structure as the value network and has a lower model parameter update frequency than the value network; Determine a benchmark value corresponding to each agent, and obtain an advantage function corresponding to the agent based on the difference between the global target action value and the benchmark value; Based on the decision data, the global action-value function, the global target action-value function and the advantage function, learning experience data corresponding to the component is obtained.

11. The multi-agent joint layout model training method according to claim 10, characterized in that: Determining the reference value includes: Add the joint action of the remaining agents to the action value corresponding to the current agent, and use the value network to obtain the target global action value; Based on the target global action value and the policy network, a benchmark value corresponding to the agent is calculated.

12. The multi-agent joint layout model training method according to claim 11, characterized in that: The calculation formula of the reference value is: ； in, is the joint layout action of the agents except agent i, t is the t-th component, is the target global action value, is the layout action of agent i, is the global status information, is the policy network of agent i, is the local environment state space of agent i.

13. The multi-agent joint layout model training method according to claim 10, characterized in that: The calculating of the global action value function by using the value network includes: The value network is used to calculate the local action value corresponding to each agent, and the global action value function is approximated based on all action values; The calculating of the global target action value function based on the decision data using the target value network includes: The target value network is used to calculate the local target action value corresponding to each agent, and the global target action value function is approximated based on all target action values.

14. The multi-agent joint layout model training method according to claim 10, characterized in that: Determining update parameters of the strategy network and value network of each agent using an objective function according to the learning experience data includes: Determine a first update parameter for each of the strategy networks according to the learning experience data and the optimization objective function; A second update parameter for each of the value networks is determined according to the learning experience data and the loss function.

15. The multi-agent joint layout model training method according to claim 14, characterized in that: The objective optimization function is constructed based on the action advantage function; the loss function is constructed based on the action value function and the target action value function.

16. The multi-agent joint layout model training method according to claim 14, characterized in that: The updating of the corresponding agent using the update parameter includes: The network parameters of the policy network are updated using the first update parameter, and the network parameters of the value network are updated using the second update parameter.

17. A printed circuit board automatic layout method, characterized in that: The multi-agent joint layout model applied to any one of claims 1 to 16 comprises: Obtaining parameter information related to components to be laid out, as well as grouping information for the components to be laid out generated based on component types, and assigning agents in the multi-agent joint layout model to each group of components; The parameter information associated with each group of components is input into the corresponding agent so that the multi-agent joint layout model can automatically layout the printed circuit board.

18. A multi-agent joint layout model training device, characterized in that: include: A grouping module is used to obtain parameter information related to the components to be placed, as well as grouping information for the components to be placed based on the component types, and configure a corresponding intelligent agent for each group of components; each intelligent agent contains its own corresponding strategy network and value network; a layout module, configured to perform layout for the corresponding components by each of the intelligent agents in a distributed manner according to the parameter information, until all components are laid out to obtain a complete layout of the printed circuit board; An evaluation module is used to obtain the decision data generated by the strategy network during the layout process of each intelligent agent and evaluate the decision data using the value network to obtain learning experience data for each component; a network update module, configured to determine update parameters for the strategy network and value network of each agent using an objective function according to the learning experience data, update the corresponding agent using the update parameters, and obtain a multi-agent joint layout model based on all updated agents; Among them, the layout module is used to layout and sort the components in the corresponding component group according to the target sorting rules through each intelligent agent; according to the parameter information and the layout sorting, each intelligent agent is used to layout the corresponding components in a distributed manner; the state space corresponding to each component is obtained by splicing the parameter information and the printed circuit board occupancy; according to the order of the layout sorting, the state space corresponding to each component is input into the corresponding strategy network in turn, and the position coordinates of the component are determined according to the output layout action; the strategy network is trained based on the deep reinforcement algorithm, and the strategy network takes the local state space as input and the layout action obtained based on the observation of the local state space as output; the layout action with the highest probability is selected from all the output layout actions corresponding to the current state space; the horizontal and vertical coordinates corresponding to the component are determined according to the layout action with the highest probability.

19. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the multi-agent joint layout model training method as described in any one of claims 1 to 16.

20. A computer-readable storage medium, characterized in that Used to store computer programs; wherein when the computer program is executed by a processor, it implements the multi-agent joint layout model training method as described in any one of claims 1 to 16.

Images