Automated Design of Distributed RF Power Amplifiers Using GAN with Deep Reinforcement Learning

Abstract

A computer-implemented method is provided for training a multi-RL agent network to generate device parameters for a circuit, the method comprising: obtaining inputs for desired circuit specifications, device parameters, a topology of the circuit, and a final state corresponding to a maximum step for the circuit, the desired circuit specifications comprising gain, bandwidth, phase margin, power consumption, output power, and power efficiency, each of the RL agent networks configured to execute a plurality of steps, the plurality of steps comprising sending an action selected from a set of actions to an environment module, updating the device parameters of the circuit related to the circuit specifications, and obtaining a current specification of the circuit by simulating a netlist of the circuit using a circuit simulator of the environment module based on the updated device parameters, the environment module comprising the netlist of the circuit, the plurality of steps further comprising obtaining a reward from the environment module, the reward being calculated based on a difference between the current specification and the desired specification, the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold value or a number of steps reach a preset value, the plurality of steps further comprising generating the satisfied updated device parameters via an interface.

Description

[Technical Field]

[0001] The present invention relates generally to radio frequency circuit design, and more particularly to circuits with distributed architectures. [Background technology]

[0002] Manual design of radio frequency (RF) circuits is notoriously complex, especially for ultra-wideband and high-frequency applications, which typically require complex methods to meet design requirements and also introduce many devices, resulting in extremely exhaustive tuning jobs. As a result, RF circuit design requires more intensive iterations to achieve fine tradeoffs between specifications, which typically rely on the designer's excellent starting experience and extensive tuning experience. Furthermore, textbook design theory has proven insufficient to effectively analyze the coupling effects between all circuit elements in practical designs. Most existing electronic design automation (EDA) software tools offer only limited design support using traditional methods, require time-consuming iterations, and are unable to handle high complexity. Unlike analog circuits that process low-frequency signals, the relatively shorter design cycles for RF circuits have led to a growing need for smarter design tools.

[0003] Recently, several RL-based circuit design automation methods, such as proximal policy optimization (PPO) and deep deterministic policy gradient (DDPG)-based methods, can learn optimal policies to find device parameters to meet design goals. However, most of the aforementioned methods are implemented for analog circuits, where the algorithms do not suffer from the curse of complex correlations between devices caused by high-frequency effects. Furthermore, all previous RL mechanisms are built using a single agent as a representation of the entire circuit or system, which makes it difficult to scale up to more complex cases with multiple parameter systems and limits their application to distributed circuits, such as the distributed power amplifiers primarily described in this invention. Summary of the Invention

[0004] Some embodiments of the present invention recognize that a target RF circuit with a distributed structure can be decomposed into several identical components using expert knowledge, but each component plays a different role in the overall picture. Our automated design is based on multi-agent reinforcement learning (MARL) implemented in the form of centralized training decentralized execution (CTDE) by applying an on-policy actor-critic (PPO) method. Specifically, each circuit component is represented by one agent (also called an actor in PPO), encoded as a graphical convolutional neural network (GCN) in which each device is represented as a node and incorporates circuit-specific information. Hidden features extracted from the actor are fed as input to a "critic" network through a graphical attention network (GAN), in which each agent is modeled as a node, to capture frequency and impedance correlations between agents. In a training scenario, all agents (actors) update their policies using Stochastic Gradient Descent (SGD) by referencing estimated state values ​​from their shared "critic" network, which acts as a central information channel to provide global guidelines for policy iteration. This training mode is therefore called "centralized training." However, in deployment, the critic is ignored, and all agents act independently based on their own observations, which is called "distributed execution."

[0005] Some embodiments of the present invention are based on the recognition that automated design of RF circuits is a long-standing challenge. We present a deep reinforcement learning (RL) framework to assist in the design of RF circuits at the pre-layout level, with the goal of finding device parameters to meet desired specifications. Unlike those prior mechanisms, our multi-agent RL model provides insightful learning strategies for circuits with distributed characteristics and also demonstrates an efficient way to decompose complex circuits or systems for faster learning.

[0006] According to some embodiments of the present invention, there is provided a computer-implemented method for training a multi-agent reinforcement learning (RL) network to generate device parameters for a circuit, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the steps including obtaining inputs relating to desired circuit specifications for the circuit, the device parameters for the circuit, the topology of the circuit, and a final state corresponding to a maximum step, the desired circuit specifications being represented by some of gain, bandwidth, phase margin, power consumption, output power, and power efficiency, or a combination thereof, each of the multi-agent RL networks configured to perform a plurality of steps, the plurality of steps including transmitting a selected action from a set of actions to an environment module including a netlist of the circuit, and using a data processor of the environment module to generate device parameters for the desired circuit specifications in accordance with the selected action. and updating device parameters of the circuit to be updated, the action changing each of the device parameters by the atomic value of each device parameter; the steps further including obtaining current circuit specifications of the circuit by simulating a netlist of the circuit using a circuit simulator of the environment module based on the updated device parameters, and obtaining a reward from the environment module, the reward being calculated based on a difference between the current circuit specifications and the desired circuit specifications; the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold value or until some of the steps reach a preset value; and the steps further including storing the updated device parameters in a memory.

[0007] Embodiments of the present invention may also provide a computer-implemented method for generating device parameters of a circuit using a pre-trained multi-agent reinforcement learning (RL) network, the method employing a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method including obtaining inputs via an interface regarding desired circuit specifications, device parameters, and topology of the circuit, each of the desired circuit specifications being represented by gain, bandwidth, phase margin, power consumption, output power, and power efficiency, and providing the inputs to the pre-trained multi-agent RL network, the steps further including generating the circuit represented by a graph modeling the topology of the circuit, the device parameters of the circuit, and the updated device parameters of the circuit. Some highlights of this invention

[0008] This invention proposes a multi-agent reinforcement learning framework that can automatically design non-uniform distributed power amplifiers with any number of cells based on the semiconductor technology GaN.

[0009] This invention presents two levels of graphical model embedding. The low-level model embedding is realized using a GCN (Graphical Convolutional Network) as an individual policy network for each agent's decision-making, which has been proven to effectively represent neighbor connectivity and interactions between device nodes. The high-level embedding, using a GAN (Graphical Attention Network), receives abstract features (hidden features extracted from the GCN embedding) from all agents and estimates state values ​​as instructions for the agent's direction for better policy learning. Through its internal attention mechanism, the GAN is expected to figure out which parts operate with more importance, which is reflected as the importance of the cell agent. This invention also introduces an implicit or gentle constraint on device parameters, derived from expert knowledge, to accelerate the agent's learning speed, formulated as a regularization term in the cost function. This differs from existing constraints based on mental rules, which present a penalty in the reward function or a formula transformation as input features. The existing constraints are strict and may sometimes cause disturbances in learning smoothness.

[0010] The presently disclosed embodiments are further described with reference to the accompanying drawings. The drawings shown are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram illustrating an overview of a multi-agent RL framework for the automated design of RF circuits, according to an embodiment of the present invention. [Figure 2] 1 is a schematic diagram of an N-cell non-uniform distributed power amplifier according to some embodiments of the present disclosure. [Figure 3] FIG. 1 illustrates a multi-agent RL model embedding based on the actor-critic algorithm, according to some embodiments of the present disclosure. [Figure 4] FIG. 1 illustrates a strategy network coding for an individual agent, according to some embodiments of the present disclosure. [Figure 5] FIG. 2 illustrates a table of specifications (scattering parameters, or S-parameters) for a three-cell non-uniform distributed power amplifier, according to some embodiments of the present disclosure. [Figure 6] FIG. 10 is a histogram of how well specifications are met after sufficient training with MARL, according to some embodiments of the present disclosure. [Figure 7] FIG. 10 illustrates a comparison of training results using single-agent RL and multi-agent RL for a three-cell NDPA, according to some embodiments of the present disclosure. [Figure 8] 1 is a schematic diagram of a system comprising a processor and a memory coupled to a storage that stores a computer-implemented method according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0013] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Also, the same reference numbers and names in the various drawings refer to the same elements.

[0014] The automated design of RF circuits can best be formulated as a parameter-to-specification (P2S) optimization problem. The goal is to find optimal device parameters (e.g., transistor width and length) to meet desired circuit specifications. In this case, the desired circuit specifications can be expressed in terms of gain, bandwidth, phase margin, power consumption, output power, and power efficiency, or some combination thereof. Traditionally, these device parameters are derived manually based on the IC designer's experience and knowledge of the circuit. Such empirical approaches aim to connect circuit specifications and device parameters to mathematical equations through simplified physical models of the devices and circuits. Manually tuning these equations requires a significant effort due to the highly nonlinear characteristics of RF circuits. In recent years, various design automation techniques have been presented, particularly with the rapid development of machine learning algorithms. Traditional optimization algorithms include geometric programming, genetic algorithms, and simulated annealing, in which a well-designed sampling strategy is performed at each parameter iteration to accelerate convergence to a global optimum. In contrast, emerging machine learning-based methods typically use well-trained neural network models to learn the complex relationships between device parameters and circuit specifications to suggest optimal tuning instructions for device parameter prediction in supervised learning and policy approximation in reinforcement learning-based methods. Circuit specifications may include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. Learning with Graph Neural Networks

[0015] Electronic circuits are inherently well suited for graphical representation, which captures interactions between nodes. Graph neural networks, such as graph convolutional networks (GCNs) and graph attention networks (GANs), are two popular proposals that have been widely applied to solving various problems. This invention employs a dual-level graphical representation in multi-agent RL model embedding to capture physical and electrical connections. At a low level, the graph reflects the internal relationships within a distributed PA cell and is encoded into a policy approximator for each agent at the algorithmic level. The higher-level graph is intended to embed the correlations between agents (PA cells) for state value estimation. The graph selection for each level may vary depending on whether the coupling between nearby or distant neighbors has a greater impact. Intuitively, GCNs can directly capture local correlations, while GANs perform better with global correlations. problem presentation

[0016] In this disclosure, we target the design automation of distributed RF circuits at the pre-layout level, where the goal is to efficiently search for optimal device parameters to meet desired specifications. We address the case where the topology of an RF power amplifier circuit is extensible from its inherently distributed structure.

[0017] An overview of the present invention, a multi-agent RL framework for automated design of non-uniform distributed power amplifiers (NDPAs), is described as follows. The RL algorithm is based on the actor-critic policy gradient PPO (neighborhood policy optimization), and the agent's policy update is realized by a well-known method called centralized training distributed execution (CTDE). The environment 110 consists of three components: a netlist 111, an EDA simulator (circuit simulator) 112, and a data processor 113. First, a netlist of a circuit with a given topology is generated, which is determined by a selected number of cells. The EDA simulator takes the netlist and calculates its electrical performance. The simulation results are processed as readable data and passed to the agents as feedback signals from the environment. At each time step, all agents read the state s received from the environment. i and reward r i Select the action you want to take to update your device parameters according to i You will be prompted to output the Multi-agent reinforcement learning framework

[0018] Collaboration: Exchange parts to design results and distributed circuit amplifiers.

[0019] The following paragraphs describe an embodiment of our multi-agent reinforcement learning approach for the automated design of N-cell non-uniformly distributed power amplifiers.

[0020] FIG. 1 illustrates a framework for multi-agent reinforcement learning (MARL), which consists of the following main elements: The agent 100 may include a group of N agents 101. Each agent corresponds to a specific PA cell in a non-uniformly distributed power amplifier (NDPA), which also acts as a decision maker in suggesting an optimal solution. The action space may be a union of executable actions 120. The actions may be represented by increasing, decreasing, or maintaining the parameters of each device by all agents 100. The state space may be a set of possible states corresponding to schematics with different device parameters, including device parameter information 111 and specifications obtained from a data processor 113. The reward may be a scalar value calculated as a quantity of simulation results 121 from the data processor 113 by a crafted reward function. The environment 110 includes the physical world in which the agents operate (i.e., a high-fidelity circuit simulation environment 112). A netlist 111 of a given NDPA circuit is fed into a simulator (electronic design automation: EDA) 112 and the simulated results are processed by a data processor 113 .

[0021]

number

[0022] Figure 2 shows a schematic diagram of a heterogeneous distributed power amplifier with N cells. It can be essentially decomposed into N power amplification units 200, 201, and 202, a bias component VDS 211 for providing a drain node operating voltage via resistor 212, and a gate line bias network 204. The input power signal is sent to the input terminal Pin 210 and amplified in parallel through all cells 200, 201, and 202. Finally, the power is combined at the output terminal Pout 211. The bias circuits are outside the coordination task in the MARL algorithm simply because their design is easier. For the main part of this schematic, each unit is represented as one agent in the RL model, and two neighboring agents are connected via gate and drain lines.

[0023]

number

[0024] We are a subsidiary of i The intermediate reward at time step t is calculated as R t Define it as:

number

[0025] A discrete action space is employed in the algorithm in the form of a vector, e.g., [-2,-1,0,1,2], with the option to decrement or increment by 1 or 2 units, or to keep a device parameter unchanged. For example, the decision to increase the transistor width by 1 unit implies x + Δx. The number of action options is a hyperparameter, which defines how aggressively we expect the agent to act. State Representation

[0026] The state used for policy learning for an individual agent contains two parts: the first part is the concatenation of two vectors [h type and h para ] is the preprocessed device information. type represents the device type, for example, the binary codes [0,0,0] and [0,0,1] represent a transistor and a transmission line (TL), respectively. para are 2-bit device parameters normalized by the upper bound of each specific parameter; zero padding is used when the device has only one physical parameter, e.g., a resistor has only resistance, ignoring the actual resistor width or length. For transmission lines, the algorithm adjusts the width and length simultaneously.

[0027] The second part of the state is the intermediate SP simulation results for each cell, including SP measurements on the drain and gate lines, which are explained in detail in the following paragraphs. Model Building

[0028] Figure 3 presents a diagram of dual-level embedding for a multi-agent reinforcement learning method, which consists of two main components: actor 300 and critic 301. For clear illustration, we represent one benchmark graph embedding method as an example in this section. Specifically, the low-level individual agents are embedded by GCN, and the high-level embedding is realized by GAN. Specifically, cells 200, 201, and 202 in Figure 2 are mapped to the graphical model G(V,E) 310, 311, or 312 in Figure 3, where each node V is a device and the connection between devices is an edge E. In addition, we define the bias node V in all subgraphs. DS , V GS , and ground GND are treated identically. The SP simulation result 320 is refined as a set of vectors including S21, S22 on the drain line and S11, S22 on the gate line. According to the operating frequency band (2 GHz to 18 GHz) defined in the specification, each SP term contains 17 sampled frequency points, with one point every 1 GHz in all SP curves.

[0029] 4 shows how two types of state information are processed through a neural network. Device information 405 is fed through a GCN model 402 and one hidden layer 401 to generate a set of hidden features. Similarly, SP results 406 are fed through an MLP (multiple layer perceptron) and another hidden layer 403, and the obtained hidden features are concatenated to the aforementioned hidden features and then encoded by a fully connected layer 400. The FC layer 400 is one embodiment of 321 in FIG. 3.

[0030] The high-level GAN ​​embedding approximates the value of the critic 301, which is similar to the low-level embedding in Figure 4, but with two major differences. First, the node information is a vector of extracted features from 321, with features for each agent. Second, the input of the MLP embedding, which is replaced with the design goal specification, is also a concatenated SP sample containing S21, S11, and S22. Interaction and strategy learning patterns

[0031] Since the employed algorithm, PPO, is based on on-policy learning, one memory component for storing recent experiences is initialized before formal training. At the beginning of each episode, all devices 200, 201, and 202 in the target circuit are parameterized. This is done randomly or with pre-determined values, depending on the strategy of the RL algorithm to handle the balance between exploration and exploitation. The corresponding netlist 111 is passed to an EDA tool 112 for electrical simulation. The simulation results are processed by a data processor 113 to output meaningful results 121 as feedback signals from the environment, for example, frequency point behaviors to sample for processing low-level SP results 406 or SP averaging for reward function formulation. The current state s t After reading the reward R, the agent 100 is expected to output the action 120 as a command to adjust the parameters, so that a new netlist is output after the parameters are changed, and the state becomes the next state s t+1 As the agent interacts with the environment, (s t, a t, R t, s t+1, π t ) is stored in the memory component. t is a set of action probability distributions that quantify how confident one should be in selecting a particular action as the optimal choice.

[0032] The common goal of the agents (i.e.,) is to produce higher quality decisions as they gain experience from searching for optimal device parameters associated with desired specifications. We can formally define the objective function for the automated design of analog circuits as follows:

number

[0033]

number

[0034]

number

[0035] Before each policy iteration, multiple episodes of interactions need to be performed for policy update, and the interaction records are stored in the memory component as described above. For model parameter updates, episode experience is used several times for stable learning. Training stops according to one predefined criterion, for example, when the averaged episode reward exceeds 8 or when the 30% design goal can be reached within the maximum episode length (which is typically empirical). Deployment Methods

[0036] Once the MARL model is fully trained, the system containing the trained MARL model is configured to perform a design task associated with a new design goal. In the deployment phase, given new specifications sampled from the specification range covered in the training phase, all agents are configured to strive to pursue an optimal solution as quickly as possible. Once all specifications are met, it means that the design task has been successfully performed by this automated design method. A metric for MARL capability is to calculate the percentage of automated design tasks that have been achieved. Design Example

[0037] As an introduction, we will explain how the automated design of a three-cell nonuniformly distributed power amplifier using the proposed MARL algorithm is achieved. Figure 5 shows an example of specifications (scattering parameters, or S-parameters) for a three-cell nonuniformly distributed power amplifier according to some embodiments of the present disclosure. The second column in Figure 5 lists the specifications in metrics for all SP simulation terms. By convention, large-signal simulations such as harmonic balance (HB) should be performed on RF power amplifiers, but this is replaced with small-signal SP simulations simply due to shorter execution times. To ensure that SP measurements can reflect actual HB measurements, it is suggested to set the SP specifications a little more restrictively than the corresponding HB. The empirical setting comes from the fact that small-signal simulations are typically more optimistic than large-signal simulations because they ignore significant parasitic effects that can sometimes dominate in some devices.

[0038] Figure 6 shows a histogram of how well the specifications are met after sufficient training with MARL, compared to the defined specifications in Figure 5.

[0039] Figure 7 shows a comparison of training results using single-agent RL and multi-agent RL for a three-cell NDPA.

[0040] FIG. 8 is a schematic diagram of a system 1100 comprising a processor 1120, a memory 1130 coupled to storage 1140 storing a computer-implemented method, and an interface 1105, in accordance with some embodiments of the present disclosure.

[0041] 8 shows a block diagram illustrating an example of a system 1100 for automated construction of artificial neural network architectures. The system 1100 includes a set of interfaces and data links 1105 configured to send and receive signals, at least one processor 1120, a memory (or a set of memory banks) 1130, and storage 1140. The processor 1120, in association with the memory 1130, executes computer-executable programs and algorithms stored in the storage 1140. The set of interfaces and data links 1105 may include a human machine interface (HMI) 1110 and a network interface controller 1150. The processor 1120 may execute computer-executable programs and algorithms in association with the memory 1130, which uploads the computer-executable programs and algorithms from the storage 1140. The computer-executable programs and algorithms stored in storage 1140 may be pre-trained RL agents (neural networks) 1141, graph neural networks 1142, fully connected neural networks 1143, instructions (programs) for performing computer-implemented methods 1144, temporary cache 1145, device parameters 1146, and environmental modules 1147.

[0042] The device parameters may be the structure of the transistor. For example, the device parameters may include gate width, gate length, transistor digits, and capacitance. The desired specifications may be gain (G), bandwidth (B), phase margin (PM), power consumption (P), power efficiency (E), and output power (P).

[0043] The system 1100 can receive signals via a set of interfaces and data links, which may be data sets consisting of training data, desired circuit description data, and test data including gain, bandwidth, phase margin, power consumption, output power, and power efficiency.

[0044] At least one processor 1120, in association with an interface and memory bank 1105, is configured to inject signals and data sets into the reconfigurable DNN block 1141. The system 1100 may receive circuit data or perform circuit simulations with an associated computing system 1195B via a network 1190 and a set of interfaces and data links 1105. Comparison with single-agent RL methods

[0045] Several existing and emerging mechanisms demonstrate the ability of RL-based automated design methods to learn strategies for both analog and RF circuit design. However, not all of these prior mechanisms implement more complex circuit topologies with a large number of devices or the ability to handle more complex frequency- and signal-amplitude-related correlations, which are common challenges in RF circuit design due to ubiquitous parasitics. In addition, circuits with scalable structures, such as distributed power amplifiers, have long been elusive to optimally design since their invention, yet have never been touched by intelligent algorithms. Our MARL method is a novel and well-suited candidate for modeling the scalability of such topologies. The intuition is that compact individual agents function as local state observers and learners, which can better focus on local complexity and thus enable more efficient learning than traditional single-agent equivalents, which require a vast approximator to capture the entire story of a given topology, resulting in redundancy. More importantly, the coupling effects between agents are adequately captured by the high-level agent networking aspect (graphical mapping in the proposed embodiment), which, although abstract, is effective in estimating direct or indirect coupling effects and is consistent with the design theory.

[0046] Our framework also demonstrates good generalization ability, i.e., generating optimal design parameters for unseen targets. We believe our method can help the IC industry accelerate analog chip design using artificial agents that can accumulate large amounts of circuit optimization experience through continuous training.

[0047] Also, particular embodiments may be described as a process, which is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While a flowchart may describe operations as a sequential process, many of the operations may occur in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not described or included in the drawings. Moreover, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the end of the function may correspond to the function returning to the calling function or main function.

[0048] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. The manual or automatic implementation may be performed or at least assisted through the use of machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0049] The above-described embodiments of the present disclosure may be implemented in any of many ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether the processors are provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit having one or more processors within an integrated circuit component. However, the processor may be implemented using circuitry in any suitable format.

[0050] Also, the various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0051] Additionally, embodiments of the present disclosure may be embodied as a method, an example of which is provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, while an exemplary embodiment shows actions as sequential, embodiments may be constructed in which actions are performed in a different order than illustrated, including performing some actions simultaneously. Additionally, the use of ordinal terms such as "first," "second," etc. in the claims to modify claim elements does not, in itself, imply any priority, precedence, or ordering of one claim element relative to another claim element, or any chronological order in which the actions of the method are performed, but is merely used as a label to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms).

[0052] Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is, therefore, the object of the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the present disclosure.

Claims

1. 1. A computer-implemented method for training a multi-agent reinforcement learning (RL) network to generate device parameters for a circuit, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the steps comprising: obtaining inputs relating to desired circuit specifications of a circuit, device parameters of the circuit, a topology of the circuit, and a final state corresponding to a maximum step, the desired circuit specifications being expressed in terms of gain, bandwidth, phase margin, power consumption, output power, and power efficiency, or a combination thereof; decomposing the circuit into a plurality of amplification units, each amplification unit being modeled as an agent in the multi-agent RL network, each agent of the multi-agent RL network being configured to perform a plurality of steps, the plurality of steps comprising: sending a selected action from a set of actions to an environment module containing a netlist of said circuit; and updating the device parameters of the circuit relative to the desired circuit specification using a data processor of the environment module in accordance with the selected action, the action changing each of the device parameters by the atomic value of each device parameter, the steps further comprising: obtaining a current circuit specification for the circuit by simulating the netlist of the circuit using a circuit simulator of the environment module based on the updated device parameters; and obtaining a reward from the environment module, wherein the reward is calculated based on a difference between the current circuit specification and the desired circuit specification, and a strategy of each agent is updated based on the desired circuit specification and features extracted from all agents of the multi-agent RL network, based on a circuit state value estimated by a shared graphical attention network and the reward, and the sending, updating, acquiring, and obtaining steps are continued until the reward reaches a threshold value or until some of the steps reach a preset value, and the plurality of steps further comprises: Storing the updated device parameters in the memory.

2. 2. The method of claim 1, wherein the circuit is modeled by a graph G(V,E), where each node V is represented by a device and edges E represent connections between devices.

3. 2. The method of claim 1, wherein power supply voltages (VP), ground voltages (VGND), and other DC bias voltages in the topology of the circuit are represented as redundant nodes V.

4. The method of claim 1 , wherein the reward is calculated by a weighted sum of the desired specifications by assigning different values ​​of weighting coefficients.

5. The method of claim 1 , wherein the multi-agent RL network includes a graph neural network (GNN) and a fully connected neural network (FCN).

6. The method of claim 5 , wherein the FCNN embeds the desired specifications of the circuit and the GNN embeds the topology and parameters of the circuit.

7. 1. A computer-implemented method for generating device parameters for a circuit using a pre-trained multi-agent reinforcement learning (RL) network, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the steps comprising: obtaining input via an interface regarding desired circuit specifications, device parameters, and topology of a circuit, the circuit corresponding to a distributed circuit including a plurality of amplification units, each amplification unit being modeled as an agent in the pre-trained multi-agent RL network, the step further comprising: providing said inputs to said pre-trained multi-agent RL network, wherein each of said desired circuit specifications is represented by a gain, a bandwidth, a phase margin, a power consumption, an output power, and a power efficiency, said step further comprising: generating a circuit represented by a graph that models the topology of the circuit, the device parameters of the circuit, and updated device parameters of the circuit.

Images