Automated Design of Distributed RF Power Amplifiers Using Deep Reinforcement Learning

Abstract

A computer-implemented method is provided for training a multi-RL agent network to generate device parameters for a circuit, the method comprising: obtaining inputs for desired circuit specifications, device parameters, a topology of the circuit, and a final state corresponding to a maximum step for the circuit, the desired circuit specifications comprising gain, bandwidth, phase margin, power consumption, output power, and power efficiency, each of the RL agent networks configured to execute a plurality of steps, the plurality of steps comprising sending an action selected from a set of actions to an environment module, updating the device parameters of the circuit related to the circuit specifications, and obtaining a current specification of the circuit by simulating a netlist of the circuit using a circuit simulator of the environment module based on the updated device parameters, the environment module comprising the netlist of the circuit, the plurality of steps further comprising obtaining a reward from the environment module, the reward being calculated based on a difference between the current specification and the desired specification, the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold value or a number of steps reach a preset value, the plurality of steps further comprising generating the satisfied updated device parameters via an interface.

Description

[Technical field]

[0001] The present invention relates generally to radio frequency circuit design, and more particularly to circuits with distributed architectures. [Background technology]

[0002] Manual design of Radio Frequency (RF) circuits is notoriously complicated, especially for ultra-wideband and high frequency applications, which typically require complex methods to meet design requirements, as well as introduce more devices, making the tuning job extremely exhaustive. As a result, RF circuit design requires more intensive iterations to achieve fine trade-offs among specifications, which typically rely on the designer's excellent starting experience and rich tuning experience. Also, textbook design theories have proven insufficient to analyze the coupling effects between all circuit elements in practical designs quite effectively. Most existing electronic design automation (EDA) software tools only provide limited design assistance using traditional methods, require time-consuming iterations, and are unable to handle high complexity. Unlike analog circuits that process low-frequency signals, the need for smarter design tools for RF circuits has been growing due to the relatively shorter design cycles.

[0003] Recently proposed several RL-based circuit automation design methods, such as proximal policy optimization (PPO) and deep deterministic policy gradient (DDPG) based methods, can learn optimal policies to find device parameters to meet design objectives. However, most of the aforementioned methods are implemented for analog circuits, where the algorithms do not suffer from the curse of complex correlations between devices caused by high frequency effects. Also, all the prior RL mechanisms are built with a single agent as a representation of the whole circuit or system, which is difficult to scale up to more complex cases with multiple parameter systems, and limits their application to distributed circuits such as distributed power amplifiers, which are mainly described in this invention. Summary of the Invention

[0004] Some embodiments of the present invention are based on the realization that a target RF circuit with a distributed structure can be decomposed into several identical components from expert knowledge, but each component plays a different role in the overall picture. Our automated design is based on Multi-agent Reinforcement Learning (MARL) realized in the form of centralized training decentralized execution (CTDE) by applying an on-policy actor-critic method PPO. Specifically, each circuit component is represented by one agent (also called actor in PPO) encoded as a graphical convolutional neural network (GCN) where each device is represented as a node and incorporates circuit-specific information. Hidden features extracted from the actor are fed as input to a "critic" network through a graphical attention network (GAN) where each agent is modeled as a node, capturing correlations in terms of frequency and impedance between agents. In the training scenario, all agents (actors) update their policies using Stochastic Gradient Descent (SGD) by referring to the estimated state values ​​from their shared "critic" network, which acts as a central information channel to provide global guidelines for policy iteration. This training mode is therefore called "centralized training". However, in deployment, the critic is to be ignored and all agents act independently based on their own observations, which is called "distributed execution".

[0005] Some embodiments of the present invention are based on the recognition that automated design of RF circuits is a long-standing challenge. We provide a deep reinforcement learning (RL) framework to assist in the design of RF circuits at a pre-layout level, with the goal of finding device parameters to meet desired specifications. Unlike those prior mechanisms, our multi-agent reinforcement learning model provides an insightful learning scheme in circuits with distributed characteristics, and also demonstrates an efficient way to decompose complex circuits or systems for faster learning.

[0006] According to some embodiments of the invention, there is provided a computer-implemented method for training a multi-agent reinforcement learning (RL) network to generate device parameters for a circuit, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the steps including obtaining inputs relating to desired circuit specifications for the circuit, device parameters for the circuit, a topology of the circuit, and a final state corresponding to a maximum step, the desired circuit specifications being represented by some of gain, bandwidth, phase margin, power consumption, output power, and power efficiency, or a combination thereof, each of the multi-agent RL networks being configured to perform a plurality of steps, the plurality of steps including transmitting an action selected from a set of actions to an environment module including a netlist of the circuit, and using a data processor of the environment module to generate device parameters for the desired circuit specifications in accordance with the selected action. and updating device parameters of the circuit to be updated, the action changing each of the device parameters by the unit value of each device parameter, the steps further including obtaining current circuit specifications of the circuit by simulating a netlist of the circuit using a circuit simulator of the environment module based on the updated device parameters, and obtaining a reward from the environment module, the reward being calculated based on a difference between the current circuit specifications and the desired circuit specifications, the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold value or until some of the steps reach a preset value, and the steps further including storing the updated device parameters in a memory.

[0007] An embodiment of the invention may also provide a computer-implemented method for generating device parameters of a circuit using a pre-trained multi-agent reinforcement learning (RL) network, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, including obtaining inputs via an interface regarding desired circuit specifications, device parameters, and topology of the circuit, each of the desired circuit specifications being represented by gain, bandwidth, phase margin, power consumption, output power, and power efficiency, and providing the inputs to the pre-trained multi-agent RL network, the steps further including generating the circuit represented by a graph modeling the topology of the circuit, the device parameters of the circuit, and the updated device parameters of the circuit. Some highlights of this invention

[0008] The present invention proposes a multi-agent reinforcement learning framework capable of automatically designing non-uniform distributed power amplifiers with any number of cells based on the semiconductor technology GaN.

[0009] The present invention presents two levels of graphical model embedding. The low-level model embedding is realized using GCN (Graphical Convolutional Network) as an individual policy network for each agent's decision making, which has been proven to represent the neighbor connectivity and interactions between device nodes well. The high-level embedding using GAN (Graphical Attention Network) receives abstract features, which are hidden features extracted from GCN embedding, from all agents to estimate state values ​​as instructions on where to go for better policy learning of the agents. The GAN is expected to figure out which part operates with more importance through its internal attention mechanism, which is reflected as the importance of the cell agent. The present invention also introduces one implicit or gentle constraint on device parameters, which is derived from expertise to accelerate the learning speed of the agent, which is formulated as one regularization term in the cost function. It is different from existing constraints based on mental rules, where the constraint presents a penalty or a formula transformation in the reward function as an input feature. The existing constraint is a hard constraint and in some cases, it brings about disturbance in learning smoothness.

[0010] The presently disclosed embodiments are further described with reference to the accompanying drawings, in which the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram illustrating an overview of a multi-agent RL framework for automated design of RF circuits, according to an embodiment of the present invention. [Diagram 2] 1 is a schematic diagram of an N-cell non-uniform distributed power amplifier in accordance with some embodiments of the present disclosure. [Diagram 3] FIG. 1 illustrates a multi-agent RL model embedding based on the Actor-Critic Algorithm, in accordance with some embodiments of the present disclosure. [Figure 4] FIG. 1 illustrates a strategy network coding for an individual agent, in accordance with some embodiments of the present disclosure. [Diagram 5] FIG. 2 illustrates a table of specifications (scattering parameters, or S-parameters) for a three-cell nonuniform distributed power amplifier, according to some embodiments of the present disclosure. [Figure 6] FIG. 13 is a histogram of how well specifications are met after sufficient training with MARL, in accordance with some embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates a comparison of training results using single-agent RL and multi-agent RL for a three-cell NDPA, according to some embodiments of the present disclosure. [Figure 8] 1 is a schematic diagram of a system configured with a processor and a memory coupled to a storage device storing a computer-implemented method according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0013] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Additionally, the same reference numbers and names in the various drawings refer to the same elements.

[0014] The automated design of RF circuits can be best formulated as a parameter-to-specification (P2S) optimization problem. The goal is to find optimal device parameters (e.g., transistor width and length) to meet the desired circuit specifications. In this case, the desired circuit specifications can be represented by some of the gain, bandwidth, phase margin, power consumption, output power, and power efficiency, or a combination thereof. Traditionally, these device parameters are derived manually based on the IC designer's experience and knowledge of the circuit. Such empirical approaches aim to connect the circuit specifications and device parameters to mathematical formulas through simplified physical models of the devices and circuits. Manually tuning those formulas requires a large effort due to the highly nonlinear characteristics of RF circuits. In recent years, various design automation techniques have been presented, especially as machine learning algorithms rapidly develop. Traditional optimization algorithm methods include geometric programming, genetic algorithms, and simulated annealing, where a well-designed sampling strategy is performed at each parameter iteration to accelerate the convergence to a global optimum. In contrast, emerging machine learning-based methods typically use well-trained neural network models to learn the complex relationships between device parameters and circuit specifications to suggest optimal tuning instructions for device parameter prediction in supervised learning and for policy approximation in reinforcement learning-based methods. Circuit specifications may include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. Learning with Graph Neural Networks

[0015] Electronic circuits are inherently well suited for graphical representation, which captures interactions between nodes. Graph neural networks, such as graph convolutional networks (GCN) and graph attention networks (GAN), are two generalized proposals that have been widely applied to solve various problems. The present invention employs dual-level graphical representation in multi-agent RL model embedding, aiming to capture physical and electrical connections. At a low level, the graph reflects the internal relationships within one distributed PA cell, which is encoded into a policy approximator for each agent at the algorithm level. The high-level graph is for embedding the correlations between agents (PA cells) for state value estimation. The graph selection for each level may be different, depending on whether the coupling between near or far neighbors has a larger impact. Intuitively, GCN can directly capture local correlations, while GAN works better for global correlations. problem presentation

[0016] In this disclosure, we target the design automation of distributed RF circuits at the pre-layout level, where the goal is to efficiently search for optimal device parameters to meet desired specifications. We address cases where the topology of an RF power amplifier circuit is extensible from its inherently distributed structure.

[0017] The inventive overview of a multi-agent RL framework for non-uniform distributed power amplifier (NDPA) automated design is described as follows. The RL algorithm is based on the actor-critic policy gradient PPO (neighborhood policy optimization), and the agent's policy update is realized in a well-known manner called centralized training distributed execution (CTDE). The environment 110 consists of three components: a netlist 111, an EDA simulator (circuit simulator) 112, and a data processor 113. First, a netlist of a circuit with a given topology is generated, which is determined by a selected number of cells. The EDA simulator takes in the netlist and calculates the electrical performance. The simulation results are processed as readable data and passed to the agents as feedback signals from the environment. At each time step, all agents read the state s received from the environment and convert it into a scalable state. i and reward r i Select the action you are sure to take to update your device parameters according to i You will be prompted to output the following. Multi-agent Reinforcement Learning Framework

[0018] Collaboration: Exchange parts to design results and distributed circuit amplifiers.

[0019] The following paragraphs describe an embodiment of our multi-agent reinforcement learning approach for the automated design of N-cell non-uniformly distributed power amplifiers.

[0020] FIG. 1 shows a framework of multi-agent reinforcement learning (MARL), which consists of the following main elements: Agent 100 may include a group of N agents 101. Each agent corresponds to a specific PA cell in a non-uniform distributed power amplifier (NDPA), which also acts as a decision maker in suggesting an optimal solution. Action space may be a union set of executable actions 120. The actions may be represented by increasing, decreasing, or maintaining the parameters of each device by all agents 100. State space may be a set of possible states corresponding to schematics with different device parameters, including device parameter information 111 and specifications obtained from data processor 113. Reward may be a scalar value calculated as an amount of simulation result 121 from data processor 113 by an elaborated reward function. Environment 110 includes the material world (i.e., high-fidelity circuit simulation environment 112) in which the agents operate. A netlist 111 for a given NDPA circuit is fed into a simulator (electronic design automation: EDA) 112 and the simulated results are processed by a data processor 113 .

[0021]

number

[0022] Figure 2 shows a schematic diagram of a non-uniform distributed power amplifier with N cells. It can be essentially decomposed into N power amplification units 200, 201, and 202, a bias component VDS211 for providing a drain node operating voltage through resistor 212, and a gate line bias network 204. The input power signal is sent to the input end Pin210 and is amplified in parallel through all cells 200, 201, and 202. Finally, the power is combined at the output end Pout211. The bias circuits are out of the coordination task in the MARL algorithm simply because they are easier to design. For the main part of this schematic diagram, each unit is represented as one agent in the RL model, and two neighboring agents are connected via gate and drain lines.

[0023]

number

[0024] We are the sub-reward i The intermediate reward at time step t is calculated as R t Define it as:

number

[0025] A discrete action space is employed in the algorithm in the form of a vector, e.g. [-2,-1,0,1,2], with the option to decrement or increment by 1 or 2 units, or to keep the device parameters unchanged. For example, the decision to increase the transistor width by 1 unit means x+Δx. The number of action options is a hyperparameter, defining how aggressively we expect the agent to act. State Representation

[0026] The state used to train the policies of individual agents contains two parts. The first part is the concatenation of two vectors [h type and h para ] is the preprocessed device information, where h type represents the device type, for example, the binary codes [0,0,0] and [0,0,1] represent transistor and transmission line (TL), respectively. And, para are 2-bit device parameters normalized by the upper limit of each specific parameter; zero padding is used when a device has only one physical parameter, e.g., a resistor has only resistance ignoring the actual resistor width or length. For transmission lines, the algorithm adjusts the width and length simultaneously.

[0027] The second part of the status is the intermediate SP simulation results for each cell, including SP measurements on the drain and gate lines, which are described in detail in the following paragraphs. Model Building

[0028] FIG. 3 presents a diagram of dual-level embedding for a multi-agent reinforcement learning method, which consists of two main components: actors 300 and critics 301. For clear illustration, we represent one benchmark graph embedding method as an example in this section. Specifically, the low-level individual agents are embedded by GCN, and the high-level embedding is realized by GAN. Specifically, cells 200, 201, and 202 in FIG. 2 are mapped to the graphical model G(V,E) 310, 311, or 312 in FIG. 3, where each node V is a device and the connection between devices is an edge E. In addition, we consider the bias node V in all subgraphs as the embedding method. DS , V GS , and ground GND are treated as exactly the same. The SP simulation result 320 is refined as a set of vectors including S21, S22 on the drain line, and S11, S22 on the gate line. According to the operating frequency band (2GHz~18GHz) defined in the specification, each SP term contains 17 sampled frequency points, with one point every 1GHz in every SP curve.

[0029] Figure 4 shows how two types of state information are processed through a neural network. Device information 405 is fed through a GCN model 402 and one hidden layer 401 to generate a set of hidden features. Similarly, SP results 406 are fed through a MLP (multiple layer perceptron) and another hidden layer 403, and the obtained hidden features are concatenated with the above hidden features and then encoded by a fully connected layer 400. The FC layer 400 is one embodiment of 321 in Figure 3.

[0030] The high-level GAN ​​embedding is to approximate the value of the critic 301, which is similar to the low-level embedding in Figure 4, with two major differences. First, the node information is a vector of extracted features from 321, with features for each agent. Second, the input of the MLP embedding, which is replaced with the design goal specification, is also the concatenated SP samples including S21, S11, and S22. Interaction and strategy learning patterns

[0031] Since the adopted algorithm PPO is based on on-policy learning, one memory component for storing recent experiences is initialized before formal training. At the beginning of each episode, all devices 200, 201, and 202 in the target circuit are parameterized. This is done randomly or with pre-scheduled values, depending on the strategy of the RL algorithm to handle the balance between exploration and exploitation. The corresponding netlist 111 is passed to an EDA tool 112 for electrical simulation. The simulation results are processed by a data processor 113 to output a meaningful result 121 as a feedback signal from the environment, for example the behavior of frequency points to sample for processing of low-level SP results 406 or averaging SP for reward function formulation. The current state s t After reading and reward R, agent 100 is expected to output action 120 as a command to adjust parameters, so that a new netlist is output after the parameters are changed, and the state becomes the next state s t+1 As the agent interacts with the environment, (s t, a t, R t, s t+1, π t ) is stored in the memory component. t is a set of action probability distributions that quantify how confident one should be in selecting a particular action as the optimal choice.

[0032] The common goal of the agents (i.e.,) is to produce higher quality decisions as it gains experience from searching for optimal device parameters associated with desired specifications. We can formally define the objective function for the automated design of analog circuits as follows:

number

[0033]

number

[0034]

number

[0035] Before each policy iteration, multiple episodes of interactions need to be performed for policy update, and the interaction records are stored in the memory component as described above. For model parameter updates, the episode experience is used several times for stable learning. Training stops according to one predefined criterion, for example, when the averaged episode reward is above 8 or when the design goal of 30% can be reached within the maximum episode length (which is typically empirical). Deployment Methods

[0036] Once the MARL model is fully trained, the system containing the trained MARL model is configured to perform design tasks associated with new design goals. In the deployment phase, given new specifications sampled from the specification range covered in the training phase, all agents are configured to strive to pursue the optimal solution as fast as possible. Once all specifications are met, it means that this design task is successfully performed by this automated design method. The metric of MARL capability is to calculate the percentage of automated design tasks that are accomplished. Design Example

[0037] As an introduction, we will explain how the automated design of a three-cell non-uniform distributed power amplifier using the proposed MARL algorithm is achieved. Figure 5 shows an example of specifications (scattering parameters, i.e., S-parameters) for a three-cell non-uniform distributed power amplifier according to some embodiments of the present disclosure. The second column of Figure 5 lists the specifications in metrics of all SP simulation terms. By convention, a large-signal simulation such as Harmonic Balance (HB) should be performed on the RF power amplifier, but this is replaced by the small-signal simulation SP simulation simply due to its shorter execution time. To ensure that the SP measurement can reflect the actual HB measurement, it is suggested to set the SP specifications a little more restrictive than the corresponding HB. The empirical setting comes from the fact that the small-signal simulation is typically more optimistic than the large-signal simulation since it ignores significant parasitic effects that may dominate in some devices.

[0038] Figure 6 shows a histogram of how well the specifications are met after sufficient training on MARL, compared to the defined specifications in Figure 5.

[0039] Figure 7 shows a comparison of training results using single-agent RL and multi-agent RL for a three-cell NDPA.

[0040] FIG. 8 is a schematic diagram of a system 1100 including a processor 1120, a memory 1130 coupled to storage 1140 for storing computer-implemented methods, and an interface 1105 in accordance with some embodiments of the present disclosure.

[0041] FIG. 8 shows a block diagram illustrating an example of a system 1100 for automated construction of artificial neural network architectures. The system 1100 includes a set of interfaces and data links 1105 configured to send and receive signals, at least one processor 1120, a memory (or a set of memory banks) 1130, and a storage 1140. The processor 1120, in association with the memory 1130, executes computer-executable programs and algorithms stored in the storage 1140. The set of interfaces and data links 1105 may include a human machine interface (HMI) 1110 and a network interface controller 1150. The processor 1120 may execute computer-executable programs and algorithms in association with the memory 1130 uploading the computer-executable programs and algorithms from the storage 1140. The computer executable programs and algorithms stored in storage 1140 may be pre-trained RL agents (neural networks) 1141, graph neural networks 1142, fully-connected neural networks 1143, instructions (programs) for performing computer-implemented methods 1144, temporary cache 1145, device parameters 1146, and environmental modules 1147.

[0042] The device parameters may be the structure of the transistor. For example, the device parameters may include gate width, gate length, transistor finger count, and capacitance. The desired specifications may be gain (G), bandwidth (B), phase margin (PM), power consumption (P), power efficiency (E), and output power (P).

[0043] The system 1100 can receive signals via a set of interfaces and data links, which may be a data set of training data, desired circuit description data, and test data including gain, bandwidth, phase margin, power consumption, output power, and power efficiency.

[0044] At least one processor 1120, in association with an interface and memory bank 1105, is configured to inject signals and data sets into the reconfigurable DNN block 1141. The system 1100 may receive circuit data or perform circuit simulations with an associated computing system 1195B via a network 1190 and a set of interfaces and data links 1105. Comparison with single-agent RL methods

[0045] There are several emerging related mechanisms that show that RL-based automated design methods can learn strategies in both analog and RF circuit design. However, not all of these prior mechanisms implement more complex circuit topologies with a large number of devices, or the ability to handle more complex frequency-related and signal amplitude-related correlations, which are common challenges in RF circuit design due to ubiquitous parasitics. In addition, circuits with scalable structures, such as distributed power amplifiers, have never been touched by intelligent algorithms, even though their optimal design has been difficult for many years since their invention. Our MARL method is a completely new and well-suited candidate to model the scalability of such topologies. The intuition is that compact individual agents acting as local state observers and learners can focus more on local complexity, thus enabling more efficient learning than traditional single-agent equivalents, where a huge approximator must capture the whole story of a given topology, causing redundancy. More importantly, the coupling effects between agents are adequately captured by the high-level agent networking aspects (graphical mapping in the proposed embodiment), which, although abstract, is effective in estimating direct or indirect coupling effects and is consistent with the design theory.

[0046] Our framework also demonstrates good generalization ability, i.e., generating optimal design parameters for unseen targets. We believe that our method can help the IC industry accelerate analog chip design by using artificial agents that can accumulate large amounts of circuit optimization experience through continuous training.

[0047] Also, individual embodiments may be described as a process, which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may occur in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not described or included in the drawings. Moreover, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to the function returning to the calling function or to the main function.

[0048] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. The manual or automatic implementation may be performed or at least assisted through the use of a machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0049] The above-described embodiments of the present disclosure may be implemented in any of many ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether the processors are provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit having one or more processors within an integrated circuit component. However, the processor may be implemented using circuits in any suitable format.

[0050] Also, the various methods or processes outlined in this specification may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0051] Also, the embodiments of the present disclosure may be embodied as a method, an example of which is provided. The operations performed as part of the method may be ordered in any suitable manner. Thus, while an example embodiment shows sequential operations, embodiments may be constructed in which operations are performed in a different order than illustrated, including performing some operations simultaneously. Also, the use of ordinal terms such as "first," "second," etc. in the claims to modify claim elements does not, in itself, imply any priority, precedence, or order of a claim element relative to another claim element, or any chronological order in which the operations of the method are performed, but is merely used as a label to distinguish a claim element having a certain name from another element having the same name (except for the use of ordinal terms).

[0052] Although the disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is therefore the object of the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the disclosure.

Claims

1. 1. A computer-implemented method for training a multi-agent reinforcement learning (RL) network to generate device parameters for a circuit, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the steps including: obtaining inputs relating to desired circuit specifications for a circuit, device parameters of the circuit, a topology of the circuit, and a final state corresponding to a maximum step, the desired circuit specifications being expressed in terms of gain, bandwidth, phase margin, power consumption, output power, and power efficiency, or a combination thereof, each of the multi-agent RL networks being configured to perform a plurality of steps, the plurality of steps comprising: sending a selected action from a set of actions to an environment module containing a netlist of said circuit; and updating the device parameters of the circuit relative to the desired circuit specification using a data processor of the environment module in accordance with the selected action, the action changing each of the device parameters by the atomic value of each device parameter, the steps further comprising: obtaining a current circuit specification for the circuit based on the updated device parameters by simulating the netlist for the circuit using a circuit simulator of the environment module; and obtaining a reward from the environmental module, the reward being calculated based on a difference between the current circuit specification and the desired circuit specification, the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold or until some of the steps reach a preset value, the steps further comprising: storing the updated device parameters in the memory.

2. 2. The method of claim 1, wherein the circuit is modeled by a graph G(V,E), where each node V is represented by a device and edges E represent connections between devices.

3. 2. The method of claim 1, wherein the power supply voltage (VP), ground (VGND), and other DC bias voltages in the topology of the circuit are represented as redundant nodes V.

4. The method of claim 1 , wherein the reward is calculated by a weighted sum of the desired specifications by assigning different values ​​of the weighting coefficients.

5. The method of claim 1 , wherein the multi-agent RL network includes a graph neural network (GNN) and a fully connected neural network (FCN).

6. The method of claim 5 , wherein the FCNN embeds the desired specifications of the circuit and the GNN embeds the topology and parameters of the circuit.

7. 1. A computer-implemented method for generating device parameters for a circuit using a pre-trained multi-agent reinforcement learning (RL) network, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, perform steps of the method, the steps including: obtaining input via an interface regarding desired circuit specifications, device parameters, and topology of said circuit; providing said inputs to said pre-trained multi-agent RL network, each of said desired circuit specifications being represented by gain, bandwidth, phase margin, power consumption, output power, and power efficiency, said steps further comprising: generating a circuit represented by a graph that models the topology of the circuit, the device parameters of the circuit, and updated device parameters of the circuit.

Images