RF Analog Circuit Electronic Design Automation Method Based on GCN and Deep Reinforcement Learning

Abstract

A computer-implemented method is provided for generating equipment parameters of a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCNN). The method is performed by steps including obtaining inputs related to a set of desired specifications or one desired specification, equipment parameters, and a fixed topology of the circuit, and providing the inputs to the trained reinforcement learning (RL) agent. The description of the desired circuit includes a graph that models the topology of the circuit and the equipment parameters of the circuit, and the desired specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. The trained reinforcement learning (RL) agent performs steps including sending an action selected from a set of actions to an environment module, updating the equipment parameters of the circuit according to the selected action using a data processor of the environment module, obtaining the current specifications of the circuit by simulating the netlist of the circuit, earning a reward from the environment module, and generating updated equipment parameters of the circuit.

Description

[Technical field]

[0001] The present invention relates generally to analog circuit design, and more particularly to radio frequency circuit design automation techniques. [Background technology]

[0002] Analog circuits are the key to bridging the physical world and the modern digital information world. Unlike digital circuits, which follow standard time-efficient flows enabled by electronic design automation tools, analog circuits require significant manual effort and lack effective design automation techniques at all stages. Summary of the Invention [Problem to be solved by the invention]

[0003] The pre-layout design of analog circuits can be expressed as a parameter-to-specification (P2S) optimization problem, whose objective is to find optimal device parameters (e.g., transistor width and finger count) that meet desired specifications (e.g., power and bandwidth) based on a selected circuit topology. Such problems are very difficult. First, it is necessary to search for optimal parameters of various devices in a huge design space. This complexity increases exponentially as the number of design parameters and desired specifications increases. Second, the practical interrelationship between device parameters and circuit specifications becomes complicated depending on multiple factors such as the topology of the circuit and the coupling of specifications. The lack of precise rules to follow makes the search even more difficult. Traditionally, human designers must manually derive device parameters from circuits using their experience and knowledge. However, it is burdensome to manually create empirical formulas, and due to the simplified physical models of devices and circuits, the derived parameters are often not immediately reliable, and tens and hundreds of iterations are required to ensure design accuracy.

[0004] Therefore, it is necessary to develop design automation techniques for designing analog / RF circuits to facilitate the design procedure and reduce the time to market. [Means for solving the problem]

[0005] Some embodiments of the present invention are based on the recognition that the automated design of analog circuits is a long-standing challenge in the field of integrated circuits. The embodiments provide a deep reinforcement learning (RL) framework to assist in the design of analog circuits at the pre-layout level, with the goal of finding device parameters to meet desired specifications. Unlike all conventional methods, the method of the present invention incorporates significant domain knowledge that determines the relationship between device parameters and circuit specifications to achieve the highest level of design accuracy and efficiency. This approach is applicable to the design of various analog circuits (e.g., high-frequency circuits) with different semiconductor technologies, and breaks the limitations of the prior art in the design of a narrow range of low-frequency analog circuits with conventional semiconductor technologies. To achieve such capabilities, the present invention tunes the policy network of the RL agent by appropriately combining a graph neural network (GNN) and a fully connected neural network (FCNN) based on the circuit topology. The proposed policy network can effectively capture the common physical features (e.g., device parameters and interactions) present in the circuit graph using the GNN and extract the specification couplings (e.g., design trade-offs) using the FCNN, so that the relationship between the circuit parameters and the design goals can be optimally modeled. Experimental results show that the method of the present invention achieves 97.5% accuracy in designing an exemplary circuit, with 1.5 times the efficiency of the existing best-performing method. The method of the present invention also has good versatility and portability.

[0006] Furthermore, some embodiments are based on the recognition that machine learning (ML) techniques, such as supervised learning and reinforcement learning, have been recently used as automated methods to improve the design efficiency of analog circuits. By training neural network models, ML methods can cover a vast design space and achieve design goals in far fewer iterations. Despite such significant advantages, most existing ML techniques treat P2S optimization as a black box, severely limiting its robustness, accuracy and efficiency, because they do not consider practical yet very important interrelationships between device parameters and circuit specifications. Moreover, these ML methods are limited to a very narrow scope of low-frequency analog circuits (i.e., operational amplifiers) implemented using conventional complementary metal-oxide-semiconductor (CMOS) technology. These ML methods cannot be easily applied to other types of analog circuits designed with different semiconductor technologies.

[0007] The present invention can provide a reinforcement learning (RL) method for P2S optimization. In this case, a well-trained RL agent can autonomously find optimal device parameters for desired specifications. Unlike all prior art, the method of the present invention achieves the highest design accuracy (97.5%) and efficiency (1.5 times the state-of-the-art) by incorporating domain knowledge that determines the relationship between device parameters and circuit specifications. Even for some failure cases, the decision sequence of the RL agent can provide hints to warm-start manual tuning methods to ensure 100% design accuracy. The agent of the present invention can also generate optimal device parameters for unknown desired specifications. Its superior ability is realized by a tuned policy network including a graph neural network (GNN) and a fully connected neural network (FCNN). The GNN is built based on the topology of a specific circuit and can capture the fundamental physical properties of the circuit, such as device parameters, connections, and interactions. The FCNN extracts the coupling of the circuit specifications. With this unique policy network, our RL agent can include pertinent information about the circuit in the loop and update device parameters by making optimal sequential decisions like an experienced human designer, resulting in the automated design of high-quality analog circuits.

[0008] The method of the present invention also breaks the limitation of the prior art of designing only low-frequency analog circuits using conventional complementary metal-oxide semiconductor (CMOS) technology. The method can design a variety of analog circuits in different semiconductor technologies by utilizing device physical features common to a wide range of semiconductor technologies to encode the device state. Specifically, the method can be used to design radio frequency (RF) circuits in gallium nitride (GaN) technology, which is a promising alternative to conventional CMOS technology and high frequency and power electronic applications. RF circuits are a subclass of analog circuits that specially handle high frequency signals, which require more advanced analysis and electromagnetic characterization in addition to the design challenges faced by low frequency analog circuits. The method of the present invention can significantly reduce training time and accelerate the design of RF circuits by leveraging a fast reward signal in the deployment phase. Our research shows that by combining RL techniques with domain knowledge, we may approach a future in which artificial agents with extensive circuit optimization experience assist circuit designers.

[0009] Some embodiments of the present invention present a deep RL framework that can automatically design both low-frequency and RF analog circuits in different implementation technologies, i.e., CMOS and GaN. Previous works either use supervised learning or focus on designing a narrow range of low-frequency analog circuits in CMOS technology.

[0010] Furthermore, some embodiments are based on the realization that by appropriately combining the RL agents GNN and FCNN, a policy network architecture is constructed. This policy network can comprehensively refine the complex relationships between device parameters and circuit specifications. Thus, this agent can achieve faster and better circuit optimization.

[0011] Furthermore, some embodiments of the present invention are based on the realization that the proposed RL framework has the ability to generalize, i.e., a trained RL agent can leverage previously learned experiences to optimize device parameters for new, unknown design specifications. Previous works lack such generalizability.

[0012] One embodiment shows that the trained RL agent achieves a state-of-the-art accuracy of 97.5% to obtain optimal device parameters for randomly given design specifications during the deployment phase. For a very small number of failure cases, the decision sequence of the RL agent can provide hints to warm-start manual tuning methods.

[0013] According to some embodiments of the present invention, a computer-implemented method is provided for generating device parameters of a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCNN). The method uses a processor connected to a memory storing instructions implementing the method, which when executed by the processor perform the steps of the method. The steps of the method include obtaining inputs related to a set or one desired specification of the circuit, device parameters, and a fixed topology of the circuit via an interface, and providing the inputs to the trained reinforcement learning (RL) agent, where the description of the desired circuit includes a graph that models the topology of the circuit and the device parameters of the circuit, and the desired specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. For example, a trained reinforcement learning (RL) agent is configured to perform the following steps, including sending an action selected from a set of actions to an environment module; and updating, using a data processor of the environment module, equipment parameters of the circuit to desired circuit specifications according to the selected action, the action modifying each of the current equipment parameters by an atomic value for each equipment parameter; obtaining, using a circuit simulator of the environment module, a netlist of the circuit based on the updated equipment parameters, the environment module including the netlist of the circuit; obtaining a reward from the environment module, the reward calculated based on a difference between the current specifications and the desired specifications; the sending, updating, obtaining, and obtaining steps are continued until the reward reaches a threshold or a number of steps reaches a predetermined value; and generating updated equipment parameters of the circuit.

[0014] Embodiments of the present disclosure will now be further described with reference to the accompanying drawings, in which the drawings are not necessarily drawn to scale, emphasis instead being placed upon illustrating the principles of embodiments of the present disclosure. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram outlining a RL framework for automated design of analog circuits in accordance with an embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic diagram illustrating a mapping between circuit topology and a graph, and a policy network architecture based on GNN-FC tuned for analog circuit design, according to an embodiment of the present invention. [Figure 3A] FIG. 1 is a schematic diagram showing an RF polar TX in 150 nm GaN technology according to an embodiment of the present invention (D1-D5 and DF are driver stages). [Figure 3B] FIG. 2 illustrates a design space of device parameters and a sampling space of desired specifications for two benchmarks in accordance with an embodiment of the present invention. [Figure 4A] FIG. 1 illustrates the evolution of average episode reward for a RL agent, in accordance with an embodiment of the present invention. [Figure 4B] FIG. 1 illustrates the evolution of the average episode length of an RL agent, according to an embodiment of the present invention. [Figure 4C] FIG. 1 illustrates the evolution of deployment accuracy of an RL agent in accordance with an embodiment of the present invention. [Figure 5A] FIG. 1 illustrates the evolution of average episode reward for a RL agent, in accordance with an embodiment of the present invention. [Figure 5B] FIG. 1 illustrates the evolution of the average episode length of an RL agent, according to an embodiment of the present invention. [Figure 5C] FIG. 1 illustrates the evolution of deployment accuracy of an RL agent in accordance with an embodiment of the present invention. [Figure 6A]FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 6B] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 6C] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 6D] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 7A] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 7B] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8A] FIG. 1 illustrates a generic example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8B] FIG. 1 illustrates a generic example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8C] FIG. 1 illustrates a generic example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8D] FIG. 1 illustrates a generic example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 9A] FIG. 1 illustrates a generic example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 9B] FIG. 1 illustrates a generic example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 10] 1A-1D illustrate a comparison of different design automation methods according to an embodiment of the present invention. [Figure 11] 1 is a schematic diagram illustrating a system including a processor, a memory storing a computer-implemented method, and an interface according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The following description provides exemplary embodiments only and is not intended to limit the scope, application, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the subject matter as set forth in the appended claims.

[0017] In the following description, specific details are given to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagrams so as not to obscure the embodiments in unnecessary detail. Also, well-known processes, structures, and techniques may be shown without unnecessary detail so as not to obscure the embodiments. Moreover, like reference numbers and names in the various drawings refer to like elements.

[0018] Design automation of analog / RF circuits can be best formulated as a parameter-to-specification (P2S) optimization problem. The goal is to find optimal device parameters (e.g., transistor width and length) that satisfy desired circuit specifications (e.g., power, speed, and bandwidth). Traditionally, these device parameters are derived manually based on IC designers' experience and knowledge of the circuit. Such empirical approaches aim to link circuit specifications and device parameters in equations through simplified physical models of the devices and circuits. Due to the high nonlinearity of analog / RF circuits, manually porting these equations is labor-intensive. It also requires dozens of iterations to ensure design accuracy.

[0019] Various design automation techniques have been proposed for the P2S optimization problem of analog circuits and are generally classified into two categories: optimization-based methods and learning-based methods. Bayesian optimization, geometric programming, genetic algorithms, and simulated annealing are representative optimization-based methods. For all these methods, the search for optimal device parameters is guided by an algorithm. At each step, intermediate circuit specifications obtained from a standard simulator are compared with the design goals. The algorithm then decides how to update the device parameters for the next search. These techniques differ from each other in sampling efficiency, convergence rate, and overall optimality. However, a key problem is that if the desired specifications change, the algorithm must be started again from the beginning. The method of the present invention can reach different target specifications without systematic retraining.

[0020] Recently, learning-based methods have emerged. Supervised learning methods have been used to learn the complex relationships between device parameters and circuit specifications. For example, FCNN is trained to design a two-stage operational amplifier (op-amp), where the input of the network is the circuit specifications and the output is the predicted device parameters. These methods cannot guarantee the design accuracy because of the inherent approximation error. RL methods have been used to learn the optimal policy from the state space of the desired specifications to the action space of the device parameters. With sufficient exploration in the training phase, high design accuracy can be achieved. For example, RL methods can learn the optimal policy from the state space of the desired specifications to the action space of the device parameters. 4 After exploring these environmental steps, optimal device parameters can be found that meet the desired specifications of the two-stage op-amp with better than 90% accuracy.

[0021] These learning-based methods barely consider the domain knowledge (i.e., circuit topology and specification coupling) that governs the relationship between device parameters and circuit specifications. Supervised learning methods use FCNN to learn a static mapping between desired specifications and device parameters without relying on circuit topology. Similarly, RL methods use a simple FCNN as a policy network, ignoring the circuit graph and specification coupling. Another RL method builds a policy network on a graph convolutional network (GCN) to solve a different problem, namely, optimizing the figure of merit (FoM) of a circuit. Nevertheless, GCN ignores the circuit specification coupling and captures only the circuit topology, which does not include the power and basis nodes that are essential parts of the circuit graph. These methods are limited to designing low-frequency analog circuits, e.g., op-amps, without unraveling the nature of the circuit's graph. The main reason is that the electromagnetic coupling of devices in RF circuits strongly depends on the circuit's topology and significantly affects the specifications.

[0022] Moreover, these learning-based methods focus only on the design of CMOS circuits. This limitation is caused by using technology-dependent features of CMOS devices, such as saturation voltage and bulk voltage, to encode the transistor state vectors, which are not applicable to all semiconductor technologies. For example, GaN is a III / V semiconductor and has a different device model than CMOS. The method described in this disclosure achieves design automation for a wide range of analog circuits (including both low-frequency analog circuits and RF circuits) implemented in different semiconductor technologies (including CMOS and GaN) by incorporating domain knowledge of analog circuit design and using technology-independent features to encode the device states.

[0023] Despite these significant advantages, the best existing ML approaches treat P2S optimization as a black box without considering practical, yet crucial, interrelationships between device parameters and circuit specifications, severely limiting their robustness, accuracy, and efficiency. Moreover, these ML methods are limited to a very narrow scope of low-frequency analog circuits (i.e., operational amplifiers) implemented using conventional complementary metal-oxide-semiconductor (CMOS) technology. These ML methods cannot be easily applied to other types of analog circuits designed in different semiconductor technologies.

[0024] Our method is inspired by human design ideas and can use a combination of GNN and FCNN to comprehensively refine the complex relationships between device parameters and circuit specifications. None of the prior art encompasses such practical design ideas and has achieved our capabilities. Autockt [Setalluri et.al., 2020] uses a simple FCNN to design a policy network with device parameters and desired specifications as input, but ignores the graph nature of the circuit and the coupling of specifications. GCN-RL circuit designers optimize the figure of merit (FoM) of a particular circuit by building a policy network on a graph convolutional network (GCN) based on the circuit topology. However, GCN only partially captures the circuit topology without considering the power nodes, which are in fact an essential part of the circuit graph. GCN also ignores the coupling of circuit specifications. Furthermore, the present invention aims to realize design automation of a wide range of analog circuits (including both low-frequency analog circuits and RF circuits) implemented in different semiconductor technologies (including CMOS and GaN) by using technology-independent features to encode the state of the device. Learning with Graph Neural Networks

[0025] Graph neural networks, such as graph convolutional networks (GCNs) and graph attention networks (GATs), are emerging neural networks that operate directly on graph-like non-Euclidean data structures. They are gaining increasing popularity in various domains, including social networks, knowledge graphs, recommendation systems, and life sciences. The present invention utilizes GCNs and GATs to incorporate the physical properties of a particular circuit, such as device parameters, connections, and interactions, into the policy network of the present invention. The present invention shows that GATs with a multi-headed attention mechanism can better model the subtle physical properties of a circuit than GCNs. problem statement

[0026] This disclosure provides design automation for analog circuits at a pre-layout level, with the goal of efficiently searching for optimal device parameters that meet desired specifications. This disclosure addresses the scenario where the topology of the analog circuit is fixed. This is practical since many classical topologies are usually available for a particular design task.

[0027] The outline of the present invention, an RL framework for automatic design of analog circuits, is as follows. The RL agent is based on the actor-critic method. The environment consists of a netlist of an analog circuit with a specific topology, a circuit simulator, and a data processor. At each time step i, the agent automatically performs an action a i and outputs the state s from the environment. i and reward r i Update the equipment parameters according to the above. Reinforcement Learning Framework

[0028] Some embodiments use deep reinforcement learning (RL) techniques to achieve design automation of analog circuits. Some embodiments of the present invention may provide a computer-implemented method for generating device parameters of a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCNN). The method uses a processor coupled to a memory storing instructions implementing the method. The memory is configured to store a trained RL agent, a GNN, and an FCNN. The instructions, when executed by the processor, perform steps of the method. The steps of the method include obtaining inputs related to a set of desired specifications or one desired specification of the circuit, device parameters, and a fixed topology of the circuit via an interface, and providing the inputs to the RL agent, where the description of the desired circuit includes a graph that models the topology of the circuit and the device parameters of the circuit, and the desired specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. For example, the trained RL agent may be configured to perform the following steps, including sending an action selected from a set of actions to an environment module; and updating, using a data processor of the environment module, equipment parameters of the circuit to a desired circuit specification according to the selected action, the action modifying each of the current equipment parameters by an atomic value for each equipment parameter; obtaining, using a circuit simulator of the environment module, a netlist of the circuit based on the updated equipment parameters, the environment module including the netlist of the circuit; earning a reward from the environment module, the reward calculated based on a difference between the current specification and the desired specification; the sending, updating, obtaining, and obtaining steps are continued until the reward reaches a threshold or the number of steps reaches a predetermined value; and generating updated equipment parameters of the circuit.

[0029] Further provided is a computer-implemented method for training an RL agent network to generate device parameters of a circuit, the method using a processor coupled to a memory storing instructions for implementing the method, the memory being configured to store the RL agent network, where the processor is configured to execute instructions including obtaining inputs related to desired circuit specifications, device parameters, a fixed topology of the circuit, and a final state corresponding to a maximum step for the circuit. The desired circuit specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. The RL agent network is configured to send a selected action from a set of actions to an environment module, and update, with a data processor of the environment module, the device parameters of the circuit relative to the circuit specifications according to the selected action, the action modifying each of the current device parameters by a unit value for each device parameter. The steps include obtaining a current specification of the circuit by simulating a netlist of the circuit based on the updated device parameters with a circuit simulator of the environment module, the environment module including the netlist of the circuit, the steps include obtaining a reward from the environment module, the reward being calculated based on a difference between the current specification and the desired specification, the steps of sending, updating, obtaining, and obtaining are continued until the reward reaches a threshold or the number of steps reaches a predetermined value, and the steps include generating, via the interface, the filled updated device parameters. Some specific examples of the system are shown in FIG. 11 of the present invention.

[0030] Figure 1 shows a framework 100 that consists of five main elements: an agent 110, i.e., a learner and decision maker (in our case, a policy 112 and a value network 111); an action space, i.e., a set of actions that the agent 110 can take (i.e., increasing, decreasing, or maintaining the parameters of each device); a state space, i.e., a set of possible states that the agent 110 can stay in the environment 120 (i.e., any combination of the device parameters of the circuit in the design space); a reward, i.e., a scalar value (e.g., the difference between the current specification and the desired specification) returned by the environment 120 if an action is taken in a certain state (this is used to evaluate and guide the actual behavior of the agent); and an environment 120, i.e., the physical world in which the agent operates (i.e., a high-fidelity circuit simulation environment 122). A data processor unit 123 is also included in the environment. A netlist of a particular analog circuit 121 is used as input for the circuit simulator 122. The parameters of the circuit being designed are updated according to instructions for action from the data processor unit onwards.

[0031]

number

[0032]

number

[0033] Since all N desired specifications are equally important, but not required, the same weight is assigned to each normalized difference, and the designer may assign different weights to different specifications.

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038] 2 is a schematic diagram illustrating a mapping between circuit topology and graphs and a GNN-FC-based policy network architecture tuned for analog circuit design in accordance with some embodiments of the present invention. The diagram illustrates a mapping of a circuit topology 210 to a graph neural network (GNN) 211 and a GNN-FC-based policy network 200 architecture tuned for analog circuit design. In this case, a two-stage op-amp, indicated at 210, is used as an example. State Representation

[0039]

number

[0040] The environment 120 of the present invention consists of a netlist 121 of a particular circuit, a high-fidelity circuit simulator 122 (e.g., Cadence Spectre or Advanced Design System (ADS)), and a data processor 123. The current specifications of the circuit are obtained by simulating the netlist with the circuit simulator. The data processor 123 then feeds back rewards to the agent 110 by processing the simulation results. The data processor also updates device parameters and rewrites the circuit netlist 121 according to actions from the agent 110.

[0041] The present invention uses an actor-critic method with a critic block (111) and an actor block (112) to construct an agent 110. In order to train an agent that can accurately optimize device parameters to satisfy a desired specification 201, we propose a neural network architecture of a policy network 200 shown in FIG. 2. This neural network architecture is composed of a graph neural network (GNN) 211 and a fully connected neural network (FCNN) 202, and is called a policy network based on GNN-FC. The role of the GNN 211 is to extract physical information about the types and connections of nodes in a circuit graph into a low-dimensional vector embedding 212. Specifically, a graph convolutional network is used to extract such physical features. In addition, the FCNN 202, which takes the design goals and embeddings 203 as input, is used to extract their coupling relationships, for example, design trade-offs between different specifications. The final FC layer 220 then updates the behavior by concatenating and further processing the graph embeddings 212 and the FCNN embeddings 203. The value network 111 maintains the same architecture as the policy network 112, except for the last layer, which outputs an estimate of the expected reward for executing the current policy.

[0042] Our goal is to have the RL agent (i.e., policy network 112) generate higher quality decisions as it gains experience from searching for optimal device parameters associated with the desired specifications. The objective function for the automated design of analog circuits can be formally defined as follows:

number

[0043]

number

[0044] We evaluate the proposed RL framework using two representative analog circuits. First, we select the two-stage opamp in Figure 2 as an exemplary low-frequency analog circuit because this opamp has been adopted as a standard benchmark by many optimization-based and learning-based methods. This opamp is implemented in 45nm CMOS technology.

[0045] Next, we choose the RF polar transmitter (polar TX) of Figure 3A implemented in 150 nm GaN technology as an example of a high frequency analog circuit. This polar TX consists of five stages of drivers D1, D2, D3, D4 and D5 shown at 310, 320, 330, 340 and 350, and a final stage DF360 shown in detail in circuit diagram 361. Circuit diagram 311 is a schematic diagram showing the driver stages with the same topology as D1-D5. The input is a differential signal V in_p1 301 and V in_p2 302. The output port is V out The number is 370.

[0046] FIG. 3B shows the design space of device parameters and the sampling space of desired specifications for the two circuits. The two-stage opamp has a total of 15 parameters, and the RF polar TX has a total of 14 parameters. Note that 1) the two transistors in the differential pair of the opamp have exactly the same parameters, and 2) the devices in each driver stage D1-D5 of the RF polar TX also have the same parameters. These are practical constraints imposed by the circuit design.

[0047] A separate RL agent (110) is trained to enable the automatic design of each circuit. In our experiments, each training of a two-stage opamp takes about 3 hours of CPU time, while each training of an RF polar TX takes about 12 hours of CPU time. RF circuits require more sophisticated simulations and therefore more time to obtain training rewards. To demonstrate the unique advantages of our GNN-FC based policy network, we use the FCNN based policy network as a baseline, which does not consider the physical topology of the circuit. All our experiments were performed on an 8-core Intel CPU. Agent Training

[0048] The episode length for the two-stage opamp RL agent (PolarTX RL agent) is set to 50 (30). The total episodes used to train the two RL agents are 3.5 × 10 4 and 3.5 × 10 3 The RF Polar TX requires fewer episodes for training because it has fewer device parameters, fewer target specifications and a smaller design space. Figures 4A, 4B and 4C are curves showing the average episode reward, average episode length and deployment accuracy, respectively, of the RL agent with different policy network architectures for designing a two-stage opamp during training.

[0049] 5A, 5B, and 5C are curves showing the average episode reward, average episode length, and deployment accuracy, respectively, of the RL agent with different policy network architectures for designing PolarTX during training.

[0050] As can be seen from Figures 4A and 5A, the more episodes used for training, the higher the average episode reward obtained from each policy network. However, the proposed GNN-FC based policy network can achieve higher reward than the normal FCNN based policy network. The average episode length of each policy network in Figures 4B and 5B is also shorter for training episodes. The policy network of the present invention shows a shorter (faster) average episode length compared to the baseline policy network. Also, Figures 4C and 5C show the deployment accuracy of the two policy networks for training episodes. Policy deployment means applying a trained policy network to automatically find device parameters for a given desired specification. Each point in the right column of Figures 4C and 5C is obtained from the statistics of deploying each policy network for 200 sets of randomly sampled desired specifications. This comparison shows that the proposed GNN-FC based policy network has higher deployment accuracy for both circuit designs compared to the baseline FCNN based policy network. The proposed policy network achieves the state-of-the-art 97.5% policy deployment accuracy. These results suggest that the proposed GNN-FC based policy network is more efficient and accurate for analog circuit design by combining both the physical circuit features via the GNN part and the coupling between design specifications via the FC part. Automating circuit design by policy deployment

[0051] The present invention is a method to reduce the number of predefined episodes (3.5×10 for a two-stage op amp) 4 pcs, and for Polar TX, 3.5 x 10 4We show how it can be applied to analog circuit design automation by deploying a policy network based on GNN-FC trained with a set of randomly sampled desired specifications. Figures 6A, 6B, 6C, and 6D each show an example of such a two-stage op-amp. In this case, the RL agent automatically searches for optimal device parameters from an initial state (step 0) for a circuit with a set of randomly sampled desired specifications. The desired specifications for the sampled two-stage op-amp are gain (G=350) in Figure 6A, power consumption (P=4×10) in Figure 6B, and power consumption (P=4×10) in Figure 6C. -3 W), phase margin (PM = 55°) in Figure 6C, and bandwidth (B = 1.8 × 10 7 Hz). Note that the lower the power consumption, the better the performance. As shown in Figures 7A and 7B, the sampled desired specifications for the RF polar TX are output power (W) and power efficiency (E). These sampled specifications are represented by horizontal dashed lines in Figures 6A-6D and Figures 7A and 7B. It can be seen that initially, the current specifications (y-axis in each figure) are generally far from the desired specifications. As the deployment continues, the RL agent follows the path guided by the trained policy to find the optimal device parameters for a particular specification, so that the current specifications move closer to the desired specifications. The RL agent requires 27 steps and 11 steps to achieve the design goal for each circuit.

[0052] An interesting phenomenon is that once a specification is first achieved, the RL agent does not over-optimize that specification, but instead tries to optimize those that have not yet been reached. For example, first, in the 14th deployment step, the gain of the two-stage opamp is realized. In the following steps, the RL agent focuses on optimizing the specifications that have not yet been achieved, namely, phase margin and bandwidth. Similarly, when deploying an RF polar TX, the output power is reached first, and then the RL agent continues to optimize the power efficiency. Such a property enabled by the RL method is very promising, since traditional manual tuning lacks the ability to well balance the trade-offs between all the desired specifications during optimization. We also analyzed several failure cases where the trained policy cannot converge to the optimal device parameters. In these failure cases, we observed that some specifications can converge to the vicinity of the desired specifications, but then deviate a little from the goal. Fortunately, we found that the design goal can be easily achieved by slightly adjusting the device parameters manually at that particular step. These results suggest that IC designers can benefit significantly from trained policies even if automated deployment fails, by using them as an efficient warm start for manual tuning. Detailed analysis and further demonstration of results are provided in the supplementary materials. Generalization to unknown specifications

[0053] Furthermore, we evaluate the generalization ability of the GNN-FC based policy network. That is, we deploy it with unknown specifications outside the sampling space shown in Fig. 3B. For a two-stage op-amp, the set of sampled unknown specifications is the gain (G = 225), bandwidth (B = 2.6 × 10 7 Hz), phase margin (M=65°), and power dissipation (P=6×10 -3The power consumption is 1.5 W. For RF polar TX, the set of sampled unknown specifications is output power (P = 2.9 W) and power efficiency (E = 69%). Figures 8A-8D, 9A and 9B show the deployment trajectories, with horizontal dashed lines indicating these unknown specifications. Compared with the policy deployment shown in Figures 6A-6D and Figures 7A and 7B (where the specifications are sampled from the training space in Figure 3A), deployment to unknown desired specifications usually requires more search steps. For example, 11 steps are sufficient for the normal deployment of RF polar TX in Figures 7A and 7B, whereas generalization of RF polar TX requires 49 steps to achieve the goal. This difference is due to the unknown desired specifications being beyond the range of the training dataset, so more steps are required to reach the optimal parameters. However, we found that the policy network based on FCNN as the baseline lacks generalization ability for most unknown specifications, even with a large number of search steps. The generalization of the proposed policy network can capture a rich representation of the state space (e.g., physical topology and coupling relationships between specifications) so that during inference, the experience learned from previous states can be leveraged to new unknown states. Comparison with other methods

[0054] Finally, we compare the efficiency of our method with previous optimal tuning methods. The comparison results are shown in Figure 10. In the deployment phase (inference phase), our RL method outperforms Bayesian optimization. In general, Bayesian optimization requires more steps to find optimal device parameters for the same desired specifications compared to our method. Compared to previous RL methods (K. Settaluri, A. Haj-Ali, Q. Huang, K. Hakhamaneshi, B. Nikolic; AutoCkt: Deep Reinforcement Learning of Analog Circuit Designs, 2020 Design, Automation & Test in Europe Conference Exhibition (DATE), pages 490-495, 2020), our method is more efficient with fewer deployment steps due to its unique policy network architecture. Finally, our method has generalization capabilities that have not been demonstrated in other methods. In summary, the proposed method has shown progress in efficiency, accuracy, and generalization.

[0055] We also compare our method with another work (H. Wang, K. Wang, J. Yang, L. Shen, N. Sun, HS Lee, S. Han; GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learning, 2020 57th ACM / IEEE Design Automation Conference (DAC), pages 1-6, 2020) in optimizing the figure of merit (FoM) of a two-stage operational amplifier. This work is technically different from our work in many ways, such as RL technique (DDPG vs. PPO), state space (technology-dependent features of the device vs. technology-independent features of the device), and action space (continuous vs. discrete). To make a fair comparison, we build a baseline using the PPO technique and discrete action space, similar to our work, but use GCN as the policy network and FoM as the reward function. Furthermore, we compare the accuracy between our method and supervised learning methods (not RL) in predicting device parameters for a given specification. For comparison, we first train an FCNN. The input is the desired specifications, and the output is the corresponding device parameters. The desired specifications are then randomly fed and used for prediction, and the correctness is verified by circuit-level simulation. The comparison results show that the method of the present invention has a much higher accuracy (98% vs. 87%).

[0056] Thus, some embodiments of the present invention can provide a deep reinforcement learning (RL) based framework for automated design of analog circuits. A key feature of the present framework is the ability to capture both the underlying physical topology and the trade-offs between specifications of a particular circuit into a newly proposed composite GNN-FC based policy network. Such a network architecture is superior to other network architectures that mainly capture physical features but fail to adequately incorporate the relationship between physical topology and specification trade-offs. The present network architecture can design both high quality low frequency and RF circuits in different implementation technologies. In particular, the present invention shows that the design of RF circuits can be accelerated compared to traditional optimization methods. The present framework also shows good generalization ability, i.e., the ability to generate optimal design parameters for unknown targets. It is believed that the present method can help the IC industry accelerate analog chip design using artificial agents that can accumulate vast circuit optimization experience through continuous training.

[0057] FIG. 11 is a schematic diagram illustrating a system 1100 configured to include a processor 1120, a memory 1120 connected to storage 1140 that stores computer-implemented methods, and an interface 1105 in accordance with some embodiments of the present disclosure.

[0058] FIG. 11 is a block diagram illustrating an example of a system 1100 for automated construction of artificial neural network architectures, according to some embodiments of the present disclosure. The system 1100 includes a set of interfaces and data links 1105 configured to send and receive signals, at least one processor 1120, a memory (or a set of memory banks) 1130, and a storage 1140. The processor 1120, in association with the memory 1130, executes computer-executable programs and algorithms stored in the storage 1140. The set of interfaces and data links 1105 may include a human machine interface (HMI) 1110 and a network interface controller 1150. The processor 1120 may execute computer-executable programs and algorithms in association with the memory 1130 uploading the computer-executable programs and algorithms from the storage 1140. The computer-executable programs and algorithms stored in storage 1140 may be trained RL agents (neural networks) 1141, graph neural networks 1142, fully connected neural networks 1143, instructions for performing computer-implemented methods (programs) 1144, temporary cache 1145, device parameters 1146, and environmental modules 1147.

[0059] The system 1100 can receive signals via a set of interfaces and data links. The signals may be a data set consisting of training data, desired circuit description data, and test data including gain, bandwidth, phase margin, power consumption, output power, and power efficiency.

[0060] At least one processor 1120, in association with an interface and memory bank 1105, is configured to inject signals and data sets into the reconfigurable DNN block 1141. The system 1100 can receive circuit data and perform circuit simulations with an associated computing system 1195B via a network 1190 and a set of interfaces and data links 1105.

[0061] Also, each embodiment may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart describes operations as a sequential process, many operations may be performed in parallel or simultaneously. Also, the order of operations may be changed. A process may be terminated when the operations of a process are completed, but the process may include additional steps not discussed or shown. Moreover, not all operations within a specifically described process need be included in every embodiment. A process may be a method, a function, a procedure, a subroutine, a subprogram, etc. When a process is a function, the termination of the function corresponds to the return of the function to the calling function or to the main function.

[0062] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. The manual or automated implementation may be implemented, or at least assisted, by machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium. A processor may perform the necessary tasks.

[0063] The embodiments of the present disclosure described above may be implemented in many ways. For example, the embodiments may be implemented in hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether located on a single computer or distributed across multiple computers. Such a processor may be implemented as an integrated circuit. An integrated circuit element may include one or more processors. However, the processor may be implemented in any suitable circuit.

[0064] Also, the various methods or steps outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Moreover, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed in various embodiments as desired.

[0065] The embodiments of the present disclosure may be embodied as a method, which is provided as an example. The operations performed as part of this method may be ordered in any suitable manner. Thus, embodiments may be constructed that perform operations in a different order than the operations performed sequentially in the exemplary embodiments, and that may include performing some operations simultaneously. Furthermore, the use of sequence terms, such as first, second, etc., to modify claim elements in the claims does not imply a priority, ante-post or order of one claim element relative to another claim element, or a time order in which the operations of the method are performed, but is merely used as a label to distinguish the claim elements and to distinguish one claim element having a particular name (by using sequence terms) from another element having the same name.

[0066] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other modifications and alterations can be made within the spirit and scope of the disclosure. It is therefore intended in the appended claims to cover all such variations and modifications that are within the true spirit and scope of the present disclosure.

Claims

1. 1. A computer-implemented method for generating device parameters for a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCN), the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, perform steps of the method, the steps including: obtaining, via an interface, inputs relating to a set or a desired specification of a circuit, device parameters, and a fixed topology of the circuit; and providing the inputs to the trained reinforcement learning (RL) agent, the description of the desired circuit comprising a graph that models the topology of the circuit and device parameters of the circuit, the desired specifications including gain, bandwidth, phase margin, power consumption, output power, and power efficiency, the trained reinforcement learning (RL) agent being configured to perform the following steps: sending a selected action from the set of actions to an environment module; and updating, using a data processor of the environment module, the device parameters of the circuit to a desired circuit specification in accordance with the selected action, the action modifying each of the current device parameters by an atomic value of each device parameter; obtaining a current specification of the circuit by simulating a netlist of the circuit based on the updated device parameters using a circuit simulator of the environment module, the environment module including the netlist of the circuit; obtaining a reward from the environmental module, the reward being calculated based on a difference between the current specification and the desired specification, the transmitting, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold or a number of steps reaches a predetermined value; generating updated device parameters of the circuit.

2. The circuit is modeled by a graph G(V,E), Each node V is represented by a device, The method of claim 1 , wherein an edge E represents a connection between devices.

3. 2. The method of claim 1 , wherein the power supply (VP), ground (VGND), and other DC bias voltages of the circuit topology are represented as extra nodes V.

4. The method of claim 1 , wherein the reward is calculated by a weighted sum of all desired specifications by assigning different values ​​of the weighting coefficients.

5. The method of claim 1 , wherein the Graph Neural Network (GNN) can be implemented by different types such as a Graph Convolutional Network or a Graph Attention Network (GNN).

6. the fully connected neural network (FCN) embeds the desired specifications of the circuit; The method of claim 1 , wherein a graph neural network (GNN) embeds the topology and parameters of the circuit.

7. 1. A computer-implemented method for training an RL agent network to generate device parameters for a circuit, the method using a processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the steps comprising: obtaining inputs relating to desired circuit specifications, device parameters, a fixed topology of said circuit, a final state corresponding to a maximum step, said desired circuit specifications including gain, bandwidth, phase margin, power consumption, output power and power efficiency of a circuit, said RL agent network being configured to perform the following steps, which include: sending a selected action from the set of actions to an environment module; and updating, using a data processor of the environment module, the device parameters of the circuit relative to a circuit specification in accordance with the selected action, the action modifying each of the current device parameters by an atomic value of each device parameter; obtaining a current specification of the circuit by simulating a netlist of the circuit based on the updated device parameters using a circuit simulator of the environment module, the environment module including the netlist of the circuit; obtaining a reward from the environmental module, the reward being calculated based on a difference between the current specification and the desired specification, the transmitting, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold or a number of steps reaches a predetermined value; generating, via the interface, updated and filled device parameters.

8. The circuit is modeled by a graph G(V,E), Each node V is represented by a device, The method of claim 7 , wherein the edges E represent connections between devices.

9. 8. The method of claim 7, wherein the power supply (VP), ground (VGND), and other DC bias voltages of the circuit topology are represented as extra nodes V.

10. The method of claim 7 , wherein the reward is calculated by a weighted sum of all desired specifications by assigning different values ​​of the weighting coefficients.

11. The method of claim 7, wherein the Graph Neural Network (GNN) may be implemented by different types such as a Graph Convolutional Network or a Graph Attention Network (GNN).

12. the fully connected neural network (FCN) embeds the desired specifications of the circuit; The method of claim 7 , wherein a graph neural network (GNN) embeds the topology and parameters of the circuit.

Images