RF Analog Circuit Electronic Design Automation Method Based on GCN and Deep Reinforcement Learning

Abstract

A computer-implemented method is provided for generating equipment parameters of a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCNN). The method is performed by steps including obtaining inputs related to a set of desired specifications or one desired specification, equipment parameters, and a fixed topology of the circuit, and providing the inputs to the trained reinforcement learning (RL) agent. The description of the desired circuit includes a graph that models the topology of the circuit and the equipment parameters of the circuit, and the desired specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. The trained reinforcement learning (RL) agent performs steps including sending an action selected from a set of actions to an environment module, updating the equipment parameters of the circuit according to the selected action using a data processor of the environment module, obtaining the current specifications of the circuit by simulating the netlist of the circuit, earning a reward from the environment module, and generating updated equipment parameters of the circuit.

Description

[Technical Field]

[0001] The present invention relates generally to analog circuit design, and more particularly to radio frequency circuit design automation techniques. [Background technology]

[0002] Analog circuits are the key to bridging the physical world and the modern digital information world. Unlike digital circuits, which follow standard, time-efficient flows enabled by electronic design automation tools, analog circuits require significant manual effort and lack effective design automation techniques at all stages. Summary of the Invention [Problem to be solved by the invention]

[0003] Pre-layout design of analog circuits can be expressed as a parameter-to-specification (P2S) optimization problem, in which the objective is to find optimal device parameters (e.g., transistor width and finger count) that satisfy desired specifications (e.g., power and bandwidth) based on a selected circuit topology. Such problems are extremely challenging. First, it is necessary to search for optimal parameters for various devices in a huge design space. This complexity increases exponentially as the number of design parameters and desired specifications increases. Second, the actual correlation between device parameters and circuit specifications becomes complex depending on multiple factors, such as the circuit topology and the coupling of specifications. The lack of precise rules to follow makes the search even more difficult. Traditionally, human designers must manually derive device parameters from circuits using their experience and knowledge. However, manually creating empirical formulas is burdensome, and due to simplified physical models of devices and circuits, the derived parameters are often not immediately reliable, requiring dozens or even hundreds of iterations to ensure design accuracy.

[0004] Therefore, there is a need to develop design automation techniques for designing analog / RF circuits to facilitate the design procedure and reduce the time to market. [Means for solving the problem]

[0005] Some embodiments of the present invention recognize that automated analog circuit design is a long-standing challenge in the integrated circuit field. The present invention provides a deep reinforcement learning (RL) framework for supporting analog circuit design at the pre-layout level, with the goal of finding device parameters that meet desired specifications. Unlike all previous methods, the present method incorporates significant domain knowledge to determine the relationship between device parameters and circuit specifications, achieving the highest level of design accuracy and efficiency. This approach is applicable to the design of various analog circuits (e.g., high-frequency circuits) using different semiconductor technologies, overcoming the limitations of previous techniques in the design of a narrow range of low-frequency analog circuits using conventional semiconductor technologies. To achieve this capability, the present invention tunes the policy network of an RL agent by appropriately combining a graph neural network (GNN) and a fully connected neural network (FCNN) based on circuit topology. The proposed policy network can effectively capture common physical features (e.g., device parameters and interactions) present in the circuit graph using a GNN and extract specification couplings (e.g., design tradeoffs) using an FCNN, thereby optimally modeling the relationship between circuit parameters and design goals. Experimental results show that the method of the present invention achieves 97.5% accuracy in designing an exemplary circuit, with 1.5 times the efficiency of the best existing method. The method of the present invention also has excellent versatility and portability.

[0006] Furthermore, some embodiments recognize that machine learning (ML) techniques, such as supervised learning and reinforcement learning, have recently been used as automated methods to improve the design efficiency of analog circuits. By training neural network models, ML methods can cover vast design spaces and achieve design goals with far fewer iterations. Despite these significant advantages, most existing ML techniques treat P2S optimization as a black box, severely limiting its robustness, accuracy, and efficiency, by failing to consider practical yet crucial interrelationships between device parameters and circuit specifications. Furthermore, these ML methods are limited to a very narrow scope: low-frequency analog circuits (i.e., operational amplifiers) implemented using conventional complementary metal-oxide-semiconductor (CMOS) technology. These ML methods cannot be easily applied to other types of analog circuits designed with different semiconductor technologies.

[0007] The present invention provides a reinforcement learning (RL) method for P2S optimization. In this case, a well-trained RL agent can autonomously find optimal device parameters for desired specifications. Unlike all prior art, the present method achieves the highest design accuracy (97.5%) and efficiency (1.5 times that of the state-of-the-art) by incorporating domain knowledge that determines the relationship between device parameters and circuit specifications. Even for some failure cases, the RL agent's decision sequence can provide hints to warm-start manual tuning methods to ensure 100% design accuracy. The agent of the present invention can also generate optimal device parameters for unknown desired specifications. This superior capability is achieved by tuned policy networks, including graph neural networks (GNNs) and fully connected neural networks (FCNNs). The GNN is built based on the topology of a specific circuit and can capture the underlying physical properties of the circuit, such as device parameters, connections, and interactions. The FCNN extracts the coupling of the circuit specifications. This unique policy network allows our RL agent to include relevant circuit information in the loop and update device parameters by making optimal sequential decisions like an experienced human designer, thereby achieving the automated design of high-quality analog circuits.

[0008] Our method also overcomes the limitations of conventional techniques, which only design low-frequency analog circuits using conventional complementary metal-oxide semiconductor (CMOS) technology. By encoding device states using device physical features common to a wide range of semiconductor technologies, our method can design a variety of analog circuits in different semiconductor technologies. Specifically, our method can be used to design radio frequency (RF) circuits in gallium nitride (GaN) technology, a promising alternative to conventional CMOS technology and high-frequency and power electronics applications. RF circuits are a subclass of analog circuits that specialize in high-frequency signals and require more advanced analysis and electromagnetic characterization in addition to the design challenges faced by low-frequency analog circuits. By leveraging a fast reward signal during the deployment phase, our method can significantly reduce training time and accelerate the design of RF circuits. Our research demonstrates the potential of combining RL techniques with domain knowledge to bring us closer to a future in which artificial agents with extensive circuit optimization experience assist circuit designers.

[0009] Some embodiments of the present invention present a deep RL framework that can automatically design both low-frequency and RF analog circuits in different implementation technologies, i.e., CMOS and GaN. Previous work either uses supervised learning or focuses on designing a narrow range of low-frequency analog circuits in CMOS technology.

[0010] Furthermore, some embodiments are based on the recognition that by appropriately combining RL agents, GNNs and FCNNs, a policy network architecture can be constructed. This policy network can comprehensively refine the complex relationships between device parameters and circuit specifications. Therefore, this agent can achieve faster and better circuit optimization.

[0011] Furthermore, some embodiments of the present invention are based on the recognition that the proposed RL framework has the ability to generalize, i.e., a trained RL agent can leverage previously learned experience to optimize device parameters for new, unknown design specifications. Previous work has lacked such generality.

[0012] One embodiment shows that the trained RL agent achieves a state-of-the-art accuracy of 97.5% in obtaining optimal device parameters for randomly given design specifications during the deployment phase. For a very small number of failure cases, the RL agent's decision sequence can provide hints to warm-start manual tuning methods.

[0013] According to some embodiments of the present invention, a computer-implemented method for generating device parameters of a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCNN) is provided. The method uses a processor connected to a memory storing instructions implementing the method, which, when executed by the processor, perform method steps. The method steps include obtaining inputs via an interface relating to a set of desired specifications or one desired specification, device parameters, and a fixed topology of the circuit, and providing the inputs to the trained reinforcement learning (RL) agent, where the description of the desired circuit includes a graph modeling the topology and device parameters of the circuit, and the desired specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. For example, a trained reinforcement learning (RL) agent may be configured to perform the following steps, including sending an action selected from a set of actions to an environment module; and updating, using a data processor of the environment module, device parameters of the circuit to desired circuit specifications according to the selected action, wherein the action modifies each of the current device parameters by an atomic value for each device parameter; obtaining, using a circuit simulator of the environment module, a netlist of the circuit based on the updated device parameters, wherein the environment module includes the netlist of the circuit; and obtaining a reward from the environment module, wherein the reward is calculated based on a difference between the current and desired specifications; the sending, updating, obtaining, and obtaining steps are continued until the reward reaches a threshold or the number of steps reaches a predetermined value; and generating updated device parameters of the circuit.

[0014] Embodiments of the present disclosure will now be further described with reference to the accompanying drawings, which are not necessarily to scale, emphasis instead being placed upon illustrating the principles of embodiments of the present disclosure. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram illustrating an overview of a RL framework for automated design of analog circuits, in accordance with an embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram illustrating a mapping between circuit topology and graphs and a GNN-FC based policy network architecture tuned for analog circuit design, according to an embodiment of the present invention. [Figure 3A] FIG. 1 is a schematic diagram showing an RF polar TX in 150 nm GaN technology according to an embodiment of the present invention (D1-D5 and DF are driver stages). [Figure 3B] FIG. 1 illustrates a design space of device parameters and a sampling space of desired specifications for two benchmarks, in accordance with an embodiment of the present invention. [Figure 4A] FIG. 10 illustrates the evolution of the average episode reward of an RL agent, in accordance with an embodiment of the present invention. [Figure 4B] FIG. 10 illustrates the evolution of the average episode length of an RL agent, according to an embodiment of the present invention. [Figure 4C] FIG. 1 illustrates the evolution of deployment accuracy of an RL agent, in accordance with an embodiment of the present invention. [Figure 5A] FIG. 10 illustrates the evolution of the average episode reward of an RL agent, in accordance with an embodiment of the present invention. [Figure 5B] FIG. 10 illustrates the evolution of the average episode length of an RL agent, according to an embodiment of the present invention. [Figure 5C] FIG. 1 illustrates the evolution of deployment accuracy of an RL agent, in accordance with an embodiment of the present invention. [Figure 6A]FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 6B] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 6C] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 6D] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 7A] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 7B] FIG. 1 illustrates an example deployment of a trained RL agent attempting to reach a set of target specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8A] FIG. 1 illustrates a generalized example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8B] FIG. 1 illustrates a generalized example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8C] FIG. 1 illustrates a generalized example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 8D] FIG. 1 illustrates a generalized example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 9A] FIG. 1 illustrates a generalized example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 9B] FIG. 1 illustrates a generalized example of a trained RL agent attempting to arrive at a set of unknown new specifications for a circuit under design, according to an embodiment of the present invention. [Figure 10] 1A-1C illustrate a comparison of different design automation methods, according to an embodiment of the present invention. [Figure 11] 1 is a schematic diagram illustrating a system including a processor, a memory storing a computer-implemented method, and an interface according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016] The following description provides exemplary embodiments only and is not intended to limit the scope, application, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter as set forth in the appended claims.

[0017] In the following description, specific details are given to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagrams so as not to obscure the embodiments in unnecessary detail. Also, well-known processes, structures, and techniques may be shown without unnecessary detail so as not to obscure the embodiments. Furthermore, like reference numbers and names in the various drawings refer to like elements.

[0018] Design automation for analog / RF circuits can best be formulated as a parameter-to-specification (P2S) optimization problem. The goal is to find optimal device parameters (e.g., transistor width and length) that satisfy desired circuit specifications (e.g., power, speed, and bandwidth). Traditionally, these device parameters are derived manually based on the IC designer's experience and knowledge of the circuit. Such empirical approaches aim to link circuit specifications and device parameters in equations via simplified physical models of the device and circuit. Due to the highly nonlinear nature of analog / RF circuits, manually deriving these equations is labor-intensive and requires dozens of iterations to ensure design accuracy.

[0019] Various design automation techniques have been proposed for the P2S optimization problem of analog circuits, generally falling into two categories: optimization-based methods and learning-based methods. Bayesian optimization, geometric programming, genetic algorithms, and simulated annealing are representative optimization-based methods. For all of these methods, the search for optimal device parameters is guided by an algorithm. At each step, intermediate circuit specifications obtained from a standard simulator are compared with the design goals. The algorithm then determines how to update the device parameters for the next search. These techniques differ from each other in sampling efficiency, convergence rate, and overall optimality. However, a key issue is that if the desired specifications change, the algorithm must be restarted from scratch. Our method can reach different target specifications without systematic retraining.

[0020] Recently, learning-based methods have emerged. Supervised learning methods have been used to learn the complex relationships between device parameters and circuit specifications. For example, FCNN is trained to design a two-stage operational amplifier (op-amp), where the network's input is the circuit specifications and the output is the predicted device parameters. These methods cannot guarantee design accuracy due to inherent approximation errors. RL methods have been used to learn an optimal policy from the state space of desired specifications to the action space of device parameters. With sufficient search during the training phase, high design accuracy can be achieved. For example, RL methods can achieve a design accuracy of 3×10 4 After exploring environmental steps, optimal device parameters can be found that meet the desired specifications of the two-stage op-amp with better than 90% accuracy.

[0021] These learning-based methods rarely consider domain knowledge (i.e., circuit topology and specification coupling) that governs the relationship between device parameters and circuit specifications. Supervised learning methods use FCNNs to learn a static mapping between desired specifications and device parameters without relying on circuit topology. Similarly, RL methods use simple FCNNs as policy networks, ignoring the circuit graph and specification coupling. Another RL method builds a policy network on a graph convolutional network (GCN) to solve a different problem: optimizing the circuit's figure of merit (FoM). Nevertheless, GCNs ignore circuit specification coupling and only capture the circuit topology, excluding power and basis nodes, which are essential parts of the circuit graph. These methods are limited to designing low-frequency analog circuits, such as operational amplifiers, without understanding the nature of the circuit's graph. This is mainly because the electromagnetic coupling of devices in RF circuits strongly depends on the circuit's topology and significantly affects the specifications.

[0022] Furthermore, these learning-based methods focus only on the design of CMOS circuits. This limitation is caused by using technology-dependent features of CMOS devices, such as saturation voltage and bulk voltage, to encode transistor state vectors, which are not applicable to all semiconductor technologies. For example, GaN is a III / V semiconductor and has a different device model than CMOS. The method described in this disclosure incorporates domain knowledge of analog circuit design and uses technology-independent features to encode device states, thereby enabling design automation for a wide range of analog circuits (including both low-frequency analog circuits and RF circuits) implemented in different semiconductor technologies (including CMOS and GaN).

[0023] Despite these significant advantages, the best existing ML approaches treat P2S optimization as a black box without considering the practical yet crucial interrelationships between device parameters and circuit specifications, severely limiting its robustness, accuracy, and efficiency. Furthermore, these ML methods are limited to a very narrow scope: low-frequency analog circuits (i.e., operational amplifiers) implemented using conventional complementary metal-oxide-semiconductor (CMOS) technology. These ML methods cannot be easily applied to other types of analog circuits designed with different semiconductor technologies.

[0024] Our method is inspired by human design concepts and uses a combination of a GNN and an FCNN to comprehensively refine the complex relationships between device parameters and circuit specifications. None of the prior art approaches encompasses such practical design concepts and has achieved our capabilities. Autockt [Setalluri et al., 2020] uses a simple FCNN to design a policy network using device parameters and desired specifications as input, but ignores the graph nature of the circuit and the coupling of specifications. GCN-RL circuit designers optimize the figure of merit (FoM) of a specific circuit by building a policy network on a graph convolutional network (GCN) based on the circuit topology. However, GCNs only partially capture the circuit topology, ignoring power nodes, which are actually an essential part of the circuit graph. GCNs also ignore the coupling of circuit specifications. Furthermore, our method aims to achieve design automation for a wide range of analog circuits (including both low-frequency analog and RF circuits) implemented in different semiconductor technologies (including CMOS and GaN) by using technology-independent features to encode device state. Learning with Graph Neural Networks

[0025] Graph neural networks, such as graph convolutional networks (GCNs) and graph attention networks (GATs), are emerging neural networks that operate directly on graph-like non-Euclidean data structures. They are gaining increasing popularity in various domains, including social networks, knowledge graphs, recommendation systems, and life sciences. The present invention utilizes GCNs and GATs to incorporate the physical characteristics of specific circuits, such as device parameters, connections, and interactions, into the policy network of the present invention. The present invention demonstrates that GATs with a multi-head attention mechanism can better model the subtle physical characteristics of circuits than GCNs. problem statement

[0026] This disclosure provides design automation for analog circuits at a pre-layout level, with the goal of efficiently searching for optimal device parameters that meet desired specifications. This disclosure addresses scenarios where the topology of the analog circuit is fixed. This is practical because many classical topologies are usually available for a particular design task.

[0027] The outline of the present invention, an RL framework for automated design of analog circuits, is as follows: RL agents are based on the actor-critic method. The environment consists of a netlist of an analog circuit with a specific topology, a circuit simulator, and a data processor. At each time step i, the agent automatically performs an action a i and outputs the state s from the environment i and reward r i Update the equipment parameters according to the Reinforcement Learning Framework

[0028] Some embodiments use deep reinforcement learning (RL) techniques to achieve design automation of analog circuits. Some embodiments of the present invention may provide a computer-implemented method for generating device parameters of a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCNN). The method uses a processor coupled to a memory storing instructions for implementing the method. The memory is configured to store the trained RL agent, the GNN, and the FCNN. When executed by the processor, the instructions perform the method steps. The method steps include obtaining inputs related to a set of desired specifications or one desired specification, device parameters, and a fixed topology of the circuit via an interface, and providing the inputs to the RL agent, where the description of the desired circuit includes a graph that models the topology and device parameters of the circuit, and the desired specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. For example, the trained RL agent may be configured to perform the following steps, including sending an action selected from a set of actions to an environment module; and updating, using a data processor of the environment module, equipment parameters of the circuit to desired circuit specifications according to the selected action, wherein the action modifies each of the current equipment parameters by an atomic value for each equipment parameter; obtaining, using a circuit simulator of the environment module, a netlist of the circuit based on the updated equipment parameters, the environment module including the netlist of the circuit; obtaining a reward from the environment module, the reward calculated based on a difference between the current and desired specifications; the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold or the number of steps reaches a predetermined value; and generating updated equipment parameters of the circuit.

[0029] Additionally, a computer-implemented method for training an RL agent network to generate device parameters for a circuit is provided. The method uses a processor coupled to a memory storing instructions for implementing the method. The memory is configured to store an RL agent network. In this case, the processor is configured to execute instructions including obtaining inputs related to desired circuit specifications, device parameters, a fixed topology of the circuit, and a final state corresponding to a maximum step for the circuit. The desired circuit specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency. The RL agent network is configured to send an action selected from a set of actions to an environment module and, using a data processor of the environment module, update the device parameters of the circuit relative to the circuit specifications according to the selected action, wherein the action modifies each of the current device parameters by an atomic value for each device parameter. The steps include obtaining current specifications of the circuit by simulating a netlist of the circuit based on the updated device parameters using a circuit simulator of the environment module, the environment module including the netlist of the circuit, obtaining a reward from the environment module, the reward being calculated based on a difference between the current specifications and the desired specifications, the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold or the number of steps reaches a predetermined value, and generating, via the interface, the satisfied updated device parameters. Some specific examples of the system are shown in FIG. 11 of the present invention.

[0030] Figure 1 shows a framework 100 consisting of five main elements: an agent 110, i.e., a learner and decision maker (in our case, a policy 112 and a value network 111); an action space, i.e., a set of actions the agent 110 can take (i.e., increasing, decreasing, or maintaining the parameters of each device); a state space, i.e., a set of possible states the agent 110 can be in the environment 120 (i.e., any combination of the circuit's device parameters in the design space); a reward, i.e., a scalar value (e.g., the difference between the current and desired specifications) returned by the environment 120 when an action is taken in a certain state (this is used to evaluate and guide the agent's actual behavior); and an environment 120, i.e., the physical world in which the agent operates (i.e., a high-fidelity circuit simulation environment 122). A data processor unit 123 is also included in the environment. The netlist of a particular analog circuit 121 is used as input for the circuit simulator 122. The parameters of the circuit being designed are updated by instructions from the data processor unit onwards.

[0031]

number

[0032]

number

[0033] Since all N desired specifications are equally important, but not required, the same weight is assigned to each normalized difference, and the designer may assign different weights to different specifications.

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038] 2 is a schematic diagram illustrating a mapping between circuit topology and graphs and illustrating a GNN-FC-based policy network architecture tuned for analog circuit design, according to some embodiments of the present invention. The diagram illustrates the mapping of circuit topology 210 to a graph neural network (GNN) 211 and illustrates a GNN-FC-based policy network 200 architecture tuned for analog circuit design. In this case, a two-stage op-amp, designated 210, is used as an example. State Representation

[0039]

number

[0040] The environment 120 of the present invention consists of a netlist 121 of a particular circuit, a high-fidelity circuit simulator 122 (e.g., Cadence Spectre or Advanced Design System (ADS)), and a data processor 123. The current specifications of the circuit are obtained by simulating the netlist using the circuit simulator. The data processor 123 then processes the simulation results to feed back rewards to the agent 110. The data processor also updates device parameters and rewrites the circuit netlist 121 in response to actions from the agent 110.

[0041] The present invention uses an actor-critic method with a critic block (111) and an actor block (112) to construct an agent 110. To train an agent that can accurately optimize device parameters to satisfy a desired specification 201, we propose a neural network architecture, the policy network 200 shown in Figure 2. This neural network architecture, consisting of a graph neural network (GNN) 211 and a fully connected neural network (FCNN) 202, is called a GNN-FC-based policy network. The role of the GNN 211 is to extract physical information about the node types and connections in the circuit graph into low-dimensional vector embeddings 212. Specifically, a graph convolutional network is used to extract these physical features. Furthermore, the FCNN 202, which takes the design goals and embeddings 203 as input, extracts their coupling relationships, e.g., design tradeoffs between different specifications. The final FC layer 220 then updates the behavior by concatenating and further processing the graph embeddings 212 and FCNN embeddings 203. The value network 111 maintains the same architecture as the policy network 112, except for the last layer, which outputs an estimate of the expected reward for the execution of the current policy.

[0042] Our goal is to have the RL agent (i.e., policy network 112) generate higher quality decisions as it gains experience from searching for optimal device parameters relative to desired specifications. The objective function for automated analog circuit design can be formally defined as follows:

number

[0043]

number

[0044] We evaluate the proposed RL framework using two representative analog circuits. First, we select the two-stage op-amp in Figure 2 as an exemplary low-frequency analog circuit because this op-amp has been adopted as a standard benchmark by many optimization-based and learning-based methods. This op-amp is implemented in 45nm CMOS technology.

[0045] Next, we select the RF polar transmitter (polar TX) of Figure 3A implemented in 150 nm GaN technology as an example of a high frequency analog circuit. This polar TX consists of five driver stages D1, D2, D3, D4, and D5, shown at 310, 320, 330, 340, and 350, and a final stage DF360, shown in detail in circuit diagram 361. Circuit diagram 311 is a schematic diagram showing the driver stage with the same topology as D1-D5. The input is a differential signal V in_p1 301 and V in_p2 302. The output port is V out It is 370.

[0046] Figure 3B shows the design space of device parameters and the sampling space of desired specifications for the two circuits. The two-stage op amp has a total of 15 parameters, and the RF polar TX has a total of 14 parameters. Note that 1) the two transistors in the differential pair of the op amp have identical parameters, and 2) the devices in each driver stage D1-D5 of the RF polar TX also have identical parameters. These are practical constraints imposed by the circuit design.

[0047] A separate RL agent (110) is trained to enable the automatic design of each circuit. In our experiments, training each two-stage op-amp requires approximately 3 hours of CPU time, while training each RF polar TX requires approximately 12 hours of CPU time. RF circuits require more sophisticated simulations and therefore require more time to obtain training rewards. To demonstrate the unique advantages of our GNN-FC-based policy network, we use an FCNN-based policy network as a baseline, which does not consider the physical topology of the circuit. All of our experiments were performed on an 8-core Intel® CPU. Agent Training

[0048] The episode length for the two-stage opamp RL agent (PolarTX RL agent) is set to 50 (30). The total episodes used to train the two RL agents are 3.5 × 10, respectively. 4 and 3.5 × 10 3 The RF polar TX requires fewer episodes for training because it has fewer device parameters, fewer target specifications, and a smaller design space. Figures 4A, 4B, and 4C are curves showing the average episode reward, average episode length, and deployment accuracy, respectively, of the RL agent with different policy network architectures for designing two-stage op-amps during training.

[0049] Figures 5A, 5B, and 5C are curves showing the mean episode reward, mean episode length, and deployment accuracy, respectively, of the RL agent with different policy network architectures for designing Polar TX during training.

[0050] As can be seen from Figures 4A and 5A, the more episodes used for training, the higher the average episode reward obtained from each policy network. However, the proposed GNN-FC-based policy network can achieve higher rewards than the conventional FCNN-based policy network. The average episode length of each policy network in Figures 4B and 5B is also shorter for training episodes. The policy network of the present invention exhibits a shorter (faster) average episode length compared to the baseline policy network. Figures 4C and 5C also show the deployment accuracy of the two policy networks for training episodes. Policy deployment refers to applying a trained policy network to automatically find device parameters for given desired specifications. Each point in the right column of Figures 4C and 5C is obtained from statistics of deploying each policy network for 200 sets of randomly sampled desired specifications. This comparison shows that the proposed GNN-FC-based policy network has higher deployment accuracy for both circuit designs than the baseline FCNN-based policy network. The proposed policy network achieves a state-of-the-art policy deployment accuracy of 97.5%. These results suggest that the proposed GNN-FC based policy network is more efficient and accurate for analog circuit design by combining both the physical circuit features via the GNN part and the coupling between design specifications via the FC part. Automating circuit design through policy deployment

[0051] The present invention is based on the predefined episodes (3.5 × 10 for a two-stage op-amp). 4 pcs, and for Polar TX, 3.5 x 10 4We demonstrate how this approach can be applied to the automation of analog circuit design by deploying a GNN-FC-based policy network trained using a set of randomly sampled two-stage op-amps. Figures 6A, 6B, 6C, and 6D each show an example of such a two-stage op-amp. In this case, the RL agent automatically searches for optimal device parameters from an initial state (step 0) for a circuit with a set of randomly sampled desired specifications. The desired specifications for the sampled two-stage op-amp are gain (G = 350) in Figure 6A and power consumption (P = 4 × 10) in Figure 6B. -3 W), the phase margin (PM = 55°) in Figure 6C, and the bandwidth (B = 1.8 × 10 7 Hz). Note that lower power consumption indicates better performance. As shown in Figures 7A and 7B, the sampled desired specifications for an RF polar TX are output power (W) and power efficiency (E). These sampled specifications are represented by horizontal dashed lines in Figures 6A-6D and Figures 7A and 7B. Initially, the current specifications (y-axis in each figure) generally deviate significantly from the desired specifications. As deployment continues, the RL agent follows the path guided by the trained policy to find optimal device parameters for the specific specifications, so that the current specifications approach the desired specifications. The RL agent requires 27 steps and 11 steps to achieve the design goal for each circuit.

[0052] An interesting phenomenon is that once a certain specification is first achieved, the RL agent does not overoptimize that specification but instead attempts to optimize those that have not yet been achieved. For example, in the 14th deployment step, the gain of a two-stage op-amp is first achieved. In subsequent steps, the RL agent focuses on optimizing the unachieved specifications, namely, phase margin and bandwidth. Similarly, when deploying an RF polar TX, the output power is reached first, and then the RL agent continues to optimize power efficiency. This property enabled by RL methods is very promising, as traditional manual tuning lacks the ability to successfully balance the trade-offs between all desired specifications during optimization. We also analyzed several failure cases in which the trained policy failed to converge to the optimal device parameters. In these failure cases, we observed that some specifications were able to converge near the desired specifications, but then slightly deviated from the target. Fortunately, we found that the design goal could be easily achieved by slightly manually adjusting the device parameters at that particular step. These results suggest that IC designers can significantly benefit from trained policies, even if automated deployment fails, by using them as an efficient warm start for manual tuning. Detailed analysis and further demonstration of the results are provided in the supplemental material. Generalization to unknown specifications

[0053] Furthermore, we evaluate the generalization ability of the GNN-FC-based policy network. That is, we deploy it with unknown specifications outside the sampling space shown in Figure 3B. For a two-stage op-amp, the set of sampled unknown specifications is the gain (G = 225), bandwidth (B = 2.6 × 10 7 Hz), phase margin (M=65°), and power dissipation (P=6×10 -3For the RF polar TX, the set of sampled unknown specifications is output power (P = 2.9 W) and power efficiency (E = 69%). Figures 8A-8D, 9A, and 9B show the deployment trajectories, with horizontal dashed lines indicating these unknown specifications. Compared to the policy deployments shown in Figures 6A-6D and 7A and 7B (where specifications are sampled from the training space in Figure 3A), deployment to unknown desired specifications typically requires more search steps. For example, 11 steps are sufficient for the normal deployment of the RF polar TX in Figures 7A and 7B, whereas generalization of the RF polar TX requires 49 steps to achieve the goal. This difference is due to the fact that the unknown desired specifications are beyond the range of the training dataset, requiring more steps to reach the optimal parameters. However, we found that the baseline FCNN-based policy network lacks generalization ability for most unknown specifications, even with a large number of search steps. The generalization of the proposed policy network can capture a rich representation of the state space (e.g., physical topology and coupling relationships between specifications), allowing it to leverage the experience learned from previous states to new, unknown states during inference. Comparison with other methods

[0054] Finally, we compare the efficiency of our method with previous optimal tuning methods. The comparison results are shown in Figure 10. In the deployment (inference) phase, our RL method outperforms Bayesian optimization. Bayesian optimization generally requires more steps to find optimal device parameters for the same desired specifications compared to our method. Compared to previous RL methods (K. Settaluri, A. Haj-Ali, Q. Huang, K. Hakhamaneshi, B. Nikolic; AutoCkt: Deep Reinforcement Learning of Analog Circuit Designs, 2020 Design, Automation & Test in Europe Conference Exhibition (DATE), pages 490-495, 2020), our method is more efficient with fewer deployment steps due to its unique policy network architecture. Finally, our method possesses generalization capabilities not demonstrated by other methods. In summary, the proposed method demonstrates advances in efficiency, accuracy, and generalization.

[0055] We also compare our method with another study (H. Wang, K. Wang, J. Yang, L. Shen, N. Sun, H.S. Lee, S. Han; GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learning, 2020 57th ACM / IEEE Design Automation Conference (DAC), pages 1–6, 2020) in optimizing the figure of merit (FoM) of a two-stage operational amplifier. This study differs from ours technically in many respects, including the RL technique (DDPG vs. PPO), state space (device technology-dependent vs. device technology-independent features), and action space (continuous vs. discrete). For a fair comparison, we construct a baseline using the PPO technique and discrete action space, similar to our study, but using a GCN as the policy network and FoM as the reward function. Furthermore, we compare the accuracy of our method with supervised learning methods (not RL) in predicting device parameters for specific specifications. For comparison, we first train an FCNN. The input is the desired specifications, and the output is the corresponding device parameters. The desired specifications are then randomly given and used for prediction, and the accuracy is verified by circuit-level simulation. The comparison results show that the method of the present invention has much higher accuracy (98% vs. 87%).

[0056] Therefore, some embodiments of the present invention provide a deep reinforcement learning (RL)-based framework for automated analog circuit design. A key feature of the present framework is its ability to incorporate both the underlying physical topology of a particular circuit and the tradeoffs between its specifications into a newly proposed hybrid GNN-FC-based policy network. Such a network architecture is superior to other network architectures that primarily capture physical features but fail to adequately incorporate the relationship between the physical topology and the tradeoffs between its specifications. The present network architecture can design both high-quality low-frequency and RF circuits across different implementation technologies. In particular, the present invention demonstrates its ability to accelerate the design of RF circuits compared to conventional optimization methods. Furthermore, the present framework demonstrates good generalization ability, i.e., the ability to generate optimal design parameters for unknown targets. It is believed that the present method can help accelerate analog chip design in the IC industry by using artificial agents that can accumulate extensive circuit optimization experience through continuous training.

[0057] FIG. 11 is a schematic diagram illustrating a system 1100 configured to include a processor 1120, a memory 1120 connected to storage 1140 that stores computer-implemented methods, and an interface 1105, in accordance with some embodiments of the present disclosure.

[0058] 11 is a block diagram illustrating an example of a system 1100 for automated construction of artificial neural network architectures, according to some embodiments of the present disclosure. The system 1100 includes a set of interfaces and data links 1105 configured to send and receive signals, at least one processor 1120, a memory (or a set of memory banks) 1130, and storage 1140. The processor 1120, in association with the memory 1130, executes computer-executable programs and algorithms stored in the storage 1140. The set of interfaces and data links 1105 may include a human-machine interface (HMI) 1110 and a network interface controller 1150. The processor 1120 may execute computer-executable programs and algorithms in association with the memory 1130, which uploads the computer-executable programs and algorithms from the storage 1140. The computer-executable programs and algorithms stored in storage 1140 may be a trained RL agent (neural network) 1141, a graph neural network 1142, a fully connected neural network 1143, instructions (programs) for performing a computer-implemented method 1144, a temporary cache 1145, device parameters 1146, and an environment module 1147.

[0059] The system 1100 can receive signals via a set of interfaces and data links, which may be a data set consisting of training data, desired circuit description data, and test data including gain, bandwidth, phase margin, power consumption, output power, and power efficiency.

[0060] At least one processor 1120, in association with an interface and memory bank 1105, is configured to inject signals and data sets into the reconfigurable DNN block 1141. The system 1100 can receive circuit data and perform circuit simulations with an associated computing system 1195B via a network 1190 and a set of interfaces and data links 1105.

[0061] Each embodiment may also be described as a process, which is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel or simultaneously. The order of operations may also be changed. A process may be terminated when its operations are completed, but the process may include additional steps not discussed or shown. Furthermore, not all operations within a specifically described process need be included in all embodiments. A process may be a method, a function, a procedure, a subroutine, a subprogram, etc. When a process is a function, the termination of the function corresponds to the function returning to the calling function or the main function.

[0062] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. The manual or automatic implementation may be implemented, or at least assisted, by machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented by software, firmware, middleware, or microcode, program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0063] The embodiments of the present disclosure described above may be implemented in many ways. For example, the embodiments may be implemented in hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or group of processors, whether located on a single computer or distributed across multiple computers. Such a processor may be implemented as an integrated circuit. A single integrated circuit element may include one or more processors. However, the processor may be implemented in any suitable circuit.

[0064] Additionally, the various methods or steps outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Moreover, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed in various embodiments as desired.

[0065] The embodiments of the present disclosure may be embodied as methods, which are provided by way of example. The actions performed as part of this method may be ordered in any suitable manner. Accordingly, embodiments may be constructed that perform actions in an order different from the sequential actions performed in the exemplary embodiments, and that may include performing some actions simultaneously. Furthermore, the use of order terms in the claims, such as first, second, etc., to modify claim elements does not imply a priority, precedence, or order of one claim element relative to another claim element, or a chronological order in which the actions of the method are performed, but rather is merely used as a label to distinguish between claim elements and to distinguish one claim element with a particular name (by using order terms) from another element with the same name.

[0066] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other modifications and variations can be made within the spirit and scope of the present disclosure. It is therefore intended in the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the present disclosure.

Claims

1. 1. A computer-implemented method for generating device parameters for a circuit using a trained reinforcement learning (RL) agent composed of a graph neural network (GNN) and a fully connected neural network (FCN), the method using a processor connected to a memory storing instructions implementing the method, the instructions, when executed by the processor, perform steps of the method, the steps including: obtaining, via an interface, inputs relating to a set of desired specifications or one desired specification for a circuit, device parameters, and a fixed topology of the circuit; and providing the inputs to the trained reinforcement learning (RL) agent, wherein the desired description of the circuit includes a graph that models the topology of the circuit and device parameters of the circuit, and the desired specifications include gain, bandwidth, phase margin, power consumption, output power, and power efficiency; and wherein the trained reinforcement learning (RL) agent is configured to perform the following steps: sending a selected action from the set of actions to an environment module; and updating, using a data processor of the environment module, the device parameters of the circuit to the desired circuit specifications in accordance with the selected action, the action modifying each of the current device parameters by the atomic value of each device parameter; obtaining a current specification of the circuit by simulating a netlist of the circuit based on the updated device parameters using a circuit simulator of the environment module, the environment module including the netlist of the circuit; obtaining a reward from the environmental module, the reward being calculated based on a difference between the current specification and the desired specification, and the sending, updating, obtaining, and obtaining steps continuing until the reward reaches a threshold or a number of steps reaches a predetermined value; generating updated device parameters for the circuit; The reward is calculated by a weighted sum of all desired specifications by assigning different values ​​of weighting coefficients; the fully connected neural network (FCN) embedding the desired specifications of the circuit; the graph neural network (GNN) embeds the topology and parameters of the circuit; the device parameters include at least one of a transistor width, length, and number of fingers; The method, wherein the trained reinforcement learning (RL) agent is based on an actor-critic approach.

2. The circuit is modeled by a graph G(V,E), Each node V is represented by a device, The method of claim 1 , wherein an edge E represents a connection between devices.

3. 10. The method of claim 1, wherein power (VP), ground (VGND), and other DC bias voltages of the circuit topology are represented as extra nodes V.

4. The method of claim 1 , wherein the graph neural network (GNN) can be implemented by different types such as a graph convolutional network or a graph attention network (GAT).

Images