Distributed circuit design using single step reinforcement learning

Abstract

A method and apparatus for generating distributed circuit configurations aimed at achieving a target performance goal for a distributed circuit. The method involves a neural network receiving a constant input for a circuit design with fixed dimensions. The neural network outputs a set of probability distributions from which sample circuit configurations are generated. Each configuration corresponds to a unique set of dimensions and is mapped to a physical representation of a potential distributed circuit. Feedback from an evaluator is received, based on the error between the target performance goal and the calculated performance of each potential circuit. The neural network is updated using a statistical loss derived from these evaluations, and the process is repeated until a circuit with the target performance is achieved. An apparatus and non-transitory computer-readable medium are also provided.

Description

FTRW-01414WO0 6000673PCT02 - 1 - DISTRIBUTED CIRCUIT DESIGN USING SINGLE STEP REINFORCEMENT LEARNING Inventors: Masood Mortazavi Jiayu Li Ning Yan CLAIM OF PRIORITY

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 600,202, entitled “Exploring Distributed Circuit Design Using Single-Step Reinforcement Learning”, filed November 17, 2023, which application is incorporated by reference herein in its entirety. FIELD

[0002] The disclosure generally relates to systems and methods for the automated design of distributed circuits. BACKGROUND

[0003] Some types of distributed circuits consist of components which can be depicted in a geometric form. These include square resonators, ring resonators, or bar resonators. As emerging 5G and 6G communication technologies and quantum computers advance towards increasingly higher frequencies, it becomes important for distributed circuit design on resonators to meet desirable performance requirements. The process of designing distributed circuits, however, is slow and burdensome. Each resonator component, and the arrangement of components on a substrate, may have a set of design parameters and constraints which are selected to achieve a design goal, such as the circuit’s transfer function. The performance of a distributed circuit (i.e., its transfer function s21) is determined by the properties of its resonators.

[0004] With the frequency ^^^^^^^ୀ^as the independent variable, the transfer function s21(^^^^^^^ୀ^) of a distributed resonator circuit describes the relationship between the input andFTRW-01414WO0 6000673PCT02 - 2 - the output signal. One can parameterize resonators in circuits and determine their transfer functions s21by an electromagnetic (EM) simulation. One current design technique uses specific candidate templates and a desired transfer function s21 in a pre-trained differentiable forward neural network model to solve the inverse design problem. However, templates differ in the topology types and the number of resonators in distributed circuits, and experts begin with a desired s21without prior preferences regarding an exact topology type. These methods assume either over-restricted candidate template topology or the differentiability of evaluation procedures. Real-world design practices should use non-restrictive template topologies and nondifferentiable evaluation procedure, without the need for prior domain knowledge on the template topology. SUMMARY

[0005] A general aspect includes a computer implemented method generating distributed circuit configurations for an established target performance goal for a distributed circuit, the distributed circuit having a plurality of circuit elements positioned on a substrate by: receiving, by one or more computerized devices containing a neural network, a constant input for a distributed circuit design having a fixed number of dimensions; outputting, by the neural network, the fixed number of probability distributions; generating sample circuit configurations for the design based on the probability distributions, where each sample configuration of the sample configurations corresponds to a different configuration of the fixed number of dimensions; mapping each of the sample configurations to a physical representation of a potential distributed circuit; receiving feedback from an evaluator on each of the physical representations, the feedback based on a calculated error between the target performance goal and a calculated performance of each potential distributed circuit; updating parameters of the neural network based on a statistical loss derived based on each of the potential distributed circuits. The method also includes repeating the generating distributed circuit configurations until a distributed circuit configuration having a target calculated error is reached. Other embodiments of this aspect include corresponding computer systems, apparatus, and computerFTRW-01414WO0 6000673PCT02 - 3 - programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0006] Embodiments of the present disclosure may include any of the foregoing embodiments wherein the computer implemented method includes mapping by applying mapping functions having an interdependence on each other. The embodiments may include any of the foregoing embodiments wherein said mapping includes defining a boundary in a potential circuit space and locating all circuit elements within the boundary. Other embodiments may include any of the foregoing embodiments wherein said mapping includes defining a transformation function which maps actions produced by the neural network to coordinates in physical space in a potential circuit space. Further embodiments may include any of the foregoing embodiments wherein said mapping functions place individual circuit elements in the potential circuit space based on a location defined for a previous individual circuit element. The embodiments may include any of the foregoing embodiments wherein the receiving feedback includes providing a fast feedback loop and a slow feedback loop, the slow feedback loop may include an simulator calculating a performance output of sample physical designs, the fast feedback loop may include an approximation of a simulator using a surrogate model to calculate the performance output of sample physical designs. The embodiments may include any of the foregoing embodiments wherein multiple instances of generating distributed circuit configurations occur in parallel, all seeking a circuit design with the same target performance goal. The embodiments may include any of the foregoing embodiments wherein ones of the multiple instances of generating distributed circuit configurations occurring in parallel, all seeking a circuit design with the same target performance goal, use a different fixed number of resonators. The embodiments may include any of the foregoing embodiments wherein the fixed dimensions include parameters that define behavior a resonator include a length of a side of the resonator, a distance between resonators, and a total number of square resonators in the circuit. The embodiments may include any of the foregoing embodiments wherein the distributed circuit is a distributed square resonator circuit. The embodiments may include any of the foregoing embodiments wherein the neural network generates the probability distributions using a weight resonant policy network with single step reinforcement learning.FTRW-01414WO0 6000673PCT02 - 4 - The embodiments may include any of the foregoing embodiments wherein the method further includes receiving a design template describing a design space, including receiving a description of each discrete decision dimension and / or each continuous decision dimension, and generating a policy network in the neural network based on the design template. The embodiments may include any of the foregoing embodiments wherein the constant input may include a single constant value, a tensor of 1s, or a tensor of hot 1s. The updating the parameters of the neural network including updating weights of the neural network. The embodiments of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0007] One general aspect includes an apparatus. The apparatus includes a storage medium may include computer instructions and one or more processors coupled to communicate with the storage medium, where the one or more processors execute the instructions to cause the apparatus to generate distributed circuit configurations for an established a target performance goal for a distributed circuit, the distributed circuit having a plurality of circuit elements positioned on a substrate, the instructions causing the processor to: create a neural network may include a single step reinforcement learning network. provide to the neural network, a constant input for a distributed circuit design having a fixed number of dimensions; output, from the neural network, the fixed number of probability distributions; generate sample circuit configurations for the design based on the probability distributions, where each sample configuration of the sample configurations corresponds to a different configuration of the fixed number of dimensions; map each of the sample configurations to a physical representation of a potential distributed circuit; generate feedback from an evaluator on each of the physical representations, the feedback based on a calculated error between the target performance goal and a calculated performance of each potential distributed circuit; update parameters of the neural network based on a statistical loss derived based on each of the potential distributed circuits; and repeating the generating distributed circuit configurations until a distributed circuit configuration having a target calculated error is reached. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.FTRW-01414WO0 6000673PCT02 - 5 -

[0008] The embodiments of the present disclosure may include any of the foregoing embodiments of an apparatus wherein the instructions cause the processor to map by applying mapping functions having an interdependence on each other. Further embodiments may include any of the foregoing embodiments of an apparatus wherein the instructions cause the processor to map by defining a boundary in a potential circuit space and locating all circuit elements within the boundary. The embodiments may include any of the foregoing embodiments of an apparatus wherein the instructions cause the processor to define a transformation function which maps actions produced by the neural network to coordinates in physical space in a potential circuit space. The embodiments may include any of the foregoing embodiments of an apparatus wherein the instructions cause the processor to place individual circuit elements in the potential circuit space based on a location defined for a previous individual circuit element. The embodiments may include any of the foregoing embodiments of an apparatus wherein the instructions cause the processor to provide a fast feedback loop and a slow feedback loop, the slow feedback loop may include a simulator calculating a performance output of sample physical designs, the fast feedback loop may include an approximation of a simulator using a surrogate model to calculate the performance output of sample physical designs. The embodiments may include any of the foregoing embodiments of an apparatus wherein the instructions cause the processor to cause multiple instances of generating distributed circuit configurations to occur in parallel, all seeking a circuit design with the same target performance goal. The embodiments may include any of the foregoing embodiments of an apparatus wherein the instructions cause the processor creating the multiple instances of generating distributed circuit configurations occurring in parallel, all seeking a circuit design with the same target performance goal, to use a different fixed number of resonators. The distributed circuit is a distributed square resonator circuit. The instructions cause the processor to create the neural network configured to generate the probability distributions using a weight resonant policy network with single step reinforcement learning. The constant input may include a single constant value, a tensor of 1s, or a tensor of hot 1s. The embodiments of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.FTRW-01414WO0 6000673PCT02 - 6 -

[0009] One general aspect includes a non-transitory computer-readable medium storing computer instructions for rendering images generating (410) distributed circuit configurations for an established target transfer function for a distributed circuit, the distributed circuit having a plurality of transformers positioned on a substrate, by: receiving, by a neural network, a constant input for a distributed circuit design having a fixed number of dimensions (430); outputting, by the neural network, the fixed number of probability distributions; generating, by the neural network, sample configurations for the design based on the probability distributions, where each sample configuration of the sample configurations corresponds to a different configuration of the fixed number of dimensions (435); mapping each of the sample configurations to a geometric representation of a potential distributed transformer circuit (440); receiving feedback from an evaluator on each of the geometric representations, the feedback based on a calculated error between the target transfer function and a calculated transfer function of each potential distributed circuit (445, 450); updating parameters of the neural network based on a statistical loss derived based on each of the potential distributed circuits(460). The instructions also includes repeating the generating distributed circuit configurations to reach a target calculated error is reached. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0010] The embodiments of the present disclosure may include any of the foregoing embodiments of a non-transitory computer readable medium wherein said mapping may include applying mapping functions having an interdependence on each other. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein said mapping includes defining a boundary in a potential circuit space and locating all circuit elements within the boundary. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein said mapping functions place individual resonators in the potential circuit space based on a location defined for a previous individual circuit element. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein the receiving feedback includes providing a fast feedback loop and a slow feedback loop, the slow feedbackFTRW-01414WO0 6000673PCT02 - 7 - loop may include an electromagnetic simulator calculating a performance output of sample distributed resonator circuit designs, the fast feedback loop may include an approximation of a simulator using a surrogate model to calculate the performance output of sample physical designs. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein multiple instances of generating distributed circuit configurations occur in parallel, all seeking a circuit design with the same target transfer function. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein ones of the multiple instances of generating distributed circuit configurations occurring in parallel, all seeking a circuit design with the same target performance goal, use a different fixed number of resonators. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein the fixed dimensions include parameters that define behavior a resonator include a length of a side of the resonator, a distance between resonators, and a total number of square resonators in the circuit. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein the neural network generates the probability distributions using a weight resonant policy network with single step reinforcement learning. The embodiments may include any of the foregoing embodiments of a non-transitory computer readable medium wherein the constant input may include a single constant value, a tensor of 1s, or a tensor of hot 1s. The embodiments of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0011] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to embodiments that solve any or all disadvantages noted in the Background.FTRW-01414WO0 6000673PCT02 - 8 - BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying figures for which like references indicate the same or similar elements.

[0013] FIG.1 is a perspective view of a distributed square resonator circuit.

[0014] FIG.2 illustrates a geographic representation of the circuit of FIG.1.

[0015] FIG. 3 is a flowchart illustrating a first method in accordance with the disclosed embodiments.

[0016] FIG.4 is a flowchart illustrating a method of steps 340, 350, and 360 of FIG.3.

[0017] FIG.5 illustrates a system for implementing the described embodiments.

[0018] FIG.6 illustrates a geometric representation of a distributed square resonator circuit and coordinates relative to the disclosed embodiments.

[0019] FIG. 7 illustrates a single step Markoff decision process reinforcement-learning model for design space exploration.

[0020] FIG.8 illustrates the inter-dependence of various sets of functions in the mapping functions module.

[0021] FIG.9 illustrates the structure of the mapping functions.

[0022] FIGs.10A – 10D illustrate an example of mapping actions produced by the resonant policy network to a physical design.

[0023] FIG.11 is a table comparing the error ^db of inverse designs on circuits with 3, 4, 5 and 6 resonators created by the disclosed method using a transformer-based network.

[0024] FIGs.12A through 12D are visualizations of the transformer-based performance of the disclosed embodiments vs. Circuit-GNN in achieving the target transfer function.FTRW-01414WO0 6000673PCT02 - 9 -

[0025] FIG. 13 illustrates a further embodiment of the described embodiments where multiple disclosed systems operate in parallel.

[0026] FIG.14 is yet another embodiment where, given a target transfer function, multiple parallel systems can run on different numbers N of resonators to determine the optimal number of resonators for a given target transfer function.

[0027] FIG.15 is a block diagram illustrating an example of a network processing device that can be used to implement various embodiments of the embodiments described herein. DETAILED DESCRIPTION

[0028] Reinforcement learning (RL) is a machine learning approach that involves an agent learning to interact with an environment in order to maximize a reward signal to improve operation of a system or process such as design space exploration processes and their efficiencies. RL can be employed to aid in the search and optimization of design configurations to find improved designs while minimizing design space exploration efforts. The described embodiments of the disclosure are directed to a system and methods for designing a distributed circuit which has a bounded set of dimensions. The described embodiments utilize a weight resonant policy network with single step RL to generate distributed circuits. In embodiments, the distributed circuits are represented in geometric form. The disclosed embodiments may be referred to as Distributed Circuit Inverse Design Automation (DCIDA) as it comprises an inverse design framework tailored for generating distributed circuits to meet desirable performance functions (which in described embodiments comprise resonator transformation functions s21). DCIDA has a policy network with trainable parameters and a constant input (representing a blank design slate), and it generates joint ‘hybrid actions’ distribution as a complete set of conditionals in each design dimension. With the joint distribution, a series of discrete and continuous actions are sampled as a compound action in a single step. Circuit design samples are generated using design boundaries and interdependent (conditional) mapping functions. The functions map the sampled conditional actions to the physical properties of resonators within design boundaries. Experimental results performed by theFTRW-01414WO0 6000673PCT02 - 10 - disclosed embodiments on data using a dataset from a prior attempt at automated design illustrate that the disclosed embodiments produce distributed circuits that surpass those generated by prior work.

[0029] The described embodiments may be applied to several types of distributed circuits built with identical circuit elements. In embodiments described herein, the individual circuit elements comprise resonators, with the distributed circuit designed to perform a specific transfer function. However, it should be understood that the embodiments are not limited to resonators but can be applied to any design where a target performance metric can be defined. In described embodiments relating to distributed resonator circuits, the target performance metric is a target transfer function.

[0030] In embodiments of a system, a neural network Netθconsumes a constant input tensor and produces a policy πθas a set of conditional probability density functions (PDFs) for sampling. These density functions comprise sample circuit configurations with a fixed set of dimensions. Policy gradient algorithms in deep reinforcement learning (D-RL) are used to evaluate feedback on sample circuit configurations to update Netθ. The weight resonant policy network with single step RL can, in one embodiment, learn to generate a distributed circuit configuration optimized with a fixed number of circuit elements to meet a target performance goal. The described embodiments may include a distributed resonator circuit with individual circuit elements comprised of square resonators. The described framework makes the Design Space Exploration (DSE) sample efficient and increases the training speed during the generation of circuits without frequently communicating with a simulator.

[0031] Each dimension comprises a set of one or more choices of an individual parameters (or continuous dimensions of choice or mixed discrete-continuous dimensions of choice where some dimensions involve continuous dimensions), and each dimension represents a different aspect of a design that can be adjusted. Each dimension in the design space corresponds to a specific design choice or configuration. A bounded design space includes a finite and fixed number of dimensions. There may be a choice of a number of value choices within each dimension, or the choices may be continuous.FTRW-01414WO0 6000673PCT02 - 11 -

[0032] FIG.1 is a perspective view of a distributed square resonator circuit 100 and FIG. 2 illustrates a geographic representation 200 of the circuit 100 of FIG.1. A square resonator is a type of resonator with a square physical geometry that resonates at specific frequencies based on its dimensions and the properties of the medium inside the resonator. It is characterized by its square shape, with all sides of equal length. A distributed circuit with multiple square resonators is a configuration where several square-shaped resonators 102, 104106, 108 are deposited or formed on a substrate 110 that are arranged to interact with each other, often through coupling mechanisms. Such circuits are common in microwave engineering, photonic systems, and optical waveguides. The parameters that define the behavior of these circuits include the side of the resonators, the distance between resonators (the spacing between adjacent resonators, referred to herein as the “gap”) and the total number of square resonators in the circuit. The transfer function s21represents how much of the input signal is transmitted through the circuit. The distributed resonator structure can be fabricated from different materials depending on the application. The square resonators 102, 104106, 108 may comprise of dielectric materials in optical resonators or conductive materials in microwave resonators. Square resonators support multiple resonant modes, which are determined by the dimensions of the square and the boundary conditions (e.g., whether the edges are reflective or transmissive). In microwave circuits, square resonators are used in filters, oscillators, and sensors. They can isolate certain frequency bands by blocking certain frequencies that do not match their resonant conditions.

[0033] As noted above, the disclosed embodiments generate distributed circuit configurations using design space exploration solved by reinforcement learning. A single-step RL approach, or θ-Resonance, as disclosed in publications PCT / US2023 / 070271 and PCT / US2022 / 026730 (each of which applications is fully incorporated by reference herein in its entirety) may be used for definite-horizon design explorations. These applications describe this approach with respect to discrete design spaces and use heuristic reward assignment to handle anomalous designs.

[0034] FIG. 3 is a flowchart illustrating a first method in accordance with the disclosed embodiments. At 310, for each distributed circuit to be generated for a given target metric (aFTRW-01414WO0 6000673PCT02 - 12 - transfer function for a distributed resonator circuit), a fixed number of actions (choices) and fixed number of elements (resonators) is defined. The aforementioned single step RL network operates on discrete design spaces having a fixed number of actions. As described herein, multiple networks may be used in parallel for the same number of resonators and / or different numbers of resonators, all seeking the best design for a given transfer function. By operating, for example, multiple systems, each operating on three, four or five resonators per transfer function, the described embodiments can determine the best circuit design using the optimal number of resonators for a target transfer function. At 320, a neural network type is selected. The use of a multilayer perceptron (MLP) or transformer policy network has been shown to be advantageous in testing scenarios. Other neural network architectures may also be used, including Recurrent Neural Networks (RNNs), or Convolutional Neural Networks (CNN). At 330, the size of the network is determined based on the number of choices (design dimensions) and fixed number of resonators selected at 310. At 340, multiple distributed circuit designs are generated. The method of circuit generation is shown in FIG.4 and an overview of a system for generating circuit designs is shown in FIG.5. At 350, each circuit design is evaluated and feedback provided to the network to improve the performance of the network for subsequent designs. At 360, the feedback is used to update the RL policy network and the method continues to generate additional design samples until a satisfactory circuit design which closely achieves the target metric is achieved. Once a target transfer function is achieved, circuits may be constructed based on the design using conventional manufacturing techniques.

[0035] FIG. 4 is a flowchart illustrating a method of steps 340, 350, and 360 of FIG. 3. FIG.5 is a block diagram illustrating the components of a system for performing the method of FIG. 4. At 405, for any distributed circuit having a fixed number of dimensions (such as circuit elements each having design parameters and locations), the design goal (e.g., a target transfer function for a distributed resonator circuit) is established at 410. At 425, interdependent mapping functions are created for the circuit of interest. Examples of interdependent mapping functions for a distributed square resonator circuity are discussed further below.FTRW-01414WO0 6000673PCT02 - 13 -

[0036] The subsequent disclosure will present embodiments where the distributed circuit is a distributed resonator circuit and the circuit elements are square resonators, however the embodiments are not limited to square resonators.

[0037] At 420, the neural network (Netθ) is configured with parameters and hyperparameters including the input I, the network (Netθ) input size, a target design goal (i.e. the target transfer function), a target time (or number of epochs) tτ, design weights W, the policy network type (Transformer, multi-layer perceptron, etc.), and batch size. At 425, the neural network is initialized based on a constant input I (described below) and an initial set of network weights θ0. At 430, the method consumes the input I and moves to continually produce sample configurations using the policy network 510 (FIG. 5), updating network weights and interdependencies using feedback provided by an evaluator (a simulator or estimator) until an optimal configuration is reached (or until an ending event (465) – a maximum run time or number of epochs, for example). At 430, a constant input I is input to the neural network and at 435 the neural network generates a batch of sample configurations in the form of conditional distributions of each of the fixed number of dimensions. At 440, the sample dimensions are mapped to physical representations of distributed circuits. In the case of a distributed resonator circuit, these are geometric representations of the circuit elements and positions. Configuring Netθis performed by auto-generating the sampling-policy neural network. Auto-generation is possible because the output parameters and elements are known, and the number of these parameters are known. Given that the input I is constant, various architectures (e.g., MLP, transformer, or the various types discussed above) can be used to transform the constant input (through multiple layers) into the desired output tensor. The weights of the neural network (Netθ) are adjusted through the reinforcement-learning-based training process (as described in PCT / US2023 / 070271 and PCT / US2022 / 026730), where the network learns to generate progressively better samples through repeated adjustment of these weights using an RL algorithm. Other network architectures are possible but in our formulation all such neural networks can be auto-generated, again, because we know the output size requirements and what they mean. It is desirable that the network be well-connected so that optimal conditional design sampling probability density functions (as a set of conditional density functions, that model theFTRW-01414WO0 6000673PCT02 - 14 - “joint” design probability density function) are possible to learn through cycles of sample generation and sample evaluation.

[0038] Optionally, at 445, anomalous designs in the physical circuit design may be detected. Anomalous design detection is discussed in patent publication PCT / US2023 / 070271. In embodiments, anomalous design as described herein is optional in the described embodiments as the mapping functions of step 440 generally eliminate any anomalous designs.

[0039] An anomalous design is one that would, for example, be physically impossible to construct, such as one where two physical square resonators are placed on top of each other. At 445, each sample physical configuration is evaluated by the design evaluator. In the case of a geometric distributed resonator circuit, this may include comparing the designs transfer function to the goal transfer function established at step 410. At 455, feedback for the design (the design’s transfer function) is returned to network 510 by the design evaluator to improve a next set of sample configurations generated by the network. At 460, the neural network (Netθ) 510 is updated and, unless an ending event 465 is detected, the process loops to step 440 until an optimal design 470 for the design space is produced. A completed design is achieved at 470 when the sample configuration which generates a physical configuration provides a circuit that meets the design goal (e.g., the design’s transfer function meets the target transfer function).

[0040] FIG.5 illustrates a system 500 for implementing the described embodiments. The system 500 includes a weight resonant neural network module 505, a mapping functions module 510, a detection module 515, an evaluation module 520 and a feedback module 525.

[0041] In the neural network module 505, a policy network 540 (Netθ) decodes the constant input I 530 as conditional distributions 545. Actions 550 are sampled from the distributions 545. The mapping functions module 510 includes mapping functions 555 which map the actions to a physical circuit design (i.e., a geometric representation of a circuit) 560. The detection module 515, optionally, uses anomaly detection techniques as discussed above and in conjunction with the evaluation module, which uses an electromagnetic simulator 565, to obtain a transfer function of the circuit design. The (calculated) transfer function is used inFTRW-01414WO0 6000673PCT02 - 15 - the feedback module 525 to compute rewards based on the target transfer function, with the rewards and losses 575 used to update the policy network.

[0042] FIG. 6 is helpful in understanding the system of FIG. 5. FIG. 6 illustrates a geometric representation of a distributed square resonator circuit. Each square resonator is described by 4 parameters: its center position (x; y), the angular position of its open slit, and its edge length a. The properties of a square resonator include length a ∈ ℝ with its center p = {(x, y) | x ∈ℝ, y ∈ ℝ } (where (x,y) is the center position of the square) in a plane and the direction of an open slit in the wall of the resonator, which can be facing up, down, left, or right. A one-hot vector u ∈ {0,1}4may represent the direction of a slit. The position of a slit is related to a vector s ∈ ℝ4,where an entry corresponding to the slit’s direction has a non-zero value and the rest of the entries can be zero. A distributed circuit with N resonators can be represented as (p, a, u ,s), where p ∈ ℝN×2, a ∈ ℝN, u ∈ {0,1}N×4and s ∈ ℝN×4. When the design evaluator is an EM simulator, it measures the distributed circuit by its transfer function ^̂^ଶ^= ^^^^p, a, u ,s) where ^^^ ∈ˆℂmand ℂ is a complex number. Each square resonator has aparameter vector (p, a, u, s). The error ^db between^ ^^ and the target transfer function y ∈ˆℂm indecibel (db) domain as:Equation 1

[0043] Returning to FIG.5, given a fixed number of resonators N (step 415) and a target transfer function s21 (step 410) without candidate templates, the system 500 generates a distributed circuit to meet the target transfer function s21. The system 500 has trainable parameters θ with a constant input I. Through decoding the constant input I, the framework learns to generate an optimal design of a distributed circuit with (p, a, u, s) = H(I, θ) (where H represents the function of system 500). The evaluator 570 computes the transfer function ^̂^ଶ^= ^^^^ p, a, u ,s). The goal of system 500 is to the error ^dbbetween the design’s transfer function^ ^^ଶ^and the target transfer function s21, such that the generated circuit meets the targetFTRW-01414WO0 6000673PCT02 - 16 - transfer function. Since the number of resonator N is defined, the DSE problem of generating distributed circuits can be solved by RL framework with definite horizon, which the number of actions / steps for completing a design are fixed. Considering that no reward is available until one achieves a complete design, the problem is formulates as a Single-Step Markov Decision Process (SSMDP) as s0 → A → sDin terms of compound actions A = {a1, a2, ..., aD}, where s0∈ S and sD∈ S represent the initial blank slate and the complete design (i.e., sD= (p, a, u ,s)), respectively. By tuning parameters θ related to the policy π that sculpts a reward R of the system 500, providing:Equation 2 where (S, A, R), Π∗is the set of a model’s optimal policies in response to the system 500 dynamics with the optimal θ∗, and L corresponds to the objective function related to ^db.

[0044] The weight resonant neural network 540 generates a sampling policy πθ. The policy has parameters θ: πθ = π(a1...aD|θ, I) where I ∈ ℝd. A constant input I 530 is the initial state of a design. Input 530 may remain the same and unchanged throughout the search for an optimal circuit design. In some embodiments, the input 530 may be a single constant value (e.g., a non- zero number). In some embodiments, the constant input 530 may be the tensor of 1s or hot-1s (e.g., a group of bits among which the allowable combinations of values are only those with a single high (1) bit and all the others low (0). The terminal state, sT = sD, is accomplished in a single step through a compound action A from πθin accord with the current policy = π(a1...aD|θ, I).

[0045] The output of the network 540 is a compound-action probability density function (PDF), or parameters of the PDF, as illustrated in FIG.7. FIG.7 illustrates a single step Markoff decision process reinforcement-learning model for design space exploration: s0 → X → sD where X is a compound action < a1, a2,・・・ , aD> and where the cost (penalty or reward) signal has all that is required to learn about compound-action inter-dependencies. Because theFTRW-01414WO0 6000673PCT02 - 17 - design space episode is of a fixed horizon (e.g., the number of the dimensions, or the number of the configuration parameters, is fixed), and because the reward is only collected at the end, the application scheme of RL can be considered as a single-step compound step action. The single-step compound action 604 in FIG. 7 is the combination of configuring all dimensions (e.g., the combination of “Configure C1 for dimension 1, Configure C2 for dimension 2, Configure C3for dimension 3, Configure C4for dimension 4, ..., Configure CT-1for dimension T-1, Configure CTfor dimension T). With the single-step compound step action 304, the application scheme is “stateless” in that there are no intermediate states. There are only two states in the improved scheme shown in FIG.7: a blank state 702 with no dimension configured and a fully configured state 706 with all dimensions configured. At the blank state 702, the single-step compound action 704 is performed. Then, the state transitions to the fully configured state 706, and reward RT is collected.

[0046] These PDF distributions can represent either discrete or continuous dimensions. For dimensions representing discrete dimensions, discrete probabilities required to establish the categorical distribution produced by the resonant policy network. For dimensions representing continuous dimensions, the policy network produces parameters for parametric modeling of those dimensions by appropriate analytic distribution functions (e.g., Gaussian, Beta, etc.). The probability distribution gives the possibility of each outcome of a random event (e.g., a possible option is configured for a dimension). The compound-action PDF is a function used to define the probabilities of different possible occurrences. A PDF sampler (not shown) generates sample design configurations in the form of compound actions.

[0047] The policy network 540 decodes the constant input into the joint “action” distribution π(a1...aD|θ, I), expressed in its conditional form:FTRW-01414WO0 6000673PCT02 - 18 - Equation 3 where the i-th design action aiis sampled from fiand fi∈ F. F includes beta distributions for sampling bounded continuous actions and categorical distributions for sampling discrete actions. Since the problem is related to RL with a fixed number of dimensions, a complete design can be achieved by applying D = 8N − 5 actions sampled from the jointly mixed action distribution, which can be represented by 20N − 12 parameters (i.e., a (20N − 12)- dimensional vector) output by the policy network.

[0048] In embodiments, multiple parallel instances of the system 500 of FIG. 5 may compute batches of configurations based on performance metrics defined by computing the transfer function of each design, consume batches of configuration samples and generate rewards based on the error between the transfer function of a sample configuration and the target transfer function. In this context, rewards 570 comprise a numerical value that provides feedback to an agent based on its actions and the current state of the environment. A reward can be positive or negative where positive rewards indicate beneficial actions that the model should try to repeat, while negative rewards indicate actions that should be avoided in the future. The evaluator module 520 computes the transfer function of fully configured designs and computes a single weighted sum, real number reward Rbfor each sample. The reward feedback 570 from batches of samples is used to update the weights of network 530 though the policy gradient RL algorithms.

[0049] The sampling policy πθ is continually updated as a collection of the conditional probabilities for each dimension by resonating on the constant tensor I, with the policy network designed with a sufficient capacity for internal resonance particularly among the dimensions so that inter-dependencies are learned through online exploration. The internal resonance causes the network to produce a joint distribution for an optimal design in the design space (or to approximate the optimal design) tending toward a multi-dimensional delta function ... , ^^^∗^ concentrated on an optimal point.

[0050] Updating the sampling policy πθis performed using proximal policy optimization (PPO) algorithms and is described further below.FTRW-01414WO0 6000673PCT02 - 19 -

[0051] Mapping module 510 takes sample configurations generated by network 530 (following sampling of the output distributions) and maps the compound actions to a pattern of distributed circuits with geometric representations (p, a, u ,s) within a circuit space M.

[0052] The mapping functions performed by the mapping module 510 may be understood with reference to FIGs.6, 8 ,9 and 10A-10D.

[0053] FIG.8 illustrates the inter-dependence of various sets of functions in the mapping functions module. As illustrated in FIG.8, the mapping module 510 takes a set of compound actions and determines individual lengths of resonators using a length mapping module 820. A slit mapping module 830 determines a resonator slit position. The offset mapping module 840 calculates offsets which define the positions of each resonator in a fixed number N of resonators in the distributed circuit, and as illustrated, is dependent on the output of the length mapping module 820. The boundary mapping module 850 determines the boundaries of available space from a potentially infinite exploration space for placement of the resonators. The interdependent function mapping module 860 maps the positions of sized resonators into the bounded space, resulting in a geometric representation of a resonator 870.

[0054] With reference to FIG.6, the position (u, s) of an open slit 610 and the length a of a resonator are shown. As noted above, the slit mapping module 830 will determine the slit position (u, s) by utilizing a discrete action auand a continuous action as. The action au∈ {0, 1, 2, 3} is sampled to determine a slit’s direction u ∈ {0, 1}4, with numbers 0, 1, 2, and 3 representing the directions up, left, down, and right, respectively. One can have u(i) = 1 when i = auand u(i) = 0 when iau. The position s ∈ ℝ4of a slit relative to the center of an edge can be determined by a hyperbolic tangent function tanh(·) with the action as∈ [0, 1]. Initial s(i) is set such that s(i) = ^ ଼tanh(2as− 1) when i = au, and otherwise s(i) = 0. In thelength mapping module 820, The length relates to an action al ∈ [0, 1]. All resonators in adistributed circuit have the same length. If the length of a resonator ranges from 50 to 100, the length a of all resonators can be a = 50(al + 1) with an action al ∈ [0, 1].

[0055] Next the positions of resonators are determined. Given the DSE exploration begins on a blank slate where a search space (the area on a substrate for location of a distributedFTRW-01414WO0 6000673PCT02 - 20 - circuit) could be infinitely large, precisely locating the position of each resonator on the slate can be inefficient. To reduce the search space of DSE, a boundary mapping module 850 within the mapping function module narrows the search space within boundaries.

[0056] With reference to FIG 10A, given a circuit space M, the length of square resonators a, a maximum gap g = agmaxwith a predefined gmaxand a defined number of resonators N, there exists an area bounding the centers of all resonators:Equation 4 with the maximum length of a distributed circuit of N resonators equal to B = aN + g(N − 1),where ^^ ⊆ ℬ. When defining the center of the leftmost resonator on x-axis can be 0, thecenter of the rightmost resonator on x-axis can be B − a. The center of the uppermost resonator on y-axis can be set as (B−a) / 2, the center of the lowermost resonator on y-axis can be (a−B) / 2. The distributed circuit is bounded by a boundary ℬ, where: ℬൌ ^^^^, ^^^^^^ ∈ ^െ ^ ଶ^ି^ ^ ^ଶ,ଶ^ , ^^ ∈ ^െଶ,ଶ ^^ Equation 5

[0057] Given the defined bounds ℬ and ^^, the mapping functions then calculate the relative positions of resonators using interdependent offset functions using the offset mapping module 840. The offsets define the positions of resonators relative to other resonators, and specifically the last resonator position. To precisely determine the position of each resonator within the boundary ℬ, a default center (x0, y0) for a leftmost (first) resonatoris set as (0, 0) as shown in FIG 10A. Subsequently, within the boundary ℬ , remainingresonators are added from left to right based on the previously placed resonator (hereinafter, the “previous resonator”). This structure of adding resonators based on the previous resonator using mapping functions is illustrated in FIG.9.

[0058] Actions af∈ {0, 1, 2}, aus∈ [0, 1] and aug∈ [0, 1] are related to the shift factor f,, the uniform shift factor usand the uniform gap factor ug, as f, us= ausand ug= augFTRW-01414WO0 6000673PCT02 - 21 - respectively, where f = 0 with af= 0, f = 0.2 with af= 1 and f = 0.5 with af= 2. A deviation function dsfor a shift, related to the position of a resonator, is defined as ds(l, r, us) = us(r − l) + l, where l = 0 and r = af. The predefined gminand gmaxare the minimum ratio and the maximum ratio of the gap, respectively.

[0059] With a default center of the first resonator, the in dterdependent functions mappingmodule 860 maps a resonator ^^ from actions ^^^௫ and ^^^௬to a position ^^^^ , ^^^^ ∈ ^^. An action^^^ௗ ∈ ^0, 1, 2^ for a resonator ^^ relates to its relative positions (i.e., up, down and right) to itsprevious resonator ^^−1 with position ^^^^ି^^^^ି^).into the circuit space M, interdependent functions are constructed which map actions about resonator i into (xi, yi) ∈ P. Note that (xi, yi) is a center of the resonator i and can be guaranteed to be within the boundary B based. An action a(i)∈ {0, 1, 2} about the resonator i corresponds to a relative position (i.e., up, down, and right respectively) of a current resonator compared with the last one. Since mapping the center of the rightmost resonator differs from mapping remaining resonators, the rightmost resonator is denoted as resonator n while remaining resonators except the leftmost one as resonator i, 1 ≤ i < n.

[0060] The structure of the mapping functions in the interdependent mapping module is illustrated in FIG.9. All resonators should be bounded within an area boundary ℬ. The initial resonator position 900 is defined at coordinates (0,0) (shown in FIG.10A and at reference 900 in FIG. 9). Each of the next N - 1 resonators are added, one-by-one, based on the previously positioned resonator though a series of mapping functions 910, mapping the actions 912, 914, 916 through transformation functions 922, 924, 926 into position coordinates 932, 934, 936. The final resonator is mapped by function 920 through a set of similar interdependent functions.The centers of the resonatorsexcept the right most resonator, can be determined for acurrent resonator i at functions 920 where ℎ^௫: ^^^௫ → ^^^ ^^^^^^ ℎ^௬: ^^^௬ → ^^^ . The center ofeach next resonator of N - 1 resonators (via functions at 920) in the distributed circuit are found as follows.

[0061] A current resonator is added above the previous resonator when ^^^ௗ ൌ 0. The centerof the current resonator in this case may be defined as ^^^ ൌ ^^^௫^^^^^^ ^^^^ି^ ^ ^^^, ^^ െ 2^^ െFTRW-01414WO0 6000673PCT02 - 22 - and ^^ ൌ ^^^^^^ ^^^ ^ ^^ ^ ^^ ,^ି^^ ^ି^ ^ଶ^ . For solving ^^^ , thetransformation function ℎ^௫is defined depending on the center of the lastresonator : ℎ^௫: ^^௫^^^௫ ^ ^^^௫ → ^^^ With the range of ^^^௫ from [0,1], ^^^ ൌ ^^^௫^^^^^^ ^^^^ି^ ^^^^, ^^ െ 2^^ െ ^^^^ ^ ^1 െ ^^^௫^^^^ି^ can be solved where ^^௫ is min^^^^ି^ ^ ^^^, ^^ െ 2^^ െ

[0062] A current resonator is added below the previous resonator when ^^^ௗ ൌ 1. Thecoordinate ^^^ in this resonator will still have the range ^^^ି^ ^ ^^^ ^ ^^^ ൌ ^^^^^^ ^^^^ି^ ^^^^, ^^ െ 2^^ െ ^^^^ ^ ^1 െ ^^^௫^^^^ି^ but ^^^ ൌ ^^^^^^ ^^^^ି^ െ ^^ െ ^^^,ୟି^ ଶ ^. With the rangeof ^^^௫ from [0,1], the center of the current resonator ^^^ ൌ ^^^௫^^^^^^ ^^^^ି^ ^ ^^^, ^^ െ 2^^ െ^^^^ ^ ^1 െ ^^^௫^^^^ି^ can be solved where ^^௫ is min^^^^ି^ ^ ^^^, ^^ െ 2^^ െ ^^^^ െ ^^^ି^ ,and ^^^௫ ൌ ^^^ି^.

[0063] A current resonator is added above the previous resonator when ^^^ௗ ൌ 2. The centerof the current resonator in this case may be defined as ^^^ ൌ ^^^^^^ ^^^^ି^ ^ ^^^, ^^ െ 2^^ ^ andwith the transformation functions summarized as:     FTRW-01414WO0 6000673PCT02 - 23 -

[0064] Functions 930 define the last, right-most resonator positions. As used herein, the right most-resonator is denoted with an n, while other resonators are denoted with an i. Thecenters of the resonators are bounded within an area ^^ ⊆ ℬ where the center can be found asfollows and inside the boundary ℬ.

[0065] The last (current) resonator can be placed above the previous resonator if ^^^ௗ ൌ 0,and the center of the output resonator is^ ^^^ ^ ^^^^^^ ^^^^ି^ ^ ^^^ , ^^ െ^^^ ^ ^1 െ ^^^௫^^^^ି^ and ^^^ ൌ ^^^^^^ ^^^^ି^ ^ ^^ ^ ^^^,^ି^^. To transform the actions intothe circuit space, the transformation function isto obtain ^^^ ൌ^^^௫^^^^^^ ^^^^ି^ ^ ^^^, ^^ െ ^^^ ^ ^1 െ ^^^௫^^^^ି^ where ^^௫ ൌ ^^^ି^ and ^^௫ ൌ ^^^^^^^^^^ି^ ^

[0066] If ^^^ௗ ൌ 1, the last (current) resonator is added below the previous resonator. Thecenter can be computed as ^^^ ൌ ^^^௫^^^^^^ ^^^^ି^ ^ ^^^ , ^^ െ ^^^ ^ ^1 െ ^^^௫^^^^ି^ but ^^^ ൌ^^^^^^^^^ െ ^^ െ^ with the same ^^௫and ^^௫transformation

[0067] The last resonator is added to the right of the previous resonator when ^^^ௗ ൌ 2.This resonator will still have the range ^^^ ^^ ^^^^, ^^ െ ^^൯ withtransform the actions into thespace, the transformation function is ℎ^௬: ^^௬^^^௬ ^ ^^^௬ → ^^^ to obtain ^^^ ൌWith the interdependent functions summarized as:FTRW-01414WO0 6000673PCT02 - 24 -

[0068] An example of the mapping of actions to physical space (i.e., constructing a physical circuit design) is illustrated with respect to FIGs.10A – 10D. As shown in FIG.10A, boundaries ℬ and ^^ are illustrated as well as the coordinates discussed above. As shown in FIG. 10B, the center of the first resonator (i.e., resonator “0) 1010 is oriented at (0, 0). Thecenter of the second resonator 1020, ^^^^, ^^^^ is located based on action ^^^௫ , ^^^௬ , ^^^ௗ and thecenter of resonator 0. Similarly, in FIG.10C, the center of resonator 1035 is based on ^^ଶ௫ , ^^ଶ௬, ^^ଶௗ and in FIG. 10D, the center of resonator 1045 is based on ^^ଷ௫ , ^^ଷ௬ , ^^ଷௗ . FIGS. 10B-10D also illustrate the gaps 1017, 1027, and 1037, and the shifts 1025, 1035 and 1045, between respective resonators.

[0069] As noted above, the sampling policy πθ of network 540 is continually updated as a collection of the conditional probabilities for each dimension by resonating on the constant tensor I, toward a multi-dimensional delta function ^^^^^∗ ∗ ∗^, ^^ଶ, ... , ^^^ ^ concentrated on anoptimal point. Updating the sampling policy πθis performed using proximal policy optimization (PPO) algorithms based on sample batches. The conditional PDFs Fi for each dimension i are used to sample a batch of possible designs ^^ (where the batch of possible designs is represented as < ξ1, ξ2, …, ξB >). where ξ = (p, a, u, s), i determined by mixed actions, which are sampled through Fi. The design sample ξ can be evaluated by evaluator 570 (an EMsimulator or estimator ^ ^^ (ξ )) and one computes the reward ^^^ from sample ^^^: ^^^ ൌ െ^^ௗ^.One defines ^^ as a collection of rewards from a batch of designs ^^, where ^^^∈^^. The reward may be computed as:FTRW-01414WO0 6000673PCT02 - 25 - Equation 6 A surrogate advantage ^^^scales the adjustments to the probability of a sampled design ^^^that led to a reward ^^^. In embodiments herein, the running reward is the exponential decay to the surrogate advantage ^^^which itself scales adjustments to the probability of a sampled design ξbthat led to the batch reward ^^^. This running reward, and the use of the surrogate advantage, eliminates the need for a value estimation network, and allows the network 540 to resonate to an optimal state. The surrogate advantage ^^_^^ for each member of the batch ^^^can be as ^^^ൌ^^^ െ ^^^ where a running reward i ^ ^^ with the renewal coefficient ^^^^^^௪ in the t-ith iterationcan be updated as:Equation 7 where ^^^^^^௪is a hyperparameter that controls the contribution of different terms in the running reward. The running reward, then, comprises an improving baseline for measuring the positive or negative advantage of generated design samples and provides the feedback to the PPO on how to adjust the probability density function.

[0070] In order to discourage anomalous designs, an appropriate reward ^^^is created to replace Rbfor anomalous cases:Equation 8 where ^^^is the anomaly punishment rate.

[0071] The policy gradient update algorithm thus uses the running reward ^^^ to reshape πθtoward an optimal sampling policy π*= πθ* =δ^^^∗^,A statistical risk Ltotal (^^) is comprised of a conditional update loss, a surrogate conditional entropy loss and a Kullback- Leibler (KL) divergence KL, rev-divergence. The conditional loss is given by:FTRW-01414WO0 6000673PCT02 - 26 -Equation 9

[0072] A KL divergence regulates how rapidly a policy is allowed to change and uses the sum of all conditional probability’s KL, rev-divergence loss as a surrogate KL regularizer:Equation 10 where ^^^,^is the beta factor for the revers KL divergence loss. In embodiments, the sum of all conditional probability’s KL is a rough estimate of the joint KL distribution and serves as a “surrogate” regularizer so that estimated conditional distributions do not change too quickly from one learning iteration to the next.

[0073] Finally, a surrogate conditional entropy loss is given by:Equation 11 where ^^^is the beta factor for entropy loss.

[0074] As the policy network produces sample configurations that approach optimal configurations, too great an entropy may adversely affect the policy into creating less optimal designs. Thus, in embodiments herein, an entropy penalty is designed to maintain entropy in the system high during the initial runs of the system and is adjusted to control the degree of exploration from an initial value to a minimum final value: ^^^,௧ = max(^^^^^, ^^^,௧ି^ ^ ^^ௗ^^^௬^Equation 12 βe, βmin and βdecay are used as hyper-parameters used to adjust entropy penalty ^^^,௧and control the degree of exploration, starting from an initial value and decaying to a minimum final value.FTRW-01414WO0 6000673PCT02 - 27 - And thus, the total statistical risk is:Equation 13

[0075] With the objective Ltotal (^^), an optimization algorithm (i.e., stochastic gradient descent) can be applied to perform back-propagation and update parameters in the neural network which gradually reshapes πθtoward an optimal policy π*θ. In each iteration of training, one can batch a group of Z design samples, partition the batch into z mini-batches and then set E epochs for training a batch.

[0076] FIG.11 is a table comparing the error ^dbof inverse designs on circuits with 3, 4, 5 and 6 resonators created by the disclosed method using a transformer-based network 540 with circuit designs produced by the techniques described in Zhang, G., He, H., and Katabi, D. “Circuit-GNN: Graph neural networks for distributed circuit design”, Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 7364–7373. PMLR, 2019 (hereinafter Circuit-GNN). In the table, the system operates on 3, 4, 5, and 6 resonators, respectively, (Column 1) in Circuit-GNN dataset which has 4, 10, 9 and 6 topology types (Column 2), respectively. Templates of circuits differ in the topology type and number of resonators. The types are identified in the CircuitGNN implementation where the authors considered variations of rules in relative placement of resonators. For example, one can have three resonators with different placements to build different types of distributed circuits, all of which consist of three resonators. Distributed circuits with the same number of resonators may have different transfer functions due to the different placement of resonators, with the various placements determining the best circuit using same number of resonators setting. In each template, there is a random sample of ten (10) transfer functions as target transfer functions. The Circuit-GNN in inverse design utilizes the topology types as one of the inputs which are invisible to experts at the beginning of the inverse design. Therefore, topology types for Circuit-GNN are masked during the inverse design. To make the comparison fair, the same pre-trained forward model was applied as an approximator (i.e., the evaluator) of the simulator to evaluate the transfer functions of all generated circuits. The errors ^db were evaluatedFTRW-01414WO0 6000673PCT02 - 28 - between the desired transfer function and the transfer functions derived from circuits generated by system 500, as well as those from Circuit-GNN. A lower value of V ^db indicates better generation performance. In the parameter settings, the default settings in Circuit-GNN were applied. During the training of a system, an ADAM (Adaptive Moment Estimation) is used optimizer with the learning rate 1 × 10−5and set iterations to 1500, which is the same as θ- Resonance. In each iteration, the batch size is set to 1024 with a mini-batch size of 512 and the number of epochs set to 1. The renew rate αrenewis set to 0.2, the anomalous rate to αa0.2, the βKL to 3, the βe to 1, the βmin to 0.02, and βdecay to 0.993.

[0077] The best results in FIG.11 are shown in bold, illustrating a reduction of error from comparisons between the method of the disclosed embodiments and the best results of baselines. transformer-based methods disclosed herein generate 3-resonator circuits with the best performance, whose transfer functions more closely align with the target transfer functions. The average error within each template for 3- resonator circuits is decreased by more than 27%, compared with that from Circuit-GNN. In the generation of the 4-resonator circuits, although the transformer-based system 500 decreases the average error by about 3% and 11% in topology type 2 and topology type 4 respectively, transformer-based system 500 outperforms Circuit-GNN in most topology types with the average error dropping more than 20%. When generating 5-resonator circuits, transformer-based system 500 has comparable performance as the Circuit-GNN in the topology type 0, 1, 2, and 6, where the average error diminishes less than 3%. However, in the remaining topology about 5-resonator circuits, the performance of generated circuits from transformer-based system 500 surpasses those from Circuit-GNN by reducing the average error by more than 7%. Compared with the average errors of generated circuits from Circuit-GNN, the average errors of the 6-resonator circuits from transformer- based DCIDA are reduced by more than 10% and the reduction can be up to 35%.

[0078] FIGs.12A through 12D are visualizations of the transformer-based performance of the disclosed embodiments vs. Circuit-GNN in achieving the target transfer function. FIG.12A illustrates a three transformer solution, with the geometric representation of the resonator produced by transformer-based system 500 on the left side of the figure next to its performanceFTRW-01414WO0 6000673PCT02 - 29 - graph, and Circuit GNN representation and performance illustrated on the right. Solid lines are transfer functions of the generated circuits from and dashed lines the ground truth for the transfer function. FIG.12B, 12C and 12D illustrates a four, five and six transformer solutions, respectively, in the same layout as FIG. 12A. As illustrated in these figures, the disclosed embodiments generate better inverse designs to meet the target transfer functions.

[0079] FIG. 13 illustrates a further embodiment where multiple systems 500 (500 x T, where T is any number of multiple systems) are run in parallel, with a fast feedback loop and a slow feedback loop. In this embodiment, multiple systems 500 x T are running in parallel with two different evaluator systems. One evaluator 520-1 may comprise an approximation of a simulator or EM estimator., such as a surrogate model. Surrogate models approximate the behavior of complex systems (such as full-wave EM simulators) while significantly reducing computation time. An EM simulator 520-2 simulator may be more accurate at computing the transfer function of a generated design while the approximator runs more quickly. This dual loop of optimization and a checking mechanism at each specific iteration can be used to stop some agents with lower performance. With a target transfer function ^^ଶ^, the system described in disclosed embodiments can run multiple systems in parallel to find the optimal quantity and parameters of resonators to generate distributed circuits. The dual loops of optimization improve the training speed and performance of each system 500 during the exploration. The framework can apply a checking mechanism at each specific iteration to stop some agents with low performance to accelerating the exploration and saving resources.

[0080] FIG.14 is yet another embodiment where, given a target transfer function, multiple parallel systems can run on different numbers N of resonators to determine the optimal number of resonators for a given target transfer function. FIG.14 shows multiple systems 500, with a first system 500-1 operating on a target of 3 resonators, a second system 500-2 operating on a second number of resonators and a third system 500-3 operating on a third number of resonators. In the figure, each system 500 runs for 1000 iterations before a determination is made as to whether any one of the systems has lower performance. In the figure, system 500- 1 is halted after this first check. After a second 1000 iterations, agent 500-3 is halted.FTRW-01414WO0 6000673PCT02 - 30 - Eventually, agent 500-2 completes a generated circuit closely matching the target transfer function.

[0081] FIG. 15 illustrates a network processing device 1500 which may be used to implement the disclosed embodiments, including system 500. It should be understood that while modules of system 500 are disclosed as incorporated into processing device 1500, modules of system 500 may be performed on different processing devices and, whether including all or part of system 500 on individual processing devices, multiples of device 1500 may be operated sequentially or in parallel in accordance with the techniques discussed herein. In embodiments, network processing device 1500 may contain multiple instances of a component, such as multiple processing units, processors, memories, interfaces, etc.

[0082] Device 1500 may comprise a central processing unit (CPU) 1510 (or multiple CPUs), a graphics processing unit (GPU) 1520 (or multiple GPUs), a memory 1525, a mass storage device 1530, and an I / O interface 1515 connected to a bus 1570. The I / O interface 1515 may be connected to one or more input / output peripherals. The bus 1570 may be one or more of any type of several bus architectures including a memory bus or memory controller, a peripheral bus, or the like. A network interface 1535 enables the network processing device to communicate over a network 1600 with other processing devices such as those described herein.

[0083] The mass storage device 1530 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus 1570. The mass storage device 1530 may comprise, for example, one or more of a solid-state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like. The mass storage device 1530 includes instructions which when executed by the CPU (or processor) cause the processor to perform the methods described herein. The mass storage 1530 may include code in the form of application modules and data stored thereon, comprising instructions for causing the CPU to implement the components of system 500 which are illustrated as present in memory 1525 in FIG.15.FTRW-01414WO0 6000673PCT02 - 31 -

[0084] The CPU 1510 may comprise any type of electronic data processor. Memory 1525 may comprise any type of system memory such as static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, memory 1525 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs. In embodiments, the memory 1525 is non-transitory. In one embodiment, the memory 1525 includes computer-readable instructions that are executed by the processor(s) 1510 and 1520 to implement embodiments of the disclosed embodiments.

[0085] In one embodiment, the mass storage 1540 instructions 1550 that are configured to cause the processor 1510 and / or the GPU 1520, when executing the instructions, to perform the functions described for such elements herein in instances of the modules of system 500. As such, in one embodiment, the memory 1525 includes one or more instances 505a of the resonant policy network module 505, one or more instances 510a of the mapping function module 510, one or more instances 515a of the detection module 515a, one or more instances 520a of the evaluation module 520a, and one or more instances 525a of the feedback module 525.

[0086] In one embodiment, the mass storage 1530 includes target transfer functions 1532 and output geographic circuit designs which may be final designs or configurations being processed by the system 500 toward reaching a final design.

[0087] As a consequence of the distinct features of the DSE system, the system scales to far larger design spaces that previous systems. For the purposes of this document, it should be noted that the dimensions of the various features depicted in the figures may not necessarily be drawn to scale.

[0088] The below pseudocode summarizes one embodiment of a method for generating distributed circuits. For generating a distributed with N resonators, a sample 8N – 5 actions are sampled from the joint distribution, and the actions mapped to the geometric representation of the distributed circuit. Given t iterations, E epoch, Z batch size and z mini batch size, theFTRW-01414WO0 6000673PCT02 - 32 - complexity of the algorithm can be O(௧^ாே௭ ). The method takes as input a target transfer function Y, the number of resonators N , constant tensor I, number of iteration t, number of epoch E, batch size Z and mini batch size z, a simulator or an approximator of a simulator Yˆ to output a generative circuit ξ. 1: Initialize a neural network Netθwith initial parameters θ and N 2: for i = 1 to t do 3: ^^ ← Netθ(I) 4: ^^ ← Sample(F) with Z times 5: ξb← Map(A) with Z batches 6: R ← Compute Reward(ξb) with Y and Yˆbased on Eq.(6)7: Save the best inverse design ξ with the best R ∈ ^^. 8: R̂ ← Comp^uாte Running Reward(R)9: for j = 1 to ௭ do 10: F′← Netθ(I) 11: A ← Compute Advantageఏe Loss (F, F′(R̂ , R)12: ^^ ← Comput , ^^, A) based on Eq.(13) 13: θ ← Optimize(^^ఏ)) 14: end for 15: end for 16: Obtain the generative circuit ξ

[0089] The disclosed embodiments improve upon the manual design and creation of distributed circuits. Engineers can apply the disclosed embodiments to obtain generated circuits with target transfer functions, which accelerates the development of distributed circuits. The disclosed embodiments can not only accelerate the development of circuits but saves the cost of development.

[0090] For purposes of this document, reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “another embodiment” may be used to describe different embodiments or the same embodiment.

[0091] For the purposes of this document, a connection may be a direct connection or an indirect connection (e.g., via one or more other parts). In some cases, when an element isFTRW-01414WO0 6000673PCT02 - 33 - referred to as being connected or coupled to another element, the element may be directly connected to the other element or indirectly connected to the other element via intervening elements. When an element is referred to as being directly connected to another element, then there are no intervening elements between the element and the other element. Two devices are “in communication” if they are directly or indirectly connected so that they can communicate electronic signals between them.

[0092] Although the present disclosure has been described with reference to specific features and embodiments thereof, it is evident that various modifications and combinations can be made thereto without departing from the scope of the disclosure. The specification and drawings are, accordingly, to be regarded simply as an illustration of the disclosure as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the present disclosure.

[0093] The embodiments described herein can be implemented using hardware, software, or a combination of both hardware and software. The software used is stored on one or more of the processor readable storage devices described above to program one or more of the processors to perform the functions described herein. The processor readable storage devices can include computer readable media such as volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer readable storage media and communication media. Computer readable storage media may be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by a computer. A computer readable medium or media does (do) not include propagated, modulated, or transitory signals.FTRW-01414WO0 6000673PCT02 - 34 -

[0094] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a propagated, modulated, or transitory data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

[0095] In alternative embodiments, some or all of the software can be replaced by dedicated hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), special purpose computers, etc. In one embodiment, software (stored on a storage device) implementing one or more embodiments is used to program one or more processors. The one or more processors can be in communication with one or more computer readable media / storage devices, peripherals and / or communication interfaces.

[0096] It is understood that the present subject matter may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this subject matter will be thorough and complete and will fully convey the disclosure to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications, and equivalents of these embodiments, which are included within the scope and spirit of the subject matter as defined by the appended claims. Furthermore, in the following detailed description of the present subject matter, numerous specific details are set forth in order to provide a thorough understanding of the present subject matter. However, it will be clear to those of ordinary skill in the art that the present subject matter may be practiced without such specific details.FTRW-01414WO0 6000673PCT02 - 35 -

[0097] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0098] The description of the present disclosure has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The aspects of the disclosure herein were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure with various modifications as are suited to the particular use contemplated.

[0099] For purposes of this document, each process associated with the disclosed embodiments may be performed continuously and by one or more computing devices. Each step in a process may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device.

[0100] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

FTRW-01414WO0 6000673PCT02 - 36 - CLAIMS What is claimed is:

1. A computer implemented method, comprising: generating distributed circuit configurations, each having a plurality of circuit elements positioned on a substrate, for an established target performance goal for a distributed circuit, by: receiving, by one or more computerized devices containing a neural network, a constant input for a distributed circuit design having a fixed number of dimensions; outputting, by the neural network, the fixed number of probability distributions; generating sample circuit configurations for the design based on the probability distributions, wherein each sample configuration of the sample configurations corresponds to a different configuration of the fixed number of dimensions; mapping each of the sample configurations to a physical representation of a potential distributed circuit; receiving feedback from an evaluator on each of the physical representations, the feedback based on a calculated error between the target performance goal and a calculated performance of each potential distributed circuit; updating parameters of the neural network based on a statistical loss derived based on each of the potential distributed circuits; and repeating the generating distributed circuit configurations until a distributed circuit configuration having a target calculated error is reached.

2. The computer implemented method of claim 1 wherein the mapping comprises applying mapping functions having an interdependence on each other.

3. The computer implemented method of any of claims 1 or 2 wherein said mapping includes defining a boundary in a potential circuit space and locating all circuit elements within the boundary.FTRW-01414WO0 6000673PCT02 - 37 - 4. The computer implemented method of any of claims 1 through 3 wherein said mapping includes defining a transformation function which maps actions produced by the neural network to coordinates in physical space in a potential circuit space.

5. The computer implemented method of any of claims 1 through 4 wherein said mapping functions place individual circuit elements in the potential circuit space based on a location defined for a previous individual circuit element.

6. The computer implemented method of any of claims 1 through 5 wherein the receiving feedback includes providing a fast feedback loop and a slow feedback loop, the slow feedback loop comprising an simulator calculating a performance output of sample physical designs, the fast feedback loop comprising an approximation of a simulator using a surrogate model to calculate the performance output of sample physical designs.

7. The computer implemented method of any of claims 1 through 6 wherein multiple instances of generating distributed circuit configurations occur in parallel, all seeking a circuit design with the same target performance goal.

8. The computer implemented method of any of claims 1 through 7 wherein the distributed circuit is a distributed square resonator circuit.

9. The computer implemented method of any of claims 7 or 8 wherein ones of the multiple instances of generating distributed circuit configurations occurring in parallel, all seeking a circuit design with the same target performance goal, use a different fixed number of resonators.

10. The computer implemented method of any of claims 7 or 8 wherein the fixed dimensions include parameters that define behavior a resonator include a length of a side ofFTRW-01414WO0 6000673PCT02 - 38 - the resonator, a distance between resonators, and a total number of square resonators in the circuit.

11. The computer implemented method of any of claims 1 through 10 wherein the neural network generates the probability distributions using a weight resonant policy network with single step reinforcement learning.

12. The computer implemented method of any of claims 1 through 10 wherein the method further includes receiving a design template describing a design space, including receiving a description of each discrete decision dimension and / or each continuous decision dimension; and generating a policy network in the neural network based on the design template.

13. The computer implemented method of any of claims 1 through 12 wherein the constant input comprising a single constant value, a tensor of 1s, or a tensor of hot 1s.

14. The computer implemented method of any of claims 1 through 13 wherein the updating the parameters of the neural network including updating weights of the neural network.

15. An apparatus, comprising: a storage medium comprising computer instructions; one or more processors coupled to communicate with the storage medium, wherein the one or more processors execute the instructions to cause the apparatus to generate distributed circuit configurations for an established a target performance goal for a distributed circuit, the distributed circuit having a plurality of circuit elements positioned on a substrate, the instructions causing the processor to: create a neural network comprising a single step reinforcement learning network. provide to the neural network, a constant input for a distributed circuit design having a fixed number of dimensions; output, from the neural network, the fixed number of probability distributions;FTRW-01414WO0 6000673PCT02 - 39 - generate sample circuit configurations for the design based on the probability distributions, wherein each sample configuration of the sample configurations corresponds to a different configuration of the fixed number of dimensions; map each of the sample configurations to a physical representation of a potential distributed circuit; generate feedback from an evaluator on each of the physical representations, the feedback based on a calculated error between the target performance goal and a calculated performance of each potential distributed circuit; update parameters of the neural network based on a statistical loss derived based on each of the potential distributed circuits; and repeating the generating distributed circuit configurations until a distributed circuit configuration having a target calculated error is reached.

16. The apparatus of claim 15 wherein the instructions cause the processor to map by applying mapping functions having an interdependence on each other.

17. The apparatus of any of claims 15 or 16 wherein the instructions cause the processor to map by defining a boundary in a potential circuit space and locating all circuit elements within the boundary.

18. The apparatus of any of claims 15 through 17 wherein the instructions cause the processor to define a transformation function which maps actions produced by the neural network to coordinates in physical space in a potential circuit space.

19. The apparatus of any of claims 15 through 18 wherein the instructions cause the processor to place individual circuit elements in the potential circuit space based on a location defined for a previous individual circuit element.

20. The apparatus of any of claims 15 through 19 wherein the instructions cause the processor to provide a fast feedback loop and a slow feedback loop, the slow feedback loopFTRW-01414WO0 6000673PCT02 - 40 - comprising an simulator calculating a performance output of sample physical designs, the fast feedback loop comprising an approximation of a simulator using a surrogate model to calculate the performance output of sample physical designs.

21. The apparatus of any of claims 15 through 20 wherein the instructions cause the processor to cause multiple instances of generating distributed circuit configurations to occur in parallel, all seeking a circuit design with the same target performance goal.

22. The apparatus of any of claims 15 through 21 wherein the distributed circuit is a distributed square resonator circuit.

23. The apparatus of any of claims 21 or 22 wherein the instructions cause the processor creating the multiple instances of generating distributed circuit configurations occurring in parallel, all seeking a circuit design with the same target performance goal, to use a different fixed number of resonators.

24. The apparatus of any of claims 15 through 23 wherein the instructions cause the processor to create the neural network configured to generate the probability distributions using a weight resonant policy network with single step reinforcement learning.

25. The apparatus of any of claims 15 through 24 wherein the constant input comprising a single constant value, a tensor of 1s, or a tensor of hot 1s.

26. A non-transitory computer-readable medium storing computer instructions for rendering images, that when executed by one or more processors, cause the one or more processors to perform the steps of: generating (410) distributed circuit configurations for an established target transfer function for a distributed circuit, the distributed circuit having a plurality of transformers positioned on a substrate, by:FTRW-01414WO0 6000673PCT02 - 41 - receiving, by a neural network, a constant input for a distributed circuit design having a fixed number of dimensions (430); outputting, by the neural network, the fixed number of probability distributions; generating, by the neural network, sample configurations for the design based on the probability distributions, wherein each sample configuration of the sample configurations corresponds to a different configuration of the fixed number of dimensions (435); mapping each of the sample configurations to a geometric representation of a potential distributed transformer circuit (440); receiving feedback from an evaluator on each of the geometric representations, the feedback based on a calculated error between the target transfer function and a calculated transfer function of each potential distributed circuit (445, 450); updating parameters of the neural network based on a statistical loss derived based on each of the potential distributed circuits(460); repeating the generating distributed circuit configurations to reach a target calculated error is reached.

27. The non-transitory computer-readable medium of claim 26 wherein said mapping comprises applying mapping functions having an interdependence on each other.

28. The non-transitory computer-readable medium of any of claims 26 or 27 wherein said mapping includes defining a boundary in a potential circuit space and locating all circuit elements within the boundary.

29. The non-transitory computer-readable medium of any of claims 26 through 28 wherein said mapping functions place individual resonators in the potential circuit space based on a location defined for a previous individual circuit element.

30. The non-transitory computer-readable medium of any of claims 26 through 29 wherein the receiving feedback includes providing a fast feedback loop and a slow feedback loop, theFTRW-01414WO0 6000673PCT02 - 42 - slow feedback loop comprising an electromagnetic simulator calculating a performance output of sample distributed resonator circuit designs, the fast feedback loop comprising an approximation of a simulator using a surrogate model to calculate the performance output of sample physical designs.

31. The non-transitory computer-readable medium of any of claims 26 through 30 wherein multiple instances of generating distributed circuit configurations occur in parallel, all seeking a circuit design with the same target transfer function.

32. The non-transitory computer-readable medium claim 31 wherein ones of the multiple instances of generating distributed circuit configurations occurring in parallel, all seeking a circuit design with the same target performance goal, use a different fixed number of resonators.

33. The non-transitory computer-readable medium of any of claims 31 or 32 wherein the fixed dimensions include parameters that define behavior a resonator include a length of a side of the resonator, a distance between resonators, and a total number of square resonators in the circuit.

34. The non-transitory computer-readable medium of any of claims 26 through 32 wherein the neural network generates the probability distributions using a weight resonant policy network with single step reinforcement learning.

35. The non-transitory computer-readable medium of any of claims 26 through 33 wherein the constant input comprising a single constant value, a tensor of 1s, or a tensor of hot 1s.

Images