Thermodynamic Artificial Intelligence for Generative Diffusion Models and Bayesian Deep Learning

Abstract

A physics-based system performs generative modeling for a given dataset by translating the diffusion process in a diffusion model into a physical process. An electrical circuit provides an exemplary implementation, where each unit cell (an electrical circuit within a network of electrical circuits) consists of a resistor, a stochastic noise source (such as a thermal or shot noise source), a programmable voltage source, and a capacitor (whose charge encodes a state variable). These unit cells can be capacitively coupled with connectivity that matches the problem geometry. A score network provides the prediction of score values ​​and can be implemented on digital, analog, or hybrid digital-analog devices. The detailed structure of the analog score network is provided in the form of a physical system that evolves over time simultaneously with the diffusion process. The analog score network can continuously provide score values ​​to the backward diffusion process without delay and efficiently evaluate the loss function when connected to the forward diffusion process.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority under 35 U.S.C. 119(e) to U.S. Application No. 63 / 483,856, entitled "Thermodynamic Artificial Intelligence System for Bayesian Deep Learning," filed February 8, 2023; U.S. Application No. 63 / 478,710, entitled "Thermodynamic Artificial Intelligence System for Generative Diffusion Models," filed January 6, 2023; and U.S. Application No. 63 / 385,891, entitled "Thermodynamic Artificial Intelligence for Generative Diffusion Models," filed December 2, 2022. Each of these applications is incorporated herein by reference for all purposes. [Background technology]

[0002] 1. Generative Models Recently, generative modeling (GM) has emerged as an application of machine learning (ML) and artificial intelligence (AI). For example, text-to-image applications of GM have captured users' imaginations, allowing them to generate their own artwork simply by inputting words. Also, generating seemingly realistic (but ultimately fake) images of human faces demonstrates the power of GM. The generation of text, speech, computer code, and molecular structures are additional applications of GM.

[0003] Generative models use a probabilistic framework and describe how a dataset is generated as a probabilistic model. New data can be generated by sampling from this model. There are several approaches to GM. Generative adversarial networks (GANs) were popular in the early days of GM. GANs use two neural networks operating in an adversarial setting where one network attempts to identify the output of the other network from real data samples. More recently, diffusion models have been introduced for GM and typically perform better than GANs.

[0004] 2. Diffusion Model Diffusion models (DMs), a class of physics-inspired models, have recently revolutionized the field of GM, becoming competitive with GANs and other GM methods in image synthesis, video generation, and molecular design.

[0005] 2.1 Score SDE The score SDE approach, a unified framework for diffusion models based on stochastic differential equations (SDEs), follows four steps: 1. Generate training data by evolving under a forward SDE and add noise to the data from the dataset of interest. 2. The data is used to train a neural network (the "score network") to match scores (the slope of the logarithm of the probability) associated with the distribution at each noise level. 3. Generate samples from the noisy distribution, i.e., the distribution associated with the final time point in step 1. 4. Evolve this sample under the backward SDE (defined using the trained score network) to generate new data points.

[0006] Once step 2 is complete, steps 3 and 4 can be repeated many times to generate many new data points. Figure 1 shows how steps 3 and 4 are repeated in the context of image processing.

[0007] In general, a forward SDE (step 1 above) takes the form:

number

[0008] The backward SDE (in step 4 above) reads as follows:

number

number

[0009] This general framework includes some interesting special cases. Variance preserving (VP) processes are continuous analogs of discrete Markov chains that are often used in denoising diffusion probabilistic models (DDPMs). The SDE of a VP process is:

number

number

[0010] A variance exploding (VE) process is a continuous version of the discrete Markov chain used in Score Matching with Langevin Dynamics (SMLD). The SDE of the VE process is:

number

number

[0011] Another special case is the integration of VP and VE processes for some time-dependent functions f(t) and g(t).

number

number

[0012] The VP and VE processes are special cases of these equations.

[0013] 2.2 Score Matching One subroutine in DM (see step 2 in the listing above) trains the score network and returns its output s θ (x,t) is the exact score ∇ x log p t (x). Ideally, the scores should match at all noise levels (i.e., all times t). To achieve this, the following loss function can be reduced or minimized with respect to the variational parameter θ:

number

[0014] This loss function is a weighted average of the input data x(0) and the noisy input data x(t)|x(0) drawn uniformly at different times t (conditional on the input data x(0)), where t is taken from the time interval [0,T] of the forward SDE. This loss function is expressed by the conditional distribution p associated with the forward process. 0t Use knowledge of the form (x(t)|x(0)).

[0015] In some cases, the form of the conditional distribution may be unknown. In such cases, an alternative loss function may be useful. Indeed, an alternative loss function based on the trace of the Hessian of the log-probabilities can be written as follows:

number

[0016] Both L1 and L2 loss functions are useful for score matching in diffusion models. In practice, these loss functions are estimated through finite sampling. Therefore, an unbiased estimator of the loss function (or its gradient) can be constructed using a finite number of samples to estimate the expected value of the loss function.

[0017] 2.3 Score Network A score network is a network whose output is a function s θ A trainable neural network, denoted by (x,t), that computes for each time instant t and each input vector x the true score ∇ x log p t The goal is to approximate (x).

[0018] On the other hand, the score network needs to be expressive enough to model complex functions. After all, the distribution p t(x) can be very complex (e.g., multimodal), and so can its associated score function. Therefore, a score network should usually have a large number of parameters to be expressive. The output s of the score network θ (x,t) can be a composition of an affine transformation and a nonlinear activation function. This is similar to how standard neural networks are constructed. Neural networks are good at approximating a variety of functions.

[0019] On the other hand, inductive biases can improve the performance of a score network, including its ease of training and generalization. In practice, inductive biases often correspond to considering the problem geometry when constructing a neural network. For example, the problem geometry may correspond to a grid (e.g., 1D, 2D, or 3D) or a graph. Implementing such geometric inductive biases is useful in fields such as molecular synthesis and materials design, where including symmetry as an inductive bias significantly improves the performance of diffusion models.

[0020] Therefore, the construction of the score network can depend on the nature of the problem. For example, when considering 2D images, it is common to use what is called a U-Net for the score network. U-Net was originally introduced for medical image segmentation and has a structure similar to convolutional neural networks. Such networks take into account spatial locality and the geometry of the problem.

[0021] In summary, the score network should be constructed to be expressive yet take into account the geometry of the problem.

[0022] 2.4 Latent diffusion model and spread spectrum model Some extensions of DM allow for pre- and post-processing of data. For example, the latent diffusion model (LDM) first passes the data through a trained autoencoder that transforms the data from data space to latent space, processes it with DM in the latent space, and finally reverses the autoencoder to return to data space. This reduces the computational difficulty of DM because the latent space has a lower dimensionality.

[0023] Similarly, the spectral diffusion model (SDM) first transforms spatial data into the spectral domain, then processes it with DM, and finally transforms it back again, allowing noise processes to be correlated in the spatial domain.

[0024] As a particularly interesting example, image generation approaches based on the heat equation and blur-diffusion can be thought of as exploiting uncorrelated noise in the frequency domain. According to this interpretation, the drift or diffusion terms should depend on frequency. In other words, the function f or G in equation (1) may have a nontrivial dependence on x.

[0025] Given the success of these methods involving data transformation, it makes sense to modify the general protocol for score SDE. Specifically, the score SDE protocol can be supplemented with additional steps: (0) preprocessing the data with an encoding process, (1)–(4) running the score SDE protocol on the encoded data, and (5) mapping the encoded space back to the data space.

[0026] 2.5 Probabilistic Flow ODEs Every SDE has an ordinary differential equation (ODE) flow associated with its underlying probability distribution. For a score SDE, the trajectory traced by x(t) follows a probability distribution pt(x) that governs its evolution as follows:

number

number

[0027] This can be used to obtain the ODE for the data x to read next.

number

[0028] This equation can be advanced forward and backward in time to accommodate the forward and backward processes. In practice, stochastic flow ODEs are sometimes used only for the backward process of sample generation because of the score function involved. Therefore, stochastic flow ODEs cannot be used for training procedures. In some cases, using stochastic flow ODEs for the backward process improves sample generation.

[0029] 2.6 Computational Bottlenecks Currently, the above framework for DM is implemented on standard digital hardware. This introduces several computational challenges: 1. A large amount of training data for training the score network; 2. Large complexity for training and evaluating score networks; 3. Long time to numerically simulate the dynamics of forward and backward SDEs or ODEs; 4. Potential instability of SDEs or ODEs due to large dimensionality and rigidity, leading to impractically small time steps. is.

[0030] The first two difficulties above are largely due to the lack of inductive bias in the problem formulation. Inductive bias is prior knowledge that is fed into the model's structure, such as knowledge about symmetries in the data. Having a strong inductive bias can reduce the training data requirements and improve the speed at which the model can be trained.

[0031] The third difficulty is the challenge of simulating time evolution using digital hardware. This includes both the challenge of digitally generating Gaussian random numbers (i.e., the dw term in SDEs) and the challenge of numerical time integration by discretizing the time evolution. Finally, the fourth difficulty refers to the many cases in which numerical integration of SDEs and ODEs can cause instabilities due to the structure of the equations, especially for SDEs with large dimensions. Another issue is known as stiffness, which can lead to small time steps and make numerical integration impractical. Summary of the Invention [Problem to be solved by the invention]

[0032] Diffusion models are influenced by physics because diffusion is a physical process, which raises the question of whether a physical process can mimic the mathematics used in the score SDE protocol described above.

[0033] Nature can perform calculations that would pose a significant computational burden on standard digital computers, and this paradigm of nature's computation motivates both analog and quantum computers.

[0034] Because diffusion is a thermodynamic process involving disorder, nature-based computers for this application can be thermodynamic, including physical disorder (e.g., due to thermal fluctuations). Thermodynamic systems can address the computational bottlenecks encountered when solving diffusion models on standard digital computers. In particular, physical systems can address the inductive bias and time-varying simulation problems mentioned above.

[0035] Examples of the inventive technique aim to remove the computational bottleneck faced by generative diffusion models implemented on purely digital hardware. These examples exploit the insight that diffusion is a natural process that occurs in many physical systems, including electrical circuits and continuous-variable optical systems. [Means for solving the problem]

[0036] An example of the inventive technology can be implemented as a system for running a generative diffusion model on a dataset. The system includes a physical system with several degrees of freedom associated with corresponding features of the dataset. Each degree of freedom has continuous state variables that evolve according to a corresponding differential equation with an adjustable diffusion term and an adjustable drift term. The degrees of freedom can be physically connected to each other according to the geometry of the problem associated with the dataset.

[0037] The system may further include a processor operatively connected to the physical system and configured to reduce the entropy of the continuous state variables to generate an output of the generative diffusion model. For example, the processor may reduce the entropy of the continuous state variables by reading the continuous state variables to modify at least one of the adjustable drift terms, by running a score network, or both. The processor may be configured to be trained by an optimization routine that reduces the loss function. The physical system may be configured to assist in estimating the loss function, or the processor may include analog circuitry configured to assist in estimating the loss function, or both. Estimating the loss function may include simultaneous time evolution of the physical system and analog circuitry within the processor. The loss function may quantify the degree of score matching.

[0038] The system may also include a function generator operatively connected to the physical system. During operation, the function generator multiplies the adjustable drift term and the adjustable diffusion term by an arbitrary time-dependent function to modify the differential equation governing the evolution of the continuous state variable. The system may include a digital device operatively connected to the physical system. During operation, the digital device uploads data sets to the degrees of freedom and downloads new data corresponding to measurements of the values ​​of the continuous state variable after the evolution of the continuous state variable.

[0039] The physical system includes a network of electrical circuits, each of which provides a corresponding degree of freedom and includes a capacitor having a charge encoding a continuous state variable, a stochastic noise source in series with the capacitor to generate an adjustable diffusion term, and a resistor in series with the capacitor to generate an adjustable drift term. Each electrical circuit may also include a first adjustable voltage source and a second adjustable voltage source operably connected to the stochastic noise source and the resistor, respectively. The first adjustable voltage source adjusts the adjustable diffusion term of the differential equation, and the second adjustable voltage source adjusts the adjustable drift term of the differential equation. The stochastic noise source may be a thermal noise source, a shot noise source, or both.

[0040] Alternatively, each electrical circuit in the network of electrical circuits may include a variable resistor in parallel with a capacitor and having a variable resistance controlled by an adjustable voltage source to adjust the adjustable drift term of the differential equation, and an amplifier operatively connected to the stochastic noise source to amplify the output of the stochastic noise source to adjust the adjustable spread term of the differential equation, The amplifier may have a variable gain determined by an additional variable resistor, the resistance of which is controlled by an additional adjustable voltage source.

[0041] The physical system may also include a switch that connects the electrical circuits in the network of electrical circuits according to a problem geometry associated with the data set. The physical system may also include a switch configured to operate between a setting for a forward diffusion process and a setting for a backward diffusion process.

[0042] The overall system may include a processor operatively connected to the network of electrical circuits to reduce the entropy of the continuous state variables and generate an output of the generative diffusion model. Again, the processor may reduce the entropy of the continuous state variables by reading the continuous state variables and modifying at least one of the adjustable drift terms. The processor may include an analog circuit configured to evolve over time simultaneously with the physical system. The analog circuit may be configured to continuously output a prediction of the score value as an analog signal used as an input to the physical system. The score value may be generated by physically modeling the partial derivative of the score value with respect to time and the continuous state variables using a trainable physical device. The trainable physical device used to model the partial derivative may be an artificial neural network. The processor may include an integration circuit operatively connected to the trainable physical device to generate the score value by integrating the output of the trainable physical device over time. The processor may further include a field-programmable gate array (FPGA).

[0043] The analog circuitry of the processor may include a network of electrical circuits, each electrical circuit in the network of electrical circuits providing a corresponding degree of freedom and including a capacitor, a resistor in series with the capacitor, and a voltage source in series with the capacitor. At least one electrical circuit in the network of electrical circuits may be capacitively coupled to at least one other electrical circuit in the network of electrical circuits according to a problem geometry associated with the dataset. The voltage source may be a multi-layer neural network configured to receive continuous state variables as inputs and output a voltage value for each electrical circuit in the network of electrical circuits.

[0044] All combinations of the above concepts, and additional concepts discussed in more detail below (to the extent such concepts are not inconsistent with one another), are contemplated as being included as part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being included as part of the inventive subject matter disclosed herein. Terms explicitly used herein should be given the meaning that most closely matches the specific concepts disclosed herein, even though they may also appear in disclosures made a part of this specification by reference.

[0045] Those skilled in the art will understand that the drawings are primarily for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale, and in some cases, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate a comprehension of different features. In the drawings, like reference symbols typically refer to like features (e.g., functionally similar or structurally similar components, or both). [Brief explanation of the drawings]

[0046] [Figure 1] 1 shows a diffusion model applied to an image. The forward diffusion process, or forward process, adds noise to each pixel, while the backward diffusion process, or backward process, removes noise from the image to generate new data points. [Figure 2]This is a schematic diagram of a thermodynamic AI device for simulating the forward process of a generative diffusion model. This physical system consists of multiple degrees of freedom (DOFs). For simplicity, four DOFs are shown here, but the number of DOFs generally corresponds to the dimensionality of the data, i.e., the number of features in the data. Each DOF has continuous state variables, which typically evolve according to a differential equation that includes both diffusion and drift terms. A function generator can multiply these diffusion and drift terms by arbitrary time-dependent functions hj(t) and kj(t), respectively, for the jth DOF. The problem geometry associated with a particular dataset can be uploaded to the device by selectively connecting the various DOFs, which mathematically connects the differential equations for the various DOFs. After some encoding, data points from the dataset of interest can be uploaded to the device by initializing the values ​​of the continuous state variables to the data point's corresponding feature values. Similarly, data can be downloaded (and decoded) from the device by measuring the values ​​of the continuous state variables after a certain time evolution. [Figure 3] This is a schematic diagram of a thermodynamic AI device for simulating the reverse process of a generative-diffusion model. In addition to all the device components present in the forward process, the reverse process uses a trained score network. The input to the score network is the value of the continuous state variables at time t, and the output is the score value. The jth component of the score, sj(t), is added as a drift term in the evolution of the jth DOF. The score network acts as Maxwell's Demon, continuously monitoring the physical system and adapting the drift term appropriately to reduce the entropy of the physical system. [Figure 4]The circuit diagram of a unit cell, the building block of an analog device, is shown. The voltage functions of k(t) and h(t) are selected by the user to allow for different drift and diffusion coefficients and are multiplied by the intrinsic circuit voltages using a voltage mixer (circle with crosses). We note that the circuit shown here is for solving the SDE formulation of the diffusion model. However, to solve the ODE formulation, we can remove the thermal noise source w shown here, or in other words, operate in a region where this thermal noise is negligible. [Figure 5] FIG. 1 is a circuit diagram of one unit cell of the forward process using variable resistors to control the drift and diffusion terms. [Figure 6] Two unit cells are shown connected together by a coupling capacitor C12, which is the basis for connecting unit cells in this analog device in general. [Figure 7] This is a circuit diagram of two capacitively coupled unit cells using variable resistors. The image above the main circuit diagram is a definition of the abbreviated circuit symbols used in the main diagram. [Figure 8] A simplified version of the circuit used in two unit cells experimentally implemented as a heat engine is shown. The plot shows that the voltage across the capacitors within the unit cell is strongly correlated with the large coupling capacitance. This suggests that the coupling capacitor effectively correlates the random walk (i.e., noise process) within the unit cell. This spatial correlation is desirable for designing an inductive bias in the model. [Figure 9] Four possible problem geometries are shown, but there are others: DNA sequences have one-dimensional geometry, images have two-dimensional geometry, solutions to partial differential equations (PDEs) in real space (e.g., fluid flow) often have three-dimensional geometry, and molecular structures have a certain graph connectivity and therefore follow a graph geometry. [Figure 10]We show the mapping of the incidence matrix to hardware. Each off-diagonal element of the incidence matrix is ​​converted to the state of a switch in the wire connecting two unit cells. For simplicity, the capacitors in series with the switches are not shown. For the hidden layer network, the switches are placed in series with the resistor bridges connecting the unit cells. For the pre-weight spreader and post-weight spreader, the switches are placed in series with the capacitor bridges connecting the unit cells. [Figure 11] A comparison of a typical scenario for Maxwell's demon (left panel) and this scenario (right panel) is shown. In this scenario, the voltages across the capacitors in an electric circuit act as dynamic variables, analogous to the positions of the gas particles in the left panel. Even if these voltages start out in a high-entropy state (represented by different capacitors in different colors), they can evolve over time to a low-entropy state (represented by a single color for the different capacitors). This reduction in entropy is facilitated by an intelligent observer (also known as Maxwell's demon) that continuously observes the state of the system and adjusts the applied voltages in each circuit appropriately. [Figure 12] Schematic diagram showing how a score network, stored and executed on a digital device such as a central processing unit (CPU) or field programmable gate array (FPGA), interacts with a full set of analog unit cells via analog-to-digital converters (ADCs) and digital-to-analog converters (DACs). [Figure 13A] To solve the backward SDE, we show that a digital score network (SN) is connected to an analog unit cell, and the output of the SN is multiplied analogically by the function g(t)2. [Figure 13B] We show that to solve the backward SDE, we connect a digital SN to an analog unit cell, where the output of the score network is multiplied by a digital device. [Figure 14] 1 is a flowchart of a thermodynamic AI system when the score network is already pre-trained. [Figure 15]This is a flowchart of the thermodynamic AI system when the score network is untrained. [Figure 16] 1 is a schematic circuit diagram of a score device, which illustrates the process used to obtain score values ​​during the development of the reverse process. [Figure 17] The circuit diagrams of a voltage adder (left) and a voltage integrator (right) are shown. [Figure 18] A simple circuit is shown that provides a subroutine to compute the square of the norm of a voltage vector v = {v1, v2, ..., vd}. [Figure 19] FIG. 1 is a circuit diagram of a root-mean-square (RMS) converter based on a thermal method. [Figure 20] FIG. 1 is a schematic diagram illustrating the training process of an analog score device. [Figure 21] This is a flowchart of a thermodynamic AI system where the score network is an analog device used to interface with the evaluation of the loss function during training and the backward process after training. [Figure 22A] One unit cell of the analog core device is shown. [Figure 22B] 1 shows two unit cells of an analog core device capacitively coupled to each other. [Figure 23] A layered analog neural network for evaluating the function r(i) is shown (also applies to evaluating q). An N+1 dimensional input (v(t),t) is fed into a parameterized layer A1 detailed in Figure 24. After each linear layer, the output is fed into a diode, followed by a resistor whose current output is a nonlinear function fd of the voltage. Here, the last nonlinear layer is shown and denoted BK. [Figure 24]The first resistive layer of an analog neural network is shown. (All resistive layers have the same general form, possibly with different hyperparameters.) Each input voltage is copied M times and fed to Mxd+1 resistors, and its inverse becomes the entry (A)j,k in matrix A. This circuit generates a weighted average of the input voltages stored in each A[v(t),t]j, which is then fed to the nonlinear layer. Although not shown, amplifiers can be added to each layer to boost the output voltage and prevent voltage attenuation from layer to layer. [Figure 25] This is a schematic diagram of a basic voltage follower. Following the two "golden rules" of operational amplifiers, V input = V output, and simultaneously, no current is flowing into either operational amplifier input. [Figure 26] FIG. 10 is a schematic diagram of a circuit for calculating the dot product of the input voltage and A1 of row jth. [Figure 27] Four cases of the backward diffusion process and its communication with the score device are shown (top), along with a simplified circuit symbol for a single unit cell (N=1) (bottom). These cases include cases where the score device is analog or hybrid digital-analog, as well as the possibility that the unit cell is based on a voltage mixer or variable resistor. [Figure 28] Shown is a circuit diagram of a universal unit cell that can be used for both the forward and reverse processes. The thick lines through a particular wire indicate that a switch is added in series with that wire. When these switches are all open (closed), the unit cell corresponds to that used in the forward (reverse) process. This allows the same device to be used for both the forward and reverse processes. [Figure 29] We show how Bayesian neural networks (BNNs) differ from standard neural networks and show that the weights of BNNs have several probability distributions. [Figure 30] Showing multimodal distribution: The posterior distribution of BNN weights is usually multimodal. [Figure 31] It shows how the four components or subsystems of a BNN interact and signal each other. [Figure 32] 1 shows a unit cell of a hidden layer network (HLN) including a capacitor Cj, a resistor Rj, a non-linear element (NLE) connected in parallel with the capacitor, and a voltage source αj. [Figure 33] Shown are two HLN unit cells connected via a resistive bridge. [Figure 34] We present an analog augmented neural ODE that can be used for HLN. [Figure 35] 1 shows the unit cell of the pre-weighted diffuser. [Figure 36] 1 shows two unit cells of a pre-weight spreader connected via a capacitive bridge. [Figure 37A] The unit cell of the posterior weight spreader is shown when the posterior drift network (PDN) is an analog device. The diagram shows a feedback loop where the voltage across the capacitor is measured and fed as an input to the PDN, which then applies the appropriate drift voltage to the unit cell. [Figure 37B] 1 shows a unit cell of the a posteriori weight spreader when the PDN is a digital device such as an FPGA. [Figure 38] We show that for two unit cells of the weight spreader, we output weights from the weight spreader device to the HLN device. This concept can be applied to any number of unit cells. [Figure 39] We show that for two unit cells of the HLN, the weights are input from the weight spreader device to the HLN device. This concept can be applied to any number of unit cells. [Figure 40]This is a circuit diagram showing that the same device can be used for pre- and post-weight spreaders. The thick line represents a switch that allows switching between two different spreaders. When the switch is open (closed), the device corresponds to a pre- (post-) weight spreader. [Figure 41] If the posterior drift network is a digital device such as an FPGA or a CPU, a feedback process between the posterior diffuser and the posterior drift network is shown. [Figure 42] Depict a layered analog neural network configured to evaluate functions r(Φ), q(Φ), or s(Φ) in the context of a PDN. A W+1-dimensional input (w(t), t) is fed into a parameterized layer corresponding to the affine matrix A1 detailed in Figure 43. After each linear layer, the output is fed into an NLE, followed by a resistor whose current output is a nonlinear function of voltage. Here, the last nonlinear layer is denoted as BK. [Figure 43] The first resistive layer of an analog neural network is shown. (Resistive layers have the same general form, possibly with different hyperparameters.) Each input voltage is copied M times, and the resulting value can be fed to MxW+1 resistors, with the inverse value becoming the entry (A)j,k in matrix A. This circuit generates a weighted average of the input voltages stored in each A[w(t),t]j, which is then fed to the nonlinear layer. Although not shown in the diagram, amplifiers can be added to each layer to boost the output voltage and prevent voltage attenuation from layer to layer. [Figure 44] FIG. 10 is a schematic diagram of a circuit used to integrate the total derivative of the drift to output a drift value from the PDN. [Figure 45] FIG. 10 is a circuit diagram for estimation of the λ term in the loss function using digital time integration. [Figure 46] FIG. 10 is a circuit diagram for estimation of the λ term in the loss function using analog time integration. [Figure 47]We show a unit cell for an analog neural ODE using voltage mixers, which allows for reversing the direction of time based on the choice of function k(t), e.g., by choosing k(t) → -k(1-t). [Figure 48] Two unit cells for an adjoint device are shown. Each cell contains a resistor and a capacitor, and the cells are connected via a resistive bridge. The adjoint device evolves an adjoint variable a over time, which can be encoded into a voltage across a capacitor in the device. This is done in the context of the adjoint sensitivity method for computing the gradient of a neural ODE. [Figure 49] The circuit diagram of the integrator circuit is shown. [Figure 50] 10 shows an analog latent ODE for fitting and extrapolating time series data. [Figure 51] 1 shows a unit cell of an analog neural SDE processor. [Figure 52] We present a variety of algorithms unified under a single mathematical framework of thermodynamic AI algorithms. [Figure 53] FIG. 1 is a circuit diagram of a physical realization of s-mode including a noise resistor and a capacitor. [Figure 54] 1A and 1B are circuit diagrams of the physical realization of the connection between s-modes using coupling resistors and coupling capacitors, respectively. [Figure 55] We present a force-based approach to constructing Maxwell's demon devices. DETAILED DESCRIPTION OF THE INVENTION

[0047] 3 General Physics-Thermodynamics Artificial Intelligence (AI) Devices Figures 2 and 3 are conceptual diagrams illustrating the general physical architecture of a thermodynamic AI device. Figure 2 shows a thermodynamic AI device 200 that performs the forward process, and Figure 3 shows a thermodynamic AI device 300 that performs the reverse process. This architecture can be implemented in an electrical circuit, as described below, or in other physical systems, such as continuous variable optical systems.

[0048] As shown in FIG. 2, the physical system 200 includes multiple degrees of freedom (DOFs) 210-1 through 210-4. The number of DOFs 210 corresponds to the dimensionality of the data, i.e., the number of features in the input dataset 201. Each DOF 210 has continuous state variables that typically evolve according to a differential equation with both a diffusion term and a drift term. Function generators 220-1 and 220-2 can multiply these diffusion and drift terms by any time-dependent function. The problem geometry 203 associated with a particular dataset can be uploaded to the device by selectively connecting the various DOFs 210 with switches 230, which mathematically connects the differential equations of the various DOFs 210. After some encoding, data points from the dataset of interest 201 can be uploaded to the device by initializing the values ​​of the continuous state variables to the data point's corresponding feature values. Similarly, new data 209 can be downloaded (and decoded) from the device by measuring the values ​​of the continuous state variables after a certain time evolution.

[0049] Furthermore, the backward process uses a score network 340 trained as shown in Figure 3. The inputs to the score network are the values ​​of the continuous state variables at time t, and the output is the score value. The jth component of the score is added as a drift term in the evolution of the jth DOF. The score network 340 acts as a Maxwell's demon, continuously monitoring the physical system and adapting the drift term appropriately to reduce the entropy of the physical system.

[0050] 4 Electrical Circuit Thermodynamic AI Devices Thermodynamic AI devices can be implemented as hybrid analog-digital systems with thermodynamic properties. This analog system generates dispersion via a thermal noise system, such as an electrical resistor. As mentioned above, Gaussian randomness is costly to generate digitally and is therefore better generated in an analog system. Furthermore, time dynamics are performed on analog devices, for example, via the natural time evolution of the voltage of an electrical capacitor. This addresses the computational bottleneck of numerical integration of dynamics by digital solvers, as analog systems naturally perform this integration.

[0051] Analog systems are composed of repeated subunits, or unit cells, with the number of unit cells equal to the dimension of the problem the analog system is designed to solve. Each unit cell consists of a thermal noise source, a resistive element, and a capacitive circuit element. Furthermore, each unit cell can in principle be connected (via capacitive bridges) to every other unit cell. An arbitrary connectivity matrix (similar to an adjacency matrix in graph theory) describes how unit cells connect to each other. This allows the connectivity to be tailored to the problem geometry.

[0052] As mentioned above, inductive bias reduces training data requirements and training complexity. This system allows users to incorporate inductive bias into their models. In particular, the connectivity matrix should be closely related to the problem geometry to maintain a strong inductive bias. Indeed, in this system, a simple switch on the connection bridge enables connections to be turned on and off, allowing users to upload the problem geometry to an analog device.

[0053] The system can be operated in a pre-training mode, where the score network is already trained, or in a training mode, where the score network is first trained using training data generated by the analog device. In this system, the digital device uploads data to the analog device by appropriately charging the capacitors in each unit cell. The charges on these capacitors become the dynamic variables of interest that evolve under the corresponding forward or reverse process.

[0054] One aspect of this system is rooted in the thermodynamic concept of Maxwell's demon. While the second law of thermodynamics dictates that overall entropy does not decrease over time, Maxwell's demon can locally decrease the entropy of a system by observing the system and adaptively interacting with it. An electrical version of Maxwell's demon can reduce the entropy of a collection of unit cells within this analog device, specifically by observing the charge on the capacitor within the unit cell and adjusting the voltage applied within the cell. As a result, this system can physically implement transitions in the inverse SDE, which tend to reduce entropy and, at first glance, appear to contradict the second law of thermodynamics. This Maxwell's demon can be implemented as a digital device, such as a central processing unit (CPU) or field-programmable gate array (FPGA), that stores a trained score network and continuously interacts with the analog device. In this sense, Maxwell's demon (or digital score network) acts as an AI agent that intelligently interacts with thermodynamic physical systems. The analog system and the digital AI agent / Maxwell's demon together form a thermodynamic AI system for generative modeling.

[0055] 5 unit cells 5.1 Warm-up Exercise: RC Circuits Before introducing this unit cell, let us first consider a simple RC circuit, a circuit consisting of a resistor R and a capacitor C connected in series. The current law states that the current I in the resistor branch R is the current I in the capacitor branch C should be equal in magnitude and opposite in sign. Therefore, I R =-I C The voltage law states that the voltage v across both branches R and v C states that the sum of v and v should be zero, and therefore v R =-v C This becomes:

number

[0056]

number

[0057] Transforming it into differential form, we get:

number

[0058] This provides a simple building block for first-order differential equations that are solved by analogy.

[0059] 5.2 Unit cell containing a voltage mixer and a fixed resistor To build a circuit for an SDE, a Brownian noise source is needed. This can be provided by a second resistor R', whose noise is assumed to be much higher than R, so that R does not generate a voltage noise and R' generates a voltage noise v w Therefore, consider a circuit with a capacitor C, a resistor R, and a resistor R' connected in series. For this circuit, obtain the following differential form:

number

[0060] However, these terms are not time-dependent and may not be general enough to simulate SDEs with the form of the score SDE (described in Section 2.1).

[0061] Figure 4 shows an RC circuit with two voltage mixers that add the time dependence of the drift and diffusion terms. The two voltage mixers multiply the circuit voltage across resistor R by time-dependent voltages k(t) and h(t).

[0062] Next, consider a complete analysis of the circuit of Figure 4. In this analysis, we define w as the noise voltage v w voltage noise sources with voltage v k and v h It is assumed that the analog mixer is an ideal multiplying mixer, i.e., the output voltage v 出力 (v a ,v b ) is the voltage between the two input voltages v a and v b is equal to the product of v 出力 (v a ,v b )=v a v b is.

[0063] The voltage law for the main loop reads as follows:

number

[0064] The ideal mixer voltage is defined as:

number

number

number

[0065] The mixer voltage is:

number

number

[0066] The voltage law now reads as follows:

number

[0067] Using Ohm's law, the current-voltage relationship for a resistor and capacitor is:

number

number

[0068]

number

number

[0069] Substituting this into equation (23) gives:

number

[0070] voltage source v h The current I passing through h teeth,

number

number

[0071] The stochastic flow ODE can also be implemented with an RC circuit corresponding to equation (16) by adding a voltage mixer to the voltage at the point between the resistor and the capacitor. Adding this voltage mixer results in the following ODE:

number

[0072] 1 / 2G(t) 2 s θ By adding a second voltage (related to the second term in equation (12)) equal to , we can generate new samples simulating the behavior of the stochastic flow ODE shown in equation (13). This is explained in more detail in Section 11.2 for N unit cells.

[0073] 5.3 Unit Cell with Time-Varying Resistor An alternative version of this unit cell does not use a voltage mixer. Indeed, avoiding the voltage mixer reduces circuit complexity, since a voltage mixer contains multiple components.

[0074] Instead of using a voltage mixer, a time-varying resistor can be used to introduce time dependence into the drift and diffusion terms. One way to build a time-varying resistor is to use a field effect transistor (FET) or a network of FETs.

[0075] Using a FET, the resistance of the FET can be controlled using the voltage applied to the gate of the FET: applying a time-varying voltage to the gate of the FET results in a time-varying resistor within its linear operating range.

[0076] Voltage-controlled resistors, such as FETs, can also be used to manipulate the gain of a simple non-inverting amplifier. This is possible because the voltage gain of this amplifier is determined by a resistive voltage divider, so if one of the resistors is time-varying, the gain will also be time-varying.

[0077] Consider the circuit in Figure 5. In this circuit, the first FET acts as a voltage-controlled resistor in the feedback network of a non-inverting amplifier, controlling the noise source (the diffusion term). The second FET in parallel with the capacitor controls the drift term.

[0078] The analysis of the voltage law is as follows:

number

number

[0079] The analysis of the current law is as follows:

number

[0080] Using equations (32) and (33) together with equation (34) and the current-voltage relationships of the resistor and capacitor, we obtain:

number

[0081] The final noise voltage after amplification is:

number

[0082] Using this relationship and rearranging, we end up with the following:

number

[0083] In the above, C=dv C / dt was used. Rearranging this, we get the following SDE:

number

[0084] 5.4 Alternative noise sources As mentioned in the previous section, the implementation of an SDE in electronics involves a source of Gaussian noise in voltage or current. Generally, the amplitude of this noise should be controllable independently of other circuit parameters. The circuits presented in this document used an arbitrary noise source. In practice, this arbitrary noise source can be of many different types.

[0085] Specifically, the circuit diagrams of Figures 4 and 5 include an abstract noise source denoted as w. Physically speaking, a physical circuit element can represent the abstract noise source given by w. A thermal noise source or a shot noise source may be a physical realization of w.

[0086] One type of noise that is possible is thermal noise. Thermal noise, also called Johnson-Nyquist noise, arises from the random thermal oscillations of charge carriers in a conductor, resulting in fluctuations in the voltage or current in the conductor. Thermal noise is Gaussian distributed, with a flat frequency spectrum (white noise) that fluctuates with the standard deviation of the voltage.

number

[0087] Another type of noise is shot noise. Shot noise arises from the discrete nature of charge carriers and the fact that the probability of a charge carrier passing a point in a conductor at any given time is random. This effect is particularly pronounced in semiconductor junctions, where charge carriers must overcome a potential barrier to conduct current. The probability of a charge carrier crossing the potential barrier is an independent random event, which causes fluctuations in the current through the junction. Shot noise is Gaussian distributed and has a flat frequency spectrum (white noise) with a standard deviation of the current fluctuations.

number

[0088] 6 Two coupled unit cells As mentioned above, incorporating the geometry of the problem into the noise process helps reduce the burden on the score network, which can be seen as providing an inductive bias to the model. To incorporate geometric information (or geometric correlations) into analog circuits, correlations are required between cells in the drift and diffusion terms. These correlations are achieved by capacitively connecting unit cells.

[0089] Consider the circuit of Figure 6, where two unit cells are connected by a coupling capacitor C 12 In the following derivation, we analyze the case where the unit cell consists of a multiplying mixer. However, the form of capacitive coupling also extends to two coupled cells with variable resistors. Figure 7 shows the circuit diagram associated with two coupled cells in the case of variable resistors.

[0090] Analyzing the circuit in Figure 6 using the methods in Section 5.2 yields the following set of coupled SDEs:

number

number

[0091] To simplify notation, these equations can be written in matrix form as follows:

number

number

[0092] Since the off-diagonal elements of the capacitance matrix C are not zero, we obtain a coupled system of equations in which both the drift and diffusion terms are correlated. Each cell has a control voltage h i (t) and k i (t), where i∈{1,2}, and the drift and diffusion terms of each cell remain independently addressable by appropriate choices of h(t) and k(t).

[0093] A simplified version of this circuit is used (see Figure 8) to study the voltage correlation between two capacitor-connected RC circuits. There is a strong correlation between the random walks of the capacitor voltages in the two circuits, suggesting that this (somewhat complex) circuit leads to a strong correlation between the unit cells.

[0094] 7 Problem geometry and incidence matrix As shown in Figure 9, different problems have different geometries. Time series data and DNA sequence data are both one-dimensional (1D), and these datasets have a 1D notion of locality. Images are 2D, and solutions to partial differential equations in real space are often 3D. Other problems, such as molecular structure problems, are best represented on graphs rather than lattices.

[0095] Ideally, we would like to account for the problem geometry in the noise processes, i.e., the drift and diffusion terms that appear in the partial differential equations. This reduces the burden on the score network, so that it is not forced to generate all correlations in the data. Ultimately, this allows the score network to train faster and requires less training data.

[0096] Now consider the geometry of the problem in the context of an array of unit cells, where each unit cell is configured as shown in Section 5. Specifically, consider N RC cells, and pairs of these cells may be capacitively coupled as in Section 6.

[0097] The connectivity of such a circuit can be represented through a combinatorial graph G(V,E). The connectivity between vertices V is represented by the adjacency matrix A of the graph G(V,E). If A ij If =1, cells i and j are connected, and A ij If =0, the pair is not connected.

[0098] Such connectivity (e.g., capacitor G in FIG. 6) 12 This is realized in the physical circuit by including a switch on each bonded wire (shown in FIG. 10 below and in the wires containing the wires containing the wires containing the wires containing the wires containing the wires containing the wires containing the wires). These switches can be implemented using transistors in the physical device and therefore can be voltage controlled.

[0099] These switches can be open or closed depending on the adjacency matrix A. Given the adjacency matrix A, the 2x2 capacity matrix defined for the two-cell case generalizes to an NxN matrix,

number

[0100] In many applications, the geometry of a circuit is specified by a sparse adjacency matrix. For example, if N cells are arranged on a d-dimensional lattice, each cell is connected only to its nearest neighbors, and each unit cell has 2d nearest neighbors. d is usually much smaller than N, so the adjacency matrix is ​​sparse. For example, if d=1, the matrix is ​​tridiagonal.

[0101] 8 Differential Equations for Forward Processes Combining ideas from the previous section, it is possible to construct a circuit of N unit cells connected according to an arbitrary adjacency matrix A, enabling the physical simulation of SDEs commonly used in generative modeling. In this section, we show how such an analog system can be utilized to simulate the forward process of a generative diffusion model.

[0102] The first step in simulating a forward SDE is to upload data points to the analog device. This is a set of data points x = {x1, x2, ..., x} consisting of N feature values. N} into an N-dimensional voltage vector v(t)={v1(t),v2(t),…,v N (t)}, where entry v i (t) is the voltage across the capacitor of the i-th unit cell at time t. Specifically, in the forward process, a data point is uploaded at an initial time t=0, which results in a direct mapping x for each i. i →v i (0). The nature of this mapping from data feature values ​​to voltages varies depending on the dataset; for images, pixel values ​​can be converted to voltages within a specific voltage window. The forward process evolves a vector v(t) over time, and eventually at a later time t, this vector is downloaded and reconverted into a vector of feature values.

[0103] Next, we analyze the case of a unit cell composed of a voltage mixer and a fixed resistor, but the treatment in this section is straightforward to extend to other variations of unit cells, such as unit cells based on variable resistors.

[0104] Each unit cell provides one Brownian source voltage in series with its resistor, and the N-tuple of all signals is an N-dimensional Brownian motion process w(t). The following construction generalizes the dual-cell system, where the N × N matrix C defined in equation (47) is the general Maxwell capacitance matrix of the coupled circuit, and the resistance of each cell is an N-vector R≡diag(R1(t),R2(t)...,R N (t)) and R' = diag(R'1(t),R'2(t)...,R' N Each individual cell j has a time-dependent voltage source h at the nodes shown in Figure 4. j (t) and k j (t). As in the case of two connected cells, we collect these signals along the diagonal of an N × N matrix, and define k(t) ≡ diag(k1(t),k2(t)...,k N (t)) and h(t)≡diag(h1(t),h2(t)...,h N By using this circuit architecture, we can obtain a vector v(t)≡(v1(t),v2(t)…,v N (t)) is an N-dimensional random variable. This random variable evolves according to an SDE.

number

[0105] Let v(0) be the vector of voltage measurements across the capacitors of each cell at the initial time t0, and v(t) be the solution of the following integral equation for any t>0:

number

[0106] The second integral is treated as the Ito integral. The signal matrices h(t) and k(t) appear only in the drift and diffusion functions, respectively. Therefore, by controlling such source signals, we can simulate any linear derivative of the form (time dependence of v and w is omitted for readability):

number

[0107] where L(t) and M(t) are N×N matrices whose entries do not depend on the value of v and are given by:

number

number

[0108] We remove the minus sign from the diffusion term in equations (49) and (50) because dw is a Gaussian distribution with zero mean, and therefore the minus sign has no effect on that term.

[0109] Thus, we obtain a system that can simulate any forward SDE of the form (1), provided that the diffusion matrix G in (1) is voltage independent and the drift term f in (1) is linearly dependent on voltage. This includes state-of-the-art diffusion models such as (7), where G(t) → g(t) is a scalar and the drift term is linear. It also includes diffusion models with an inductive bias, due to the correlation introduced by the C matrix, which depends on the overall circuit connectivity.

[0110] The primary role of the forward process in a diffusion model is to generate sample trajectories, which serve as training data used to evaluate the loss function associated with training the score network. Therefore, the intention of introducing an analog device for the forward process described in Equation (50) is to efficiently generate training data for efficiently evaluating loss functions such as Equation (9) or Equation (10).

[0111] 8.1 Working Example: Denoising Diffusion Probabilistic Model To see how to use this circuit architecture to implement a generative model, consider the forward SDE (Equation (3)) used in DDPM. This model has matrix parameters,

number

number

number

number

[0112] 9 Noise distribution (stationary distribution of the forward process) Consider a forward process with the SDE given in equation (1). Assume that the coefficients f(x,t) and G(t) are set so that the process is ergodic (i.e., for any points x1 and x2, and any time interval [0,T], there is a nonzero probability that the sample trajectory of the process based on x1 at t=0 will reach x2 at t=T). Under this ergodic assumption, the forward process has a unique stationary distribution p ノイズ (x) and the forward process has an arbitrary initial distribution p データ (x) becomes p as t approaches ∞. ノイズ In practice, the initial distribution pデータ Although we may only have access to training samples from (x), the stationary distribution p ノイズ The functional form of (x) should be accessible. This system perturbs the training samples under a specified drift or diffusion model, and the distribution of the perturbed training samples increases as the forward process continues. ノイズ The backward process begins to look like this: ノイズ From the distribution p データ This is useful because it allows you to sample samples that are distributed approximately according to a known distribution p ノイズ By sampling from the unknown p データ (x) can be sampled from the unknown p データ The goal is to learn this distribution by knowing which choices end up where they are after a forward process and tracing how they got there.

[0113] 10 Maxell's Devil Score Network The second law of thermodynamics states that the entropy of an isolated system cannot decrease with time. At first glance, this may seem like an obstacle to performing the reverse process of a diffusion model in a physical system. After all, by transforming highly noisy data into highly structured data, the reverse process tends to reduce the entropy of the data. However, this simply means that the reverse process cannot be performed in an isolated physical system.

[0114] It is possible to decrease entropy locally by increasing it somewhere else. For example, this is the basis of a refrigerator. It is also the basis of a thought experiment called Maxwell's Demon (MD). In the MD experiment, an intelligent observer can spatially separate two components of a gas mixture, transitioning them from a state of high entropy to a state of low entropy, as shown in the left panel of Figure 11. To accomplish this, the observer observes the gas particles, sees which components are approaching a barrier, and then removes the barrier if it aids the sorting process.

[0115] This circuit for solving the reverse process of the diffusion model is a physical embodiment of Maxwell's demon scenario. Instead of the positions of gas particles, the dynamic variables are the voltages across the capacitors in each unit cell. Even if these voltages start out in a high-entropy state (i.e., uncorrelated), they can evolve over time to a low-entropy state (i.e., highly correlated).

[0116] This entropy reduction is facilitated by a neural network called a score network. The score network can take a variety of forms. For example, it can be stored and executed on a digital field programmable gate array (FPGA), or it can be stored in memory and executed by a digital CPU connected to the memory. The score network can also be an analog device, as described in Section 15 below.

[0117] This neural network / processing device combination acts as the demon in Maxwell's demon scenario, continuously observing the state of the analog system and adapting the voltages applied to each circuit appropriately. This is shown in the right panel of Figure 11. It therefore makes sense to call this system a thermodynamic AI system. Ultimately, this system utilizes an artificial intelligence unit (a neural network on a CPU or FPGA) to locally reduce the entropy of the analog system thermodynamically.

[0118] Figure 12 provides a detailed view of how the scoring network interacts with the analog system. As shown in Figure 12, a voltage meter reads the voltage across the capacitors in each unit cell, and the set of voltages forms a state vector, denoted x. Each of these voltages is converted to a digital signal via an analog-to-digital converter (ADC), which then sends these digital signals to a CPU or FPGA. Inside the CPU or FPGA, the scoring network is evaluated at points related to the input signals. The output of the scoring network is a predicted score, a vector (the score vector). This score vector is converted from a digital signal to an analog signal via a digital-to-analog converter (DAC), which becomes a vector of voltages. Each element of this voltage vector corresponds to the voltage applied within the corresponding unit cell. This entire process occurs continuously in real time during the time evolution associated with the reverse process of the diffusion model.

[0119] Figures 13A and 13B detail how the output of the score network is incorporated as an applied voltage to each unit cell. The main issue here is how to convert the output of the score network into a function g(t) that appears in the backward SDE or backward ODE. 2 (More generally, when G is a matrix, the function g(t) 2 is G(t)G(t) T is replaced by .)

[0120] Figures 13A and 13B show two alternative ways of connecting score networks to analog unit cells to solve the backward process. The method shown in Figure 13A is analog in spirit. Here, g(t) 2 The multiplication by g(t) is physically done in the circuit using analog components. The other method, shown in Figure 13B, is digital in mind. Here, g(t) 2 Multiplication by is done digitally in a CPU or FPGA, and the resulting digital signal is then sent to an ADC and applied directly as a voltage in the unit cell. These two methods each have their advantages and challenges. The analog method is more elegant as it directly utilizes the physical system for the multiplication, while the digital method is more convenient as it requires fewer analog components.

[0121] 11 Equations for the backward process 11.1 SDE Formulas for Backward Processes As explained in Section 8, assume that the SDE associated with the forward process is given by equation (50). In this case, the SDE corresponding to the backward process is given by

number

number

[0122] When t=T, v(T)~P ノイズ We initialize the path of the inverse model by sampling τ and start a physical clock that records the time τ that has elapsed since the start of the backward process, where τ is a physical time variable that increases positively from 0 to T as the backward process takes place, and t is abstract time that runs against the flow of physical time.

[0123] Substitution t → T-τ is made. Thus, v(T-τ) is the voltage value after τ seconds in the reverse process starting from v(T). The coefficients of the drift and diffusion terms are also evaluated at T-τ, which corresponds to τ seconds in the reverse process. Furthermore, dt is replaced by d(T-τ) = -dτ,

number

number

[0124] Ideally, we would like to solve this equation, but in practice we use a score network s θ We can approximate the score function by (v,T-τ). This approximation leads to the following SDE:

number

number

[0125] The backward process can be simulated by integrating the SDE described in equation (58) on this electrical hardware. We have already shown how to integrate the linear SDE (50) on analog electronic hardware, and after evaluating all drift / diffusion terms at decreasing time values ​​T−τ rather than the physical time τ, the SDE for the backward process differs from its forward counterpart only by the nonlinear drift term given by the e(v,T−τ) function.

[0126] As mentioned in Section 10 above, Figures 13A and 13B provide two alternative methods for programming the nonlinear drift component e(v,T-τ) that are approximated in the existing circuit model simulating the forward SDE. Both of these methods involve programming the s passed to the circuit unit cell as a voltage signal via a DAC. θ However, the two methods involve the digital evaluation of (v,T-τ). T is incorporated into the e(v,T-τ) function. Figure 13B shows the case where this multiplication is done digitally, where M(T-τ)M(T-τ) T is displayed on a digital computer. θ (v,T-τ), and the resulting signal is passed through a DAC and then applied as a voltage to the circuit. Meanwhile, Figure 13A shows an example of how to perform this multiplication using analog circuitry in the special case where M(T-τ) is diagonal and can thereby be thought of as a set of scalar quantities (each of the form g(T-τ)). For each i, e(v,T-τ) i into the i-th unit cell.

[0127] Therefore, we encode the integral equation in the probability transition of the circuit. We will now consider the integral equation. Score network s θ Suppose (v,T-τ) is trained on data from a forward process stopped at time T. The reverse process is then run at t=T, where the voltage v(T)~P ノイズ If the voltage is initialized at τ, the voltage v(T-τ) after τ seconds of the reverse process is given by the integral equation:

number

[0128] If we run the backward process until time τ=T, the solution to the integral equation at this point is P データ The final value of the reverse process, v(0), is P データAlthough it is not exactly distributed in , the approximation becomes more accurate as T → ∞ and as the score network is trained more accurately. Therefore, the backward process is ノイズ From unknown P データ The sample is varied towards samples that are approximately distributed according to

[0129] 11.2 Stochastic Flow ODE for N Cells As discussed in Sections 2.5 and 5, a stochastic flow ODE can be used to generate the samples. Consider the N-cell system shown above and the applied voltage signal e(v,t) = -(1 / 2)G(t)G(t). T s θ For (v,t), the probability flow ODE reads as follows:

number

[0130] Therefore, a new data sample x can be generated in exactly the same way as with the inverse SDE. The performance of this approach depends on the noise level of the system, as it assumes the absence of noise from the RC circuit.

[0131] 12 Switch between forward and reverse processes Ideally, rather than having two separate physical devices, the same physical device (eg, the same electronic chip) can be used for both the forward and reverse processes.

[0132] However, a comparison of the unit cell circuits in Figures 4 and 13 associated with the forward and reverse processes, respectively, shows that the unit cells are not exactly the same. In particular, the unit cell for the reverse process has additional circuit elements that are not used in the forward process.

[0133] The circuit of Figure 28 addresses this problem. Figure 28 shows the unit cell of Figure 13A with switches added at various locations. These switches allow the unit cell to be switched between forward and reverse process. That is, when the switch is open, the unit cell corresponds to a forward process unit cell, and when the switch is closed, it corresponds to a reverse process unit cell. In practice, the switches can be made from voltage-controlled transistors.

[0134] Thus, by adding these switches, the same physical device can be used for both forward and reverse processes.

[0135] Figure 28 illustrates this concept for a unit cell structure based on a voltage mixer. However, the concept applies broadly to other unit cell structures (e.g., structures based on variable resistors). Indeed, switches can be added to other unit cell structures to switch between forward and reverse processes.

[0136] 13 Flowchart of the entire system (with digital score network) We then present an overall thermodynamic AI system for generative modeling and discuss its application in two scenarios: (1) a scenario where the score network is already trained, and (2) a scenario where the score network is untrained.

[0137] FIG. 14 shows a flowchart for a scenario in which the score network is already trained. In this case, there is no need to run the forward process; simply run the backward process to generate new data points. The digital device stores previous information in the form of a connectivity matrix (1402). This information is uploaded to the analog device by, for example, opening or closing the associated switches (1406), as shown in FIG. 10. Similarly, the digital device stores samples from the noise distribution (1404), which are uploaded to the analog device by appropriately charging the capacitors (1408). With the selected connectivity and charged capacitors, the analog device is ready to evolve over time in the backward process (either the backward SDE or backward ODE) (1410). This evolution involves continuous communication with the pre-trained score network (1412), which is stored on the digital device as shown in FIG. 12. At the end of evolution, the data is downloaded to the digital device by reading the voltage across the analog device's capacitors and passing the resulting voltage through an ADC (1414). Finally, the data is decoded if necessary (e.g., mapping from latent space back to the original feature space) (1416).

[0138] Figure 15 shows a flowchart of a scenario in which a score network is trained by a user. In this case, the user generates (noisy) training data by evolving under a forward process and stores the training data in a digital device (1502). First, the data can be encoded into the digital device (e.g., into latent space or the spectral domain) (1504). The data is then uploaded to the analog device by appropriately charging the unit cell capacitors (1510). The problem geometry is stored in the digital device (1402) and, as in the pre-trained case, is uploaded to the analog device by selecting the unit cell connectivity (1406). Once the connectivity is selected and the capacitors are charged, the analog device evolves over time (1518), following a so-called forward process (forward SDE or forward ODE), which adds noise to the data. The noisy data is downloaded to the digital device by reading the capacitor voltages and passing them through an ADC (1520). This data serves as training data for the score matching optimization, with the loss function given by Equation (9). This results in a trained score network that is stored on the digital device (1512). At this point, the trained score network is accessible, which means we are in the same position as the starting point of the algorithm in Figure 14. This means that the remainder of the protocol follows the same steps as the protocol in Figure 14.

[0139] 14 Computational advantages (assuming a digital score network) Performing the forward and reverse processes in analog hardware has several advantages over doing it digitally. These include: 1. SDEs are generally numerically unstable for large dimensions, requiring sophisticated numerical integrators and fine-tuned time-step schedules. In analog systems, these difficulties disappear, and there is no time step to choose. 2. The total physical time is adjustable and depends on the dynamic damping rate of the system, which is proportional to 1 / (RC) for a single unit cell. This means that the physical time of integration can be shortened, which is an advantage of analog computation. 3. For the forward process, voltages must be initialized to the input data x, and then the voltage of each unit cell must be measured at each instant of the discretization chosen to train the score network. For the backward process, no intermediate measurements are needed, and the initial voltages are set to the noisy data x(T). In both cases, this does not involve the matrix multiplications and inversions present in the SDE, since the hardware already follows the SDE. In the general case (especially when G is a matrix), these operations must be performed digitally, and take O(N 2 ) operation requires more steps.

[0140] 15 Analog Score Network 15.1 Weaknesses of the Digital Score Network One of the problems associated with digital score networks is that of latency, which introduces time delays as will be explained shortly.

[0141] As the backward process evolves, a feedback loop involving the score network occurs: at each step in the backward process, the score network generates a predicted score s θ Querying the digital score network is time-consuming because it involves a forward pass through the deep neural network. This forward pass involves multiple matrix-vector multiplications and nonlinear activation functions. Therefore, we perform this forward pass and obtain a score s for the predicted value. θ Receiving (x,t) takes a significant amount of time. There is also additional delay from the DAC and ADC as the state variable x is converted to a digital signal and the predicted score is converted to an analog signal. The time delay associated with this process is called the latency of the score network.

[0142] When the backward process is performed on a digital device, a numerical SDE solver discretizes time and digitally integrates the backward process. In this context, the latency of the score network means that the numerical SDE solver must be slow. In other words, each time step of the numerical integration can take longer due to this latency, because each single time step involves a forward pass through the score network.

[0143] When the backward process is performed on an analog device, such as the device described above, the latency of the score network is also an issue. In this case, the backward process evolves continuously on the analog device. Because this system evolves continuously in a physical system, once it begins, its evolution cannot be interrupted or slowed down. This means that the score values ​​received by the backward process may not be perfectly accurate; they may have a time delay. Therefore, the backward process returns the score s at time t. θ When we expect the value of (x,t), we actually get the score s at the delayed or shifted time t+τ. θ (x, t + τ). (Note that a time delay corresponds to a positive shift in t, since t decreases over time during the backward process.) In other words, the latency of the score network translates into inaccurate values ​​of the score. As a result, the time evolution of the state variable x may be inaccurate, which may affect the quality of the samples generated by the generative model. Of course, if these inaccuracies are unacceptable, the physical parameters of the backward process can be selected to slow down the backward process, trading off speed for accuracy. This approach can still be useful if the time delay caused by the digital evaluation of the score network is sufficiently small.

[0144] In summary, in a digital diffusion process, the latency of the score network translates into a delay in the sample generation process, whereas in an analog diffusion process, it translates into an inaccuracy in the time evolution or a delay in the sample generation process, or both.

[0145] This motivates us to overcome the delay problem with the analog version of the score network disclosed below.

[0146] 15.2 General Strategies for Eliminating Latency A common strategy for eliminating this latency is to have an analog device that acts as a score network, called a score device.

[0147] The score device evolves in real time simultaneously with the time evolution of the backward process. Specifically, the score device evolves from τ=0 to τ=T f where τ denotes real (physical) time. At each time point, the score device outputs a prediction of the score.

[0148] The subtle point is that there may be some transformation f(τ) that relates real-time values ​​τ to abstract time values ​​t = f(τ), where t is the abstract time associated with the backward diffusion process. The backward process evolves such that the abstract parameter t evolves from t = T to t = 0. Typically, we set f(τ) = Tt, where τ evolves from 0 → T and t evolves from T → 0. When the real-time τ reaches some time τ = τ0, the scoring device generates a predicted score s θ (x,f(τ0)) is output, and actually f is selected and s θ (x,f(τ0))=s θ (x,T-τ0).

[0149] The fact that the score device and the backward process evolve simultaneously with each other allows the two devices to communicate with each other in real time, addressing latency issues without relying on the intervention of a digital device. Overall, the two devices evolve together according to a system of difference equations, and the two difference equations are coupled.

[0150] The concept of simultaneous time evolution is useful even if one of the devices is digital: for example, if the backward diffusion process is solved digitally with a numerical SDE solver and the analog score network evolves in time, or if the backward diffusion process is analog and the score network is integrated digitally with an ODE solver.

[0151] However, the advantages are likely to be more pronounced if both devices are analog. In this case, the two devices can be physically connected with an analog link, and all signals can remain analog (i.e., no analog-to-digital conversion is required). Therefore, the following will primarily focus on the case where both the score device and the reverse process correspond to analog devices.

[0152] 15.3 General Differential Equations for Analog Systems Assuming a backward process corresponding to an analog device as described above, the backward process evolves as follows:

number

[0153] Hereafter, s(v,t)=s θ We use the notation *(v,t) to denote the output of the trained score network, i.e., the output associated with the parameter θ* obtained after training the score network.

[0154] Note that t is an abstract variable, which can be converted to physical time by the transformation of variables τ=Tt, as follows:

number

[0155] Here, we use the fact that dt = -dτ.

number

number

[0156] The score device evolves according to a unique differential equation. The score differential equation can be written as follows:

number

number

[0157] 15.4 General integral equation for score devices This section concerns the scores predicted during the back-diffusion process.

[0158] The actual score value predicted by the score network depends on both the differential equation and the initial conditions. The predicted score value has the form:

number

[0159] In this equation, s0 = s(v0,τ=0) is the initial condition associated with the transition, which corresponds to the score value at τ=0 of the initial vector v0 = v(τ=0). Assuming a relationship t=T-τ, the initial condition s0 can also be thought of as the score value s(v(t=T),t=T), i.e., the score at abstract time t=T. As mentioned in Section 9, the distribution at abstract time t=T is the so-called noise distribution P ノイズ Therefore, the initial condition s0 corresponds to P ノイズ Related to the score.

[0160] In many cases, P ノイズ is simple, so that it can have an analytical description of the score function (e.g., P ノイズ is often a multivariate Gaussian distribution with known mean and covariance.) Thus, in some cases, it can be assumed that the functional form of the term S0 is known in advance before running the diffusion model.

[0161] In other cases, we may not know the exact form of the initial condition, but we can learn this function through the score matching process described in Section 2.2. In this case, we can think of the initial condition for the score as a free parameter to be learned, so we define the initial condition as s θ (v, t=T), which indicates that it is parameterized. Therefore, the predicted score value for some set of parameters θ can be written as

number

[0162] In general, we go one step further and the initial condition is an affine function, of the form s θ (v,t=T)=A θ v+b θwhere A θ is a parameterized matrix, and b θ is a parameterized vector. In other words, for an initial vector v0, we can write the initial conditions as

number

[0163] The reason why the condition τ = 0 (or t = T) is nearly affine is as follows: In the forward diffusion process, the final distribution as t → ∞ is P ノイズ It is called a unique initial data distribution P データ This means that at time t=0, any initial distribution (i.e., any data distribution P データ ) for the forward process, eventually reaching the same final distribution as t → ∞. For example, we can choose a data distribution that has a peak at a particular data point x(0). In this case, the stationary distribution is a multivariate Gaussian distribution (with an affine drift term for the forward process we consider), and in this case the stationary distribution is p 0t This corresponds to a conditional distribution of the form (x(t)|x(0)). データ , the corresponding stationary distribution P ノイズ is a multivariate Gaussian distribution. Therefore, assuming T is large, the distribution at t=T corresponds approximately to a multivariate Gaussian distribution. The scores of a Gaussian distribution are affine functions, so the condition at τ=0 (which is approximately P ノイズ This is why we can assume that the score of

[0164] Combining equations (66) and (67) gives the parameterized output formula for the score device.

number

[0165] 15.5 Example differential equations for score devices based on total derivatives Consider two different embodiments of how the score device can be constructed.

[0166] Example 1: In the first example, we consider building a score device based on the theoretical concept of total derivatives. The score s(v(τ),τ) is a function of both τ and v(τ). Therefore, we can write the total derivative with respect to τ as follows:

number

[0167] Here, ∇v denotes the gradient with respect to the v vector. Comparing the form of equation (69) with the form of equation (64),

number

[0168] If we view the score as a static function s(v,τ), the form of this static function is (1) the data distribution P データ and (2) a forward diffusion process (i.e., a forward SDE). However, if we imagine that τ and v(τ) progress over time, we can also view the score as following a trajectory. In the latter case, the trajectory has an initial condition and evolves dynamically over time. From the trajectory perspective, the evolution of the score is governed by the derivative term

number

number

number

[0169] This suggests that a model of the score function can be developed by rewriting equation (69) as follows:

number

number

[0170] In other words, a trainable function q θ (v,r) is the partial derivative

number

[0171] Function q θ and r θ The form of is flexible, but some intuition is helpful.

number

[0172] These two functions are quite complex and could potentially be constructed using a neural network-inspired approach. Section 15.6 below discloses circuit architectures suitable for such a neural network-inspired approach.

[0173] Once the training process is complete, thereby fixing the parameter θ, the score device can be used in combination with the backward diffusion process. After training, the function q θ and r θ and q=q respectively θ* and r = r θ* where θ* corresponds to the post-training parameters. Since the score device is used in combination with the backward process,

number

number

[0174] We use w here because we previously defined wdt = dw, and we also emphasize that this is the same w (the same stochastic noise source) that appears in the backward diffusion process. In summary, equation (72) is the differential equation that describes how the score evolves over time during the backward process, assuming the full differential model presented in this section.

[0175] Equation (72) is a formal mathematical proposition. In fact, the reverse process leads to the score device.

number

number

number

[0176] 15.6 Circuit Architecture for the Full Differential Approach FIG. 16 is a general schematic diagram of an exemplary score device 340, including the six components (sets) of the all-derivative approach. 1. The output is r θ a voltage source 341, which is (v, τ); 2.r θ The components of (v,τ) are physically encoded into a voltage vector

number

number

number

number

[0177] Figure 16 depicts several circuits as black boxes, including summer circuits 344 and 347, integrator circuit 345, and voltage source q θ (v,τ) and r θ (v,τ) and the associated circuits 341 and 343. The form of these voltage sources is detailed in the next subsection.

[0178] FIG. 17 shows an example of an adder circuit (left) and an integrator circuit (right) suitable for use in the score device 340.

[0179] 15.7 Voltage Source Form q θ (v,τ) and r θ (v,τ) score function s θ Recall that (v,t) is an N-dimensional vector. Therefore, the model q θ (v,t) is also an N-dimensional vector trained to approximate the partial time derivative of the score function. Furthermore, each component of the score function

number

number

number

[0180] Model Q θ (v,τ) and r θ(v,τ) contains the parameters of the neural network that needs to be trained. As mentioned before, problems arise when these outputs are evaluated digitally (e.g., from a neural network programmed on a CPU). Latency from the evaluation itself, as well as digital-to-analog (and analog-to-digital) conversion, can hinder performance.

[0181] Therefore, q and each τ (i) Figure 23 shows the relationship between q and r at position v and time t>0. (i) A score device is coupled to the trajectory v(t) of the forward process, so that at every space-time point (v(t),t) along the trajectory, q and r (i) These values ​​are sent to each network as a continuous voltage signal with the component v(t) coming directly from the device running the forward process. The time t is sent continuously to the analog network as a ramp voltage with unit slope that starts rising from zero when the combined forward trajectory begins, ensuring that the network receives v(t) and also the time at which these readings occurred. Next, an N-valued function

number

[0182] Each network receives N+1 voltage signals (v(t),t) as input, and the analog neural architecture performs a series of layered transformations. Each layer transforms a pair {A,B} k is shown by A k is a block that implements a parameterized affine transformation, which includes an element-wise nonlinear transformation B k The number of layers K and the width of each layer M kare left as hyperparameters. Figure 23 shows the electrical system that implements these layered transformations.

[0183] The first affine sublayer A1 is implemented by passing a voltage signal (v(t),t) through a network of resistors, and for now we restrict the affine transformation to the case where it is simply a matrix multiplication (no additional bias term). This is shown in Figure 24.

[0184] To obtain the dot product of matrix A1 with the input, we make M1 copies of (v(t),t) and use each to compute the dot product with one of the M1 rows of A1. This copying procedure means, thanks to Kirchhoff's second law, splitting each wire carrying a component of (v(t),t) into M branches according to Figure 24. To compute the dot product of a copy of (v,t) with the jth row of A1, we attach a resistor in series with each wire carrying a component of the copy, and then connect all wires after the resistor at a node. The voltage at the node is the dot product of (v(t),t) with the vector of inverse resistances (conductances) connected to each wire. The voltage component signal (v,t) (i) If each wire carrying is placed in series with a resistor with an inverse resistance equal to the (i,j)th entry of A1, then the voltage reading at the node where all the wires meet will be the dot product of (v,t) and the jth row of A1. If we have M1 copies of (v(t),t), then A k This structure is repeated for each row of the vector A1(v(t),t), and each component of the vector A1(v(t),t) can be output as M voltage signals. k also applies to

[0185] The above structure assumes that the current exiting each layer from the node after the resistor is zero. This assumption can be realized in practice by inserting a voltage follower as a buffer between each summing node and the next layer. An example voltage follower is shown in Figure 25.

[0186] Another construction is possible using a summing amplifier. Consider again the voltage copy procedure above and the example of multiplying a copy of (v,t) by row j of A1. Figure 26 shows the circuit that performs this dot product.

[0187] We prove the dot product property as follows: First, we can assume that the current flowing into the input of the operational amplifier is zero due to the high impedance input of the operational amplifier. This allows us to use Kirchhoff's current law at node x to write:

number

[0188] Next, because the operational amplifier is used in closed-loop operation (there is a path from the output to the inverting input), we can further assume that the voltages at the inverting and non-inverting inputs are equal. Hence, V x =0, and we get the following:

number

[0189] To get a pure dot product, we can set R=1 and invert the output.

[0190] Nonlinear Layer B k As shown in Figure 23, each linear layer A k can be constructed by connecting a diode followed by a resistor. Measuring the voltage at each resistor gives a nonlinearly transformed voltage, thanks to the nonlinearity of the characteristic function of the diode (or any other electrically nonlinear element). Thus the transformed voltage is v'=Rf d (v) takes the form of f d is the characteristic function of a diode. The current-voltage characteristic of a diode under forward bias resembles the activation function associated with the rectified linear unit (ReLU). This provides the motivation for using a forward-biased diode, as ReLU is one of the most common activation functions used in neural networks.

[0191] The output of this analog neural network is

number

number

[0192] The parameter θ is the value of an adjustable resistor used throughout the affine layers of the network. In the spirit of digital neural networks, the nonlinear layers with diodes do not have adjustable components. The resistors can be adjusted to reduce or minimize the cost function L, the evaluation of which can be done in the analog domain as described in the next subsection.

[0193] Another possibility is to consider analytical forms for q and r, whose parameters become the coefficients of the selected series expansion. These have the general form

number

number

number

number

[0194] Another solution between fully analog and digital evaluation score functions is to use an FPGA for inference, in which case q θ (v,τ) and r θ (v,τ) is a neural network programmed on an FPGA. One advantage of this approach is that it can be fed into a voltage mixer to obtain h with low latency, while retaining all the benefits of digital inference using CPUs and GPUs, such as parallel processing.

[0195] 15.8 Alternative Score Devices Based on Simple Circuits Example 2: In the second example, we consider building a score device based on a simple circuit. In this case, the score device is composed of the following elements:

[0196] Figure 22A shows the circuit diagram of a single unit cell, where the resistor is split into two resistors R since the introduction of connections between unit cells makes the resistors unequal. i and p i and divided into

[0197] Figure 22B shows a circuit diagram of two unit cells connected through a capacitor, essentially the same type of connection we introduced in the forward diffusion process. 1. The score device consists of N unit cells. 2. Each unit cell consists of three components in series: (1) capacitance C θ,i (2) resistor R θ,i (3) a voltage source V whose output depends on the input vector x θ,i Contains (x). 3. All three of these components are parameterized by trainable parameters (hence the subscript θ) during the training process. 4. The N unit cells are capacitively coupled to each other according to a connectivity matrix. This coupling can follow the same geometry as used in the forward diffusion process described in Section 7 and Figure 10. 5. Voltage Source V θ,i (x) is essentially an analog neural network, containing multiple layers including affine layers and nonlinear activation functions.

[0198] For the connections between unit cells, the same strategy as employed in Figure 10 can be used, including switches that can be turned off or on based on the connection matrix. In fact, for the score devices, we can choose the exact same connection matrix as used in the forward diffusion process.

[0199] Voltage source V θ,i (x) can have a structure very similar to the sources of q and r discussed in Section 15.7. The main difference is that V θ,i The key difference is that (x) does not explicitly depend on the time parameter t, so the trainable weights are time-independent. Apart from that difference, the structure is similar to that of q and r, and V θ,i (x) consists of alternating layers of affine functions followed by a nonlinear activation function.

[0200] 15.9 Partial Differential Equations for Alternative Score Devices Next, we present the time evolution of the alternative score device discussed in the previous subsection within this general framework. Recall that equation (64) gives the general form of the differential equation. The alternative score device isθ If the function is τ or

number

number

[0201] capacity matrix C θ is defined in a similar way to equation (47), and the resistance matrix is ​​R θ ≡diag(R θ,1 ,R θ,2 ,...,R θ,N ) Then we obtain the following differential equation for the entire score device:

number

number

[0202] The intuition as to why this differential equation is potentially useful is that it captures some of the intuition of the total derivative approach. Equation (79) expresses the time derivative term

number

[0203] This approach can also be converted into a hybrid digital-analog score device, using a digital device such as an FPGA to drive the voltage source V θ (v) is generated. In other words, V θ (v) can be represented by a digital neural network. The DAC converts the output of this neural network into an analog voltage that is applied to an analog unit cell of the score device, which in turn uses the analog score used in the backward diffusion process.

[0204] We then focus primarily on the total derivative approach in Section 15.5, but the formalism we develop is also applicable to the alternative scoring devices presented here.

[0205] 15.10 Incorporating Problem Geometry and Inductive Bias into Analog Scoring Devices Two approaches for constructing analog score networks (or hybrid digital-analog score networks) are presented.

[0206] In practice, both of these approaches benefit from considering the geometry of the problem. Section 7 and Figure 9 show that different problems have different geometries. Considering this geometry improves performance in many machine learning tasks, including generative models.

[0207] In the context of analog core devices, the geometry of the problem can be considered in the circuit used to implement a parameterized voltage source. Specifically, this is the total derivative-based source q of the device. θ (v,τ) and r θ (v,τ) and the source V of an alternative score device based on a simple circuit. θ (v) refers to

[0208] These voltage sources can be modeled as an analog neural network in which each layer performs an affine transformation followed by a nonlinear activation function. The affine transformation is described in Section 15.7 and is represented by the matrix A as shown in Figure 24. The problem geometry can be incorporated into each A matrix associated with these affine transformations. For example, these A matrices can be sparse, with nonzero elements corresponding only to the connectivity of the problem. In other words, these A matrices can be chosen to have a structure similar to the adjacency matrix A in Section 7. By matching the zero elements of the A matrix with the zero elements of the adjacency matrix A, the problem geometry can be modeled as the analog voltage source q θ (v,τ), r θ (v,τ), and V θ (v) A method is provided for incorporating the same into an analog score device.

[0209] Physically speaking, this corresponds to limiting the number of wires used in the circuit structure of Figure 24. In other words, if the adjacency matrix A has a non-zero element associated with the wires in Figure 24, the wires can be bundled. Thus, the adjacency matrix can be used to guide the construction of the circuit of Figure 24 to incorporate the geometry of the problem.

[0210] This comment is q θ (v,τ), r θ (v,τ), and V θ While (v) applies when it is an analog device, the geometry in question can also be incorporated into a digital device, and the overall score device is thus a hybrid digital-analog. In this case, standard techniques can be used to incorporate inductive biases into the digital neural network.

[0211] 15.11 Evaluating loss functions based on scores from conditional distributions The loss function can be written in several different ways. We discuss the loss function in equation (9) here, and then discuss alternative loss functions in the next subsection.

[0212] The loss function in equation (9) is the conditional distribution p 0t Based on (x(t)|x(0)), it can be expressed as follows:

number

[0213] In some special cases, we may have an analytical expression for the score of the conditional distribution. That is, if the forward SDE is particularly simple, the conditional distribution will also be simple. The score of the conditional distribution is an affine function as long as the drift term in the SDE is affine. In general, however, the form of this affine function may not be known. A particular case in which the score of the conditional distribution can be solved analytically is when the matrix L(t) defined in Equation (51) is a diagonal matrix.

[0214] Consider the case where the analytical form of the conditional distribution of scores is known and is affine. In this case, we can write

number

[0215] We now explain the various steps required to evaluate the above loss function.

[0216] (1) Generate the score of the conditional distribution (SCD) as a voltage. We can now discuss how to generate the SCD as an analog voltage.

[0217] The vector v(t) is the voltage vector corresponding to the data vector x(t). This vector v(t) is generated by a forward process that evolves up to time t. Therefore, v(t) can be stored in an analog device. On the other hand, the matrix A(t) and vector b(t) can be stored digitally, such as in the memory of a CPU. This provides two options for how to calculate the SCD value.

number

[0218] The first option is to send the vector v(t) through an analog-to-digital converter and then calculate equation (82) on a digital device (the same device that stores A(t) and b(t)). The resulting vector can be fed back to an analog device using a digital-to-analog converter.

[0219] The second option is to calculate equation (82) on an analog device. This involves treating b(t) as a voltage vector and using a voltage adder to sum this vector to A(t)v(t). The latter calculation can be done, for example, using a circuit similar to the adder circuit that encodes the elements of A into resistor values ​​and then calculates a weighted average of the elements of v.

[0220] So, let us assume that one of these options is selected, and as a result we present the analog device with a voltage vector s x(t)|x(0) will be saved.

[0221] (2) Predictions for score values θ Obtain (x(t),t). Next, s θ Consider how to obtain the value of (x(t),t) as a voltage vector at a particular time point of interest, t=t. m where 0≦t m ≦T. It can be written as follows:

number

[0222] Here, since we are considering a forward process, we use the time variable τ instead of the time variable t, which is usually written as τ = Tt. Note that ∂t = -∂r.

number

[0223] Therefore, in the total differential model,

number

number

[0224] Thus, the total differential model with respect to t is:

number

[0225] In any case, we rewrite equation (83) as follows:

number

[0226] If we know the score at the final time T, we can calculate the desired score s θ (x(t m ),t m ) can be calculated. As mentioned above, the distribution at time T is essentially a noise distribution P ノイズ In particular, in Section 15.4, P ノイズWe can assume that is Gaussian distributed, in which case we argue that the score is an affine function. We write this affine function as

number

number

number

[0227] Equation (89) provides a blueprint for how to obtain score values ​​from the model to evaluate the loss function, namely by following the process: 1. Initialize the analog forward process at a fixed data point x(0). 2. Time t=0 to t=t m The analog forward process is evolved in time until 3. Time t=t m At time t m Initialize the analog score device based on the values ​​of the variables in 4. Analog score device and analog forward process simultaneously, t=t m While developing in the forward direction from t = T, h appearing in equation (89) θ Calculate the integral of and output it as a voltage vector. 5. At time T, obtain the final result of the forward process, vector x(T), and use it to θ x(T)+b θ is calculated as a voltage vector. 6. On the analog device, add the two voltage vectors from the previous two steps (step 5 and step 6) to get the resulting score value s θ (x(t m ),t m) is obtained as the voltage vector.

[0228] (3) Take the difference between the score values. So far, s θ (x(t),t) and s x(t)|x(0) We've explained how to get the voltage vectors. All of these voltage vectors are defined relative to a reference. So if you want to calculate the difference between them, you can simply take the relative voltage between them. In other words, if you want a vector,

number

[0229] (4) Calculation of the norm square of the voltage vector The next step in calculating the loss function is to calculate the norm squared of the vector d(t). This can be done using the circuit shown in Figure 18, with a proportionality constant. When d(t) is input to this circuit, the output voltage will be proportional to the norm squared.

number

[0230] (5) Calculating the expected value of the loss function The final step is to calculate the three expected values ​​that appear in equation (9). The first expected value, E x(t)|x(0) involves sampling different x(t) from a fixed starting point of x(0). Operationally, this involves running the forward process (with a fixed starting point) multiple times, say K times, and then taking the average of all the runs. N kl (t m )=||d kl (t m )|| 2 Define as the squared norm of the difference in the scores of (90) in the k-th run of the forward process, with a fixed starting point x(0) l and the forward process is mThen,

number

[0231] In fact, it may be easiest to calculate equation (92) digitally, which is the voltage N kl (t m ), passing it through an analog-to-digital converter, and then calculating the average of equation (92) on a digital device such as a CPU. lm is stored on a digital device.

[0232] The next step is the expected value E x(0) The digital device computes the data distribution P データ We generate a sample of x(0) from , and therefore use an estimator to estimate this expectation.

number

[0233] This involves generating L samples of x(0) on a digital device and computing the average in equation (92) on the digital device, because E lm is already stored digitally.

[0234] Finally, the expected value E t This may involve a weighting function λ(t) (which can be stored digitally). Again, a digital device can estimate this expectation. m We uniformly sample and introduce an estimator.

number

[0235] G is an unbiased estimator of the loss function L1 in (80). This means that the expected value of G is equal to L1. The implication is that the convergence guarantee of optimization methods that use G as an estimator is guaranteed, since convergence guarantees often depend on estimators being unbiased. This is relevant to our discussion of optimization in Section 15.14.

[0236] 15.12 Evaluating loss functions based on the Hessian trace An alternative loss function based on the trace of the Hessian of the log-probabilities can be written as follows:

number

[0237] In the context of the full derivative model for this scoring device, this loss function is particularly easy to evaluate because it constructs an explicit model of the Hessian matrix of the log-probabilities. θ is actually a Hessian model of the log-probabilities, i.e.

number

[0238] Therefore, equation (95) can be rewritten as follows:

number

[0239] A method for evaluating this loss function will now be described.

number

number

[0240] Calculating the terms of equation (98) then involves: 1. For voltage vector input v=x(t) and time input τ=Tt, each function

number

number

[0241] Next, we will explain how to calculate the terms.

number

[0242] In the previous subsection, we presented the protocol for obtaining the score value output from the model based on the formula in Equation (89). Therefore, we follow the following steps: 1. Following the protocol in the previous subsection, calculate the score value s using the formula in (89). θ Obtain (x(t),t). 2. Take the square of the norm of this voltage vector using an analog circuit like that shown in Figure 18. 3. An appropriate proportionality factor can be added to the output of the previous step via a voltage amplifier. The result is the output voltage α(t,x(t),x(0)).

[0243] The next step is to add these voltages together, for example on an analog device, to get a voltage such as:

number

[0244] The final step is to calculate the expectation in equation (95). This can be done digitally, involving first converting the analog voltage γ(t,x(t),x(0)) to a digital signal using an analog-to-digital converter. After that, we can follow a procedure very similar to the one described in the previous subsection: we introduce an estimator for the loss function in equation (95) as follows:

number

[0245] where x(0) l HA P データ denotes the l-th sampling of x(0) from x(t m ) kl is the initial point x(0) l From time t m x(t m ) is the l-th sampling of Equation (101). Equation (101) can be computed on a digital device. Furthermore, G provides an unbiased estimator of the loss function in Equation (95) and therefore has desirable convergence properties when used within an optimization routine.

[0246] 15.13 Evaluation of Alternative Loss Functions by Analog Time Integration In the loss functions discussed above, the evaluation of the expected value is done digitally, which does not take full advantage of analog devices. In particular, analog devices can efficiently compute the time integral, so they can calculate the expected value E over time t. t may be useful in assessing

[0247] For this purpose, the expected value E t Consider alternative loss functions, including computing E first rather than doing it last. t into the mathematical equation and find other expected values ​​E x(0) and E x(t)|x(0)appear before . In general, this may change the loss function, i.e., shuffling the order of these expectations may actually change the loss function. Nevertheless, the intuition of the score matching process seems to still be captured when the expectations are reversed, and so the new loss function may still be useful for score matching.

[0248] This strategy can be used for both of the two loss functions considered previously, i.e., equation (80) and equation (95). Adapting equation (80) in this way gives:

number

[0249] where T denotes the trajectory of the forward process, and T|x(0) denotes such a trajectory given an initial starting point x(0). Hence, E T|x(0) denotes the expected value of the entire trajectory of the forward process given an initial starting point x(0).

[0250] In Section 15.11, we outlined a protocol for obtaining the argument to the norm of this loss function, i.e., we provided a protocol for obtaining the value of d(t) as an analog voltage.

number

[0251] As an analog voltage

number

number

[0252] The next step is to

number

[0253] Consider the thermal method for RMS (root mean square) voltage measurement. In this method, an unknown time-dependent voltage heats resistor R1. A digital device (e.g., a CPU) applies a DC voltage to resistor R2 until both resistors reach the same temperature. At this point, the DC voltage is equal to the RMS voltage of the unknown power source. Temperature sensing is performed by two semiconductor diodes, which can be viewed as thermistors. The overall circuit diagram is shown in Figure 19, and also includes an operational amplifier. This subroutine is used to measure the voltage source.

number

number

number

[0254] Next, run the subroutine in Figure 18 on the vector a = {a i}, which returns the square of the norm of the vector, and the output voltage is proportional to:

number

[0255] An analog-to-digital converter converts this analog voltage into a digital value. This same procedure is then repeated for many different runs of the forward process, and the results are averaged to produce the expected value E t|x(0) The expected value E x(0)is obtained by averaging the values ​​of x(0).

[0256] Similarly, the loss function in equation (95) can be adapted to obtain a new loss function.

number

[0257] The time expectation of the norm squared term E t can be evaluated on an analog device using the thermal method described around equation (163), while the time expectation value of the trace term, E t can be evaluated on an analog device using a standard integrator circuit. The two contributions from the two terms can be summed together, and the remaining calculations associated with this loss function can be easily performed.

[0258] 15.14 Training the parameters of the score device FIG. 20 shows a hybrid analog-digital feedback loop 2000 for training the parameters of the score device 340. In this feedback loop 2000, the analog device evaluates the loss function (or its gradient) for a fixed parameter (2210) value θ. The results of this evaluation are sent to a digital device, which selects a new value θ' for the parameter based on some optimization routine (2220). (For example, examples of such optimization routines could be gradient descent, stochastic gradient descent, or a gradient-free method such as Nelder-Mead.) The new parameter value θ' is then programmed into the analog device to re-evaluate the loss function (or its gradient).

[0259] This feedback loop 2000 can be repeated any number of times until a convergence criterion is met. For example, the convergence criterion may indicate that the optimization terminates when the loss function does not substantially decrease in value for several consecutive iterations. The final parameter values ​​after the convergence criterion is met are θ * This indicates:

[0260] This final set of parameters θ * can be uploaded to the score device for use in the reverse process. In other words, after the training process, the parameters of the score device are * and use these parameters throughout the evolution of the backward process.

[0261] 15.15 Feeding an Analog Signal to the Backward Spreading Process Figure 27 shows the communication process between the backward diffusion process and the analog score network or hybrid digital-analog score network. For simplicity, this is shown for the case of a single unit cell (N=1). In Figure 27, two possible unit cell configurations are considered: a voltage mixer-based unit cell and a variable resistor-based unit cell. The reason for the complex circuitry is that the score value is calculated as g(t) 2 where g(t) is the prefactor of the diffusion term.

[0262] 16 Flowchart of the entire system including the analog score network Figure 21 shows a flowchart of the entire thermodynamic AI system when the diffusion process (forward process and backward process) and the score network all correspond to analog devices. This includes a hybrid analog-digital optimization loop for the training process, as shown in Figure 20, in which analog and digital devices evaluate the loss function (2010), and a digital device executes an optimization routine (2020) to provide a trained score network (2112). Furthermore, after training, the analog score device establishes an interface with the backward process and provides score values ​​used in the backward process. This flowchart can be compared to the flowchart in Figure 15, which shows the case of a digital score network. Overall, the two flowcharts are similar, but Figure 21 has more steps performed by analog components, which can provide additional computational advantages, as explained immediately below.

[0263] 17 Computational advantages by analog or hybrid analog-digital score networks If the entire device is analog, i.e., each RC cell is a diffusion device representing a data element and the score device is also completely analog, it retains the computational advantages shown in 14. Solving the reverse SDE on a digital device involves a very numerically costly algorithm such as the Euler-Maruyama method. Therefore, when performed on a digital computer, the calculation of the reverse SDE solution at time intervals 0 < τ < T may take more than T seconds. Furthermore, the calculation time deteriorates as the dimension of the reverse SDE increases. On the other hand, this analog SDE integrator can solve the reverse SDE in just T seconds at the same time interval. In addition, regardless of the dimension of the SDE, the calculation time should remain at T seconds.

[0264] With the established speed advantage of analog SDE integration, it shows how to ensure that the solution of the reverse process provided by the analog score network by the analog integrator is accurate. The analog score network addresses the latency problem in the continuous trajectory of the reverse process. Specifically, the latency problem encountered during the analog reverse process is not a problem for a digital device simulating the same process because the digital system solves the reverse SDE in discrete time steps and the time to query the score network does not affect the solution of the discrete solver.

[0265] By solving the latency problem, this analog SDE integrator can be used to accurately solve the backward process. In short, the backward process can be solved much faster on an analog device, and the analog scoring network can ensure that such a solution matches the accuracy of a digital solution that takes longer to obtain. Specifically, the advantages an analog system including an analog scoring network has when solving the backward process are: The drift term in the backward process does not need to be calculated in the digital score network. Therefore, there is no latency from the digital evaluation of the score network by a typical neural network (see Figure 3). Therefore, the drift term in the backward process can more accurately track the true shape along the trajectory. The DACs used at each time step in the forward and backward processes have no latency. The only DACs remaining are those used for initialization, where the data vector x is converted into a voltage vector v, and v is collected at the end of the time evolution. If the derivative of the h function is split into q and r functions, see Section 15.5, the size of the analog score device may be smaller (fewer variable parameters) because this approach is based on splitting the terms that contribute to the derivative of the score function. Splitting the terms allows simpler contributions to be tracked, using fewer resources. Using an analog score network, scores along a continuous locus of points can be resolved, and these continuous evaluations can be ported to another analog device against which the loss function is evaluated (depending on its exact form, see Section 15.13). This can be less computationally expensive than a digital evaluation of the loss function, which involves many evaluations of a digital neural network representing the scores, significantly increasing the number of score evaluations used in evaluating the loss function.

[0266] 18 Deep Learning and Neural Networks Deep learning (DL) and neural networks (NN) are used in language translation, product recommendations, social media, dynamic pricing, and other applications. Deep learning refers to the use of artificial neural networks as trainable mathematical models, and neural networks typically have many layers, making them deep. Deep learning systems extract high-level features about a dataset that help classify the data within the dataset. Typical applications of deep learning include classifying images of handwritten digits and classifying images of cats and dogs. In fact, deep learning has very broad applications in many fields.

[0267] The left panel of Figure 29 shows a simple representation of a neural network, where the input variable x is mapped to the output y through a hidden layer (typically multiple hidden layers are used). The trainable weights are the weights that map from x to the hidden layer variables {h j} and the mapping from hidden layer variables to the output y. The hidden layer variables represent high-level features of the data and improve classification performance.

[0268] 19 Overconfidence in Deep Learning Machine learning systems, such as neural networks, are often overconfident in their predictions. In low-risk applications, this may not be a major issue. However, some applications, such as self-driving cars and medical diagnostics, are high-risk, where incorrect decisions can have life-threatening consequences. In such high-risk applications, overconfidence can be devastating.

[0269] On a technical level, this overconfidence often arises from neural networks being trained with a limited amount of training data. These training data points exist in a vast feature space. Therefore, it is common for some regions of the feature space to be poorly represented by the training data, meaning that these regions may be far away from the training data. When a trained neural network is tested with test data, the test data may exist in a region far away from the training data, but the neural network still attempts to make predictions. Because such regions are unfamiliar to the neural network, it does not realize that it needs to be careful when making predictions.

[0270] 20 Quantifying Uncertainty in Deep Learning One strategy to address overconfidence is uncertainty quantification (UQ). UQ aims to quantify the uncertainty of predictions made by neural networks. UQ is useful for high-risk applications (e.g., cancer detection in medicine) because it provides guidance to users on when to defer to human judgment over machine predictions. UQ is widely recognized as a way to make machine learning more reliable and trustworthy.

[0271] There are several different approaches to UQ in machine learning. A simple example of UQ is adding confidence intervals to predictions made by a neural network.

[0272] A more advanced and rigorous approach to UQ is the Bayesian framework, which quantifies uncertainty by considering prior knowledge (often called a prior distribution) and updating that knowledge based on data or observations (often called a posterior distribution). Bayesian methods aim to quantitatively capture knowledge in the form of a probability distribution.

[0273] 21 Bayesian Neural Network (BNN) Neural networks are machine learning models that typically have multiple layers of linear and nonlinear transformations, allowing them to represent a variety of latent functions. Bayesian neural networks (BNNs) allow for quantification of uncertainty in the neural network's predicted output, which improves the reliability and trustworthiness of the model.

[0274] Figure 29 shows the difference between a Bayesian neural network (right) and a standard neural network (left). In the Bayesian case, the weights do not have deterministic values. Instead, the weights are drawn randomly from a particular probability distribution.

[0275] Instead of predicting a definite y for a specific input x, the BNN predicts y according to a certain probability distribution. Therefore, there is some randomness in the output prediction for a specific input. Specifically, the BNN makes a probabilistic prediction based on the following formula:

number

[0276] 22 Challenges of Large-Scale Bayesian Deep Learning The posterior distribution p(w|D) of the BNN weights is high-dimensional, non-convex, and often has multiple modes. Therefore, computing (108) involves integrating over a multidimensional, multimodal posterior distribution with unusual topological properties such as modal connectivity. Accurate methods for sampling from the posterior distribution can require thousands of training epochs to generate a single sample from the posterior. In practice, it is computationally impractical with modern computational methods to compute (108) in an exact manner.

[0277] Therefore, for computational reasons, researchers use computationally inexpensive methods to approximate the posterior distribution. For example, researchers often use mini-batch methods that use only a subset of the data. However, these methods can provide significantly biased estimates of the true posterior expectation in Eq. (108). The mean field approximation is another approach, in which the true posterior distribution is approximated by a unimodal Gaussian distribution, but this approach can also be a poor approximation of Eq. (108). To illustrate, Figure 30 shows a multimodal distribution. Fitting this multimodal distribution to a unimodal Gaussian distribution is a poor approximation.

[0278] In conclusion, obtaining highly accurate predictions from large-scale BNNs remains an open problem. While it is possible to obtain crude approximations of these predictions at reasonable computational speeds, it is also possible to obtain highly accurate predictions at very slow (prohibitively slow) speeds. However, there is currently no method to accurately calculate Eq. (108) at large scales and at high computational speeds.

[0279] 23 Bayesian Deep Learning Technical Details and Subroutines This section provides additional technical details about the various components, subsystems, and subroutines used in Bayesian deep learning. Specifically, we focus on a Bayesian deep learning approach called the Stochastic Differential Equation Bayesian Neural Network (SDE-BNN). The SDE-BNN approach is state-of-the-art in Bayesian deep learning, performing competitively or even better than other methods. The SDE-BNN approach is loosely inspired by physics and can be implemented on digital hardware.

[0280] 23.1 Variational Inference Bayesian inference involves updating a prior distribution to a posterior distribution based on data or observations. Variational inference (VI) is a form of Bayesian inference that assumes an Ansatz Q (i.e., a family of possible solutions) for the posterior distribution. An approximate posterior distribution is obtained by optimizing the distribution q(w) of the Ansatz Q. This optimization problem can be written as

number

[0281] 23.2 Evidence Lower Limit (ELBO) Minimizing or reducing the KL divergence is equivalent to maximizing or increasing the so-called evidence, which is

number

number

[0282] Therefore, the optimization problem becomes:

number

[0283] 23.3 Probabilistic Variational Inference Stochastic variational inference (SVI) combines natural gradients with stochastic optimization. The idea behind SVI is to use a cheaply computed, noisy, unbiased estimator of the natural gradient within a gradient ascent optimization. The natural gradient differs from the standard Euclidean gradient because it takes into account the geometry of the problem. SVI works using an unbiased estimator of the natural gradient rather than the natural gradient itself.

[0284] 23.4 Neural ODEs Neural ordinary differential equations (ODEs) represent a continuous depth version of an artificial neural network. The values ​​of hidden units in an artificial neural network represented by a neural ODE are h t and the weight value is w t In general, these quantities are time-dependent and evolve according to simultaneous differential equations.

number

[0285] A forward pass through the neural network therefore involves integrating this system of differential equations.

[0286] 23.5 Neural SDE Neural stochastic differential equations (SDEs) represent a continuous-depth version of a Bayesian neural network. Again, continuous-depth versions of BNNs evolve according to a system of simultaneous differential equations, except that the system is inherently stochastic,

number

[0287] 23.6 SDEs for Prior Distributions The prior distribution can be formulated as an SDE on the weights of the BNN. Specifically, the SDE is the function f w (t,w t ) and g w (t,w t ) can take the form

[0288] As an example, consider the Ornstein-Uhlenbeck (OU) process. For the OU process, f w (t,w t )=-w t and g w (t,w t )=σI dTherefore, the SDE system for the prior is:

number

[0289] 23.7 SDEs for the Posterior Distribution The posterior distribution can also be formulated as an SDE on the weights of the neural network (NN). The posterior distribution should be very expressive, so the SDE can be more complex than the SDE on the prior distribution. Therefore, we can model the weights as a neural network (NN) with trainable parameters Φ. Φ Specifically, we can choose the following SDE for the posterior distribution:

number

[0290] The trainable parameters of this model include both the parameter Φ that appears in the drift term and the initial condition w0 for the weights. Φ can be called a posterior drift network (PDN) because it determines the drift associated with the posterior distribution.

[0291] 23.8 Simplified Representation of the ELBO The expression for the ELBO can be simplified if the diffusion terms in the prior and posterior SDEs are the same. In this case,

number

number

[0292] where:

number

number

[0293] 23.9 Computational Difficulties in Digital Approaches to SDE-BNNs In practice, digital processors are used to train and store SDE-BNNs. However, digital approaches to SDE-BNNs face certain computational challenges, such as: 1. A large amount of training data to train the posterior drift network, 2. High complexity for training and evaluating posterior drift networks; 3. The long time it takes to numerically simulate the evolution of ODEs and SDEs, 4. Potential instabilities of SDEs / ODEs due to large dimensionality and rigidity leading to impractically small time steps. Includes.

[0294] The first two difficulties above are largely due to the lack of inductive bias in the problem formulation. Inductive bias refers to prior knowledge that is fed into the model's structure, such as knowledge about symmetries in the data. Having a strong inductive bias can reduce the training data requirements and improve the speed at which the model can be trained.

[0295] The third difficulty refers to the challenge of simulating time evolution using digital hardware. This includes both the challenge of digitally generating Gaussian random numbers (i.e., the dB terms in SDEs) and the challenge of numerical time integration by discretizing the time evolution. Finally, the fourth difficulty refers to the fact that there are many cases in which the numerical integration of SDEs and ODEs can cause instabilities due to the structure of the equations, especially for SDEs with large dimensionality. Another issue is known as stiffness, which can lead to small time steps and make numerical integration impractical.

[0296] 24 Thermodynamic AI System for Bayesian Deep Learning Figure 31 shows a thermodynamic AI system for Bayesian deep learning with a subsystem that can perform four subroutines. Each subsystem can be implemented as a physical analog device (subsystem / component) that performs the corresponding subroutine. Some subroutines may also be stored and processed on digital devices. These four subsystems are: 1.Weight Diffuser (WD) 3310, 2. Hidden Layer Network (HLN) 3320, 3. Posterior Drift Network (PDN) 3330, and 4. Loss Estimator (LE) 3340, including:

[0297] At a high level, Figure 31 shows how these four subsystems interact. The weight diffuser (WD), which can represent both prior and posterior distributions, feeds weight values ​​to the hidden layer network (HLN). The WD also communicates bidirectionally with the posterior drift network (PDN), feeding weight values ​​to the PDN and vice versa. The loss evaluator (LE) receives signals from the other three subsystems (HLN, WD, and PDN) to evaluate the loss function.

[0298] 25 Hidden Layer Networks 25.1 Overview of HLN The hidden layer network (HLN) is expressed by the following differential equation:

number

[0299] The output of the HLN is given by the integral equation:

number

[0300] Here, h0 is usually set to some input data value. For example, the training data D={x (m) , y (m) During the training process in , for some value of index m, h0=x (m) Once the HLN is trained, it can make predictions for some test input x where h0 = x.

[0301] An HLN can be viewed as a neural ordinary differential equation (neural ODE). This neural ODE can be stored and processed on a digital device or implemented on an analog device. Below, we discuss both the digital and analog cases. We also discuss how to add additional dimensions to the model to improve performance. Finally, we discuss how to obtain a prediction of the output value y from the HLN.

[0302] 25.2 Digital Hidden Networks A possible setup for the whole system is that the HLN is stored and processed in a digital device, and the other devices (WD, PDN, and LE) have analog components. The digital device that stores and processes the HLN could be a central processing unit (CPU) or a field programmable gate array (FPGA).

[0303] This setup includes conversion between digital and analog signals. For example, there may be an analog-to-digital converter that converts the weights (output by the WD) from analog to digital signals so that the weights can be processed by the HLN. There may also be a digital-to-analog converter, for example, to convert the output predictions made by the HLN from digital to analog signals.

[0304] One of the technical advantages of this approach is the concept of Drop-in Uncertainty Quantification (Drop-in UQ), which is inspired by the idea of ​​providing a non-invasive service that adds uncertainty quantification as a feature on top of existing (e.g., digital) architecture.

[0305] In industrial applications, users often have their own HLN and proprietary datasets stored on digital devices. In this case, a physical device can provide a drop-in UQ service that provides uncertainty quantification to the user's application. This involves interfacing the user's HLN and dataset with this physical device (including WD, PDN, and LE).

[0306] 25.3 Analog Hidden Layer Networks The HLN can be implemented in analog hardware as follows: Given a data set D={x (m) ,y (m)}M m=1 consists of M data points. Each input vector

number

[0307] (1) Unit cell An analog HLN can have N unit cells, one for each data feature. Figure 32 shows a possible structure of a unit cell in an HLN. The voltage across the capacitor is the state variable h t When performing circuit analysis, the resistors in the unit cell are expressed as time derivative terms

number

[0308] (2) Two coupled unit cells Figure 33 shows two HLN unit cells connected through a resistive bridge. The differential equation below describes the voltage across the capacitor, v j and v j’ Represents.

number

number

[0309] Here I NL,j and I NL,j are the currents in the nonlinear element branches of the jth and j′th unit cells, respectively. The matrix form of this equation is:

number

number

[0310] (3) N unit cells and problem geometry In general, an analog HLN implementation has N unit cells, each of which may or may not be connected to other unit cells via resistive bridges. We can extend the above analysis to the general case of N unit cells, which leads to differential equations of the same form:

number

number

[0311] Furthermore, the elements of the matrix J are given as follows:

number

[0312] where A is an adjacency matrix that represents the geometry of the problem. For example, some problems have one-dimensional geometry (e.g., time series data) and other problems have two-dimensional geometry (e.g., images). If A jk If =1, cells j and k are connected, and if the pair is not connected, A jk = 0. Matrices A and J allow for incorporating problem-specific geometry into the circuit. This can improve model performance, as accounting for the problem geometry introduces an inductive bias to the model, often improving the trainability and generalization of the model. In practice, switches (e.g., voltage-gated transistors) can upload the problem geometry into the circuit connectivity, as shown in Figure 10.

[0313] (4) Intuition The differential equation in (122) is partially similar to that of a neural network, because neural networks typically contain layers with alternating affine and nonlinear transformations. Considering the limit of infinitesimally small layers, the affine and nonlinear transformations appear to occur simultaneously. This limit of infinitesimally small layers is a restriction relevant to neural ODEs and therefore to this HLN. Therefore, the right-hand side of (122) involves the simultaneous action of affine and nonlinear transformations. That is, the term Jv-J s α provides the affine transformation for the vector v, and the term I NL is a nonlinear transformation of v.

[0314] Regarding the latter, the jth nonlinear element (NLE) is in parallel with the capacitor in the jth unit cell, and therefore the voltage across the jth NLE is v jThe current-voltage characteristic of the jth NLE is equal to the function f j This allows us to write

number

number

[0315] By definition, f j is a nonlinear function because it corresponds to the current-voltage characteristic of the NLE. Therefore, equation (125) corresponds to a nonlinear transformation of the elements of the vector v.

[0316] 25.4 Physical Implementation of Nonlinear Elements There are several different ways to implement nonlinear elements (NLEs) in a circuit, including diodes and transistors. The choice of NLE influences the type of activation function implemented within the neural ODE.

[0317] Consider a diode as an NLE. The current-voltage characteristic of a diode has positive feedback, whether forward or reverse biased. This means that the greater the voltage applied to the diode, the greater its conductivity. The NLE is in parallel with the capacitor in the unit cell, which creates a negative feedback on the voltage vector v, as seen by the negative sign appearing in the nonlinear term in equation (122). Because the current-voltage characteristic of a diode rises very rapidly with applied voltage, this causes a threshold effect, and the nonlinear drift term in equation (122) becomes very large as v increases. This creates a rapid negative feedback on v, causing the magnitude of v to saturate at a certain value. This is similar to certain activation functions used in neural networks. There are several activation functions that have a saturation effect, such as the sigmoid function and the hyperbolic tangent function. In this sense, choosing an NLE as a diode mimics these types of activation functions.

[0318] Alternatively, consider using a transistor as an NLE. In fact, the current-voltage characteristic of a transistor can be nonlinear, and the shape of the function depends on the input or output characteristics. In the case of a transistor with a common-emitter and common-base configuration, the input characteristic is convex, meaning that the current rises sharply with respect to the applied voltage. In this case (i.e., considering the input characteristic), the situation is similar to that of a diode, and the corresponding activation function exhibits saturation effects, such as sigmoid and hyperbolic tangent functions.

[0319] Conversely, when using a transistor to exploit its output characteristics, the current-voltage characteristic is concave and saturates. This leads to a convex activation function with positive feedback. Activation functions with positive feedback include the soft plus function and the rectified linear unit (ReLU) function. ReLU, in particular, is commonly used in neural networks. Therefore, using the output characteristics of a transistor is a strategy to obtain an activation function with positive feedback.

[0320] 25.5 Uploading Data to Analog HLN Each unit cell corresponds to a feature of the data. At time t=0, the feature vector x is encoded into the hidden layer value h0 (e.g., x is the feature vector x from the training dataset). (m) ) This encoding may involve a digital-to-analog converter that converts the digital signal associated with x into an analog signal. Once in analog form, x can be stored in an analog HLN by appropriately charging the capacitors in the HLN. Here, the jth feature of x is mapped to the capacitor of the jth unit cell with some permutation of the indices. Specifically, the initial voltage across the jth capacitor at time t=0 is v j =x j is set to

[0321] The symbol v represents the physical voltage vector between the capacitors of the unit cell, and mathematically corresponds to the vector h associated with the hidden layer values. Thus, v and h are essentially interchangeable from a symbolic point of view.

[0322] 25.6 Analog Enhanced HLN The feature vector x can be encoded into the initial hidden layer value h0. However, more generally, x can be encoded into a higher dimensional space by padding the vector with zeros. This concept leads to what is known as an augmented neural ODE. Augmented neural networks aim to expand the dimensionality of the feature space so that data can be more easily separated. A neural network (without augmentation) may not be able to implement functions whose trajectories intersect, and therefore a neural ODE may not be able to represent all functions. Augmented neural networks, represented by augmented neural ODEs, are useful for addressing this problem.

[0323] Figure 34 shows an analog version of an augmented neural network that implements an augmented neural ODE. The concepts shown in Figure 34 can be used as the basis for building an HLN, which can be thought of as an augmented HLN. Mathematically, the initial hidden layer values ​​can be expressed as the direct sum of the data point x and a vector of zeros, as follows:

number

[0324] Physically, this includes devices with N or more unit cells, i.e., devices with N+N a It has N unit cells a is the dimension of the zero vector in equation (126).

[0325] The extended HLN consists of N unit cells in data space and N aIt is possible to have non-trivial connectivity between the unit cells. This is shown in Figure 34. From an initial time to a later time t, additional features can evolve into non-trivial states a(t), and the overall state can be written as:

number

[0326] The rest of the protocol can proceed as if there were no extensions.

[0327] 25.7 Making predictions from the final hidden layer The value of the final hidden layer (i.e., the value at t=1) is called h1.

[0328] In the context of supervised machine learning, the goal is to transform h1 into a prediction of the output y. Assume there is a probabilistic map that maps h1 to y. This map can be described via a conditional distribution p(y|h1), where:

number

[0329] Assume that at the end of the HLN, there is a process that converts the value h1 of the last hidden layer into an output y. The subsystem or device associated with this process is called the Output Predictor (OP). The OP takes h1 as input and outputs a y value through a deterministic or stochastic function.

[0330] Physically, an OP may be implemented with an analog or digital device. If h1 has an analog form and y has a digital form, the OP can convert between the different types of signals using analog-to-digital and digital-to-analog converters. Alternatively, h1 and y can both be analog or both digital, in which case no conversion (between different signal types) is necessary.

[0331] 26 Weight Diffuser 26.1 Weight Diffuser Overview Diffusion is a physical thermodynamic process that is often mathematically mimicked in machine learning models, which raises the question of whether the diffusion in these mathematical models can be performed by physical thermodynamic devices.

[0332] In the context of Bayesian neural networks, weights can be modeled as undergoing a diffusion process, which adds uncertainty to the weights. For this application, a physical thermodynamic device called a weight diffuser (WD) diffuses the weights over time. The thermodynamic aspect of the WD device is fed by an analog stochastic noise source, which causes physical diffusion of the system's state variables.

[0333] Below, we first describe the WD device for the prior distribution, then discuss the WD device for the posterior distribution, and finally discuss how these two devices can be represented by a single device with a switch to switch between the prior and posterior cases.

[0334] 26.2 Preweight Diffuser The prior distribution is generally assumed to be relatively simple, and therefore the WD device for the prior distribution is also relatively simple. In some cases, the simplified loss function in Equation (115) does not include a prior distribution, so a weight prior spreader may not be necessary. On the other hand, the loss function in Equation (110) does include a prior distribution, so a weight prior spreader is useful in this case. In this sense, the weight prior spreader is an optional component in the overall system of Bayesian deep learning.

[0335] One way to construct a weight spreader is to create one unit cell for each weight. Let's say there are W weights in total. Typically there are more weights than neurons, so W>N. In any case, there is no reason to assume that W=N, so in general the number of unit cells in a WD can be different from the number of unit cells in a HLN.

[0336] (1) Unit cell of the pre-weighted diffuser Figure 35 provides a circuit diagram of a unit cell of the pre-weight spreader. The unit cell contains a capacitor C i , resistor R i , stochastic noise source B i are arranged in series.

[0337] The voltage across the capacitor of the unit cell is

number

number

number

[0338] Due to the presence of stochastic noise sources in the circuit, the state variables evolve according to a stochastic differential equation (SDE). For the unit cell of Figure 35, the SDE for the state variables is:

number

[0339] (2) Two coupled unit cells Now consider two coupled unit cells with either resistive or capacitive coupling. Figure 36 shows the case of capacitive coupling between two unit cells. In the case of capacitive coupling, the SDE associated with the two unit cells is:

number

number

[0340] (3) W unit cell and problem geometry The extension to W unit cells is straightforward: each cell may or may not be connected to other cells, but the choice of whether two cells connect can be based on the geometry of the problem. (This is similar to the discussion for HLN above.) Figure 10 (mentioned above) shows how the adjacency matrix A can be used to determine the connectivity of unit cells.

[0341] Assuming that the connections between cells include capacitive bridges, the SDE for the W unit cell is,

number

number

[0342] Furthermore, the elements of matrix C are given as follows:

number

[0343] 26.3 Posterior Weight Diffuser The architecture of the post-weight spreader is similar to that of the pre-weight spreader.

[0344] Consider again the W unit cell. The unit cell of the a posteriori weight spreader is shown in Figures 37A and 37B. This includes the case where the a posteriori drift network (PDN) is analog (Figure 37A) and the case where the PDN is digital (Figure 37B). In a digital PDN, an analog-to-digital converter converts the measured voltage on the capacitor to a digital signal, and a digital-to-analog converter converts the output of the PDN to an analog voltage applied within the unit cell. An analog PDN operates without a converter. Except for the feedback loop for this PDN, the unit cell is similar to that of the a posteriori weight spreader.

[0345] Again, the capacitance bridge can connect the unit cells, and the connectivity can be chosen according to the adjacency matrix A, which in turn follows the geometry of the problem. Again, this is illustrated by Figure 10. It is reasonable to choose the same connectivity for the pre-weight spreader and the post-weight spreader.

[0346] Assuming capacitive coupling between unit cells, the overall SDE of the state vector associated with the W unit cell can be expressed as

number

[0347] Compared to the pre-weighted diffuser, the difference here is the drift term

number

number

[0348] The a posteriori weight diffuser is parameterized by a parameter Φ that appears in the drift term. Additionally, there is a parameter in the initial conditions. The initial conditions are:

number

[0349] where θ represents a parameter,

number

number

[0350] Here, the set of parameters is as follows:

number

[0351] During the training process, the digital processor stores the values ​​of the ψ parameters and proposes updates to these parameters during an optimization routine, which is described in more detail below.

[0352] initial conditions

number

[0353] 26.4 Pre- and Post-Weight Spreaders as the Same Device Figure 40 shows a weight spreading device with a switch, such as a voltage-gated transistor, that can be toggled to switch the weight spreader between pre- and post-weight spreader settings, allowing a single device to perform the functions of both pre- and post-weight spreaders.

[0354] 26.5 The origin of Brownian motion So far, we have described the stochastic noise source in the weight spreader as an abstract device. This abstract circuit element is shown in Figures 35 and 37. i From a mathematical point of view, it is desirable for the stochastic noise source to be uncorrelated in time and have a Gaussian distribution with zero mean.

[0355] The stochastic noise source can be a thermal noise source, a shot noise source, or both a thermal and shot noise source. A variable amplifier can amplify the output of the stochastic noise source, allowing the standard deviation σ appearing in the differential equation (e.g., as in equations (132) and (134)) to be adjustable.

[0356] One type of possible stochastic noise is thermal noise. Thermal noise, also called Johnson-Nyquist noise, arises from the random thermal oscillations of charge carriers in a conductor, resulting in fluctuations in the voltage or current in the conductor. Thermal noise is Gaussian distributed and has a flat frequency spectrum (white noise) that fluctuates with the standard deviation of the voltage.

number

[0357] Another type of stochastic noise is shot noise. Shot noise arises from the discrete nature of charge carriers and the fact that the probability of a charge carrier passing a point in a conductor at any given time is random. This effect is particularly important in semiconductor junctions, where charge carriers must overcome a potential barrier to conduct current. The probability of a charge carrier crossing the potential barrier is an independent random event, which causes fluctuations in the current through the junction. Shot noise has a flat frequency spectrum (white noise) with a Gaussian distribution and a standard deviation of the current fluctuations.

number

[0358] 27 HLN and WD Interface 27.1 Weight Output from Weight Diffuser Figure 38 shows how weights are output from the weight spreader device to the HLN. The voltages across the capacitors in the unit cells of the WD are taken and the voltage vector

number

[0359] The elements of this voltage vector can be rearranged using a permutation map. Physically, this permutation map corresponds to the routing scheme of the wires associated with the w(t) vector. By routing the wires in different ways, we obtain different permutations of the elements of this vector, which results in a new vector.

number

[0360] The permutations provide flexibility in how weight values ​​are assigned to specific parameters that appear in the HLN device.

[0361] 27.2 Inputting weights into HLN FIG. 39 shows how weights are input from WD to HLN.

[0362] The first step is to parameterize the HLN in terms of voltages. The HLN includes a variable resistor that can be implemented by a voltage-controlled circuit element. For example, Figure 39 shows a variable resistor as a voltage-controlled transistor, where the gate voltage controls the value of the resistor.

[0363] Specifically, the variable resistor is as follows:

number

[0364] Together, the voltage vectors α and β and the voltage matrix Γ represent the set of parameters used by the HLN device, which can be written as:

number

[0365] To accommodate the routing scheme that maps the outputs of the WD devices to the inputs of the HLN devices, the permutation function in Eq. (140) was applied.

[0366] 28 Post-Drift Network 28.1 PDN Overview As discussed previously and shown in Figure 37, the posterior drift network (PDN) plays a central role in generating the posterior distribution of the weights. The PDN takes in real-time measurements of the weights and outputs drift values ​​as voltages applied to the WD unit cells. Below, we discuss several different approaches to constructing a PDN. First, we consider a digital neural network for the PDN. Then, we consider the PDN as an analog neural network with digitally stored time-dependent weights. Finally, we consider the PDN as an analog device with analog time evolution.

[0367] 28.2 PDN as a Digital Neural Network One approach is to use a digital device to store and process the PDN. This digital device could be a CPU or an FPGA. The advantage of using an FPGA is that it can achieve low latency when interacting with the analog circuitry. Therefore, the feedback loop between the FPGA and the analog circuitry in the WD can be fast.

[0368] Figure 41 shows how the PDN interacts with the unit cells of the WD when it is stored on a digital device. The PDN takes the entire weight vector as input to the neural network. Analog-to-digital and digital-to-analog converters are used to appropriately convert signals between the analog and digital domains.

[0369] Relative to analog PDNs, digital PDNs have the advantage of being more flexible in design because their structure can be changed in software (rather than hardware). Also, digital neural networks allow for more parameters than their analog counterparts and are therefore more expressive.

[0370] On the other hand, one drawback of digital PDNs is latency. Analog-to-digital and digital-to-analog converters can add latency to the feedback loop between the WD and the PDN. Furthermore, the forward pass through the neural network involves matrix-vector multiplication, which can take time and increase latency.

[0371] 28.3 PDN as an Analog Neural Network with Time-Dependent Parameters Figure 42 shows a possible architecture for an analog PDN. This analog neural network has alternating layers of affine transformations followed by nonlinear transformations.

[0372] A nonlinear transformation is shown in Figure 42. It can include a nonlinear element (NLE) in series with a resistor. For example, the NLE can correspond to a diode or a transistor, and Sec. Section 25.4 describes these NLEs in more detail. A resistor in series with the NLE allows the current through the NLE to be read as an output voltage. The output voltage is therefore a nonlinear function of the input voltage, which is a desirable feature of activation functions commonly used in neural networks.

[0373] Figure 43 details a possible affine transformation. It can include a layer of resistors in series with the input, followed by a wire connecting the output. This serves to generate the output voltage as a weighted average of the input voltages. This creates a specific affine transformation. Also, by placing amplifiers after each layer, we can give this affine transformation more flexibility (e.g., allowing us to change the norm of the voltage vector).

[0374] The resistances of an affine layer are free parameters that are trained during the optimization process. Furthermore, these resistances can be time-dependent. Each resistance can be represented as a time-dependent function that is parameterized in some way. For example, the resistance can be represented as a linear function of time, with the slope and intercept of the linear function being trainable parameters. Physically speaking, the resistors can be transistors, and the time-dependent resistance can be implemented as a time-dependent voltage applied to the gate of the transistor.

[0375] Overall, this PDN structure has the advantage of being virtually latency-free because no ADCs or DACs are required at the interface between the WD and the PDN. This is also because the forward path through the PDN is essentially instantaneous, as voltages propagate through the neural network at the speed of light. On the other hand, this structure offers less flexibility to modify the design and may be less expressive than a digital PDN.

[0376] 28.4 PDN as an Analog Device with Analog Time Evolution Another way to implement an analog PDN is as follows: Instead of using time-dependent parameters, the time dependence can be incorporated through differential equations that describe the time evolution of the PDN.

[0377] Let s be a state variable associated with the PDN. This variable represents the drift vector output by the PDN and fed to the WD. For generality, we can write the differential equation for the time evolution of s as follows:

number

[0378] Using this general framework, we can write differential equations for the entire system, including the HLN, WD, and PDN. The state variables of these three devices evolve over time according to the following system of differential equations:

number

[0379] Next, the function e (Φ) Consider the possible forms of s. s can be a function of both time t and a weight vector w (w = w(t)) that is a function of time. Ultimately, w(t) evolves according to the differential equation in equation (134). In summary, s depends on time through two sources: it depends directly on t and indirectly on t via w(t).

[0380] This encourages a total derivative based approach, which can be written as

number

[0381] Assume that there are functions that approximate the partial derivatives in (145). These functions can be used to construct analog PDNs. The function q (Φ)(t,w) and r (Φ) (t,w) and (t,w). Rewriting equation (145) gives the model of the drift function.

number

number

[0382] In other words, a trainable function q (Φ) (t,w) is the partial derivative

number

[0383] Figure 44 provides a schematic diagram of a circuit 4440 that can output a drift value from a PDN using a full derivative approach. This circuit has several components. 1. The output is r (Φ) Voltage source 4441, which is (t, w), 2.r (Φ) The components of (t,w) are

number

number

number

number

number

[0384] Figure 44 depicts several circuits as black boxes: adder circuits 4444 and 4447, integrator circuit 4445, voltage source q θ (v,τ) and r θ 17 shows an example of an adder circuit (left) and an integrator circuit (right) suitable for use in score device 340.

[0385] Voltage vector to each voltage mixer

number

number

number

[0386] The circuit consists of a voltage source q (Φ) (t,w) and r (Φ) (t,w). One option is to construct these sources as a layered neural network with alternating layers of affine and nonlinear transformations. Thus, q (Φ) (t,w) and r (Φ) (t, w) can be constructed based on the circuit structure shown in FIG. 42, and the affine layer can be constructed similarly to the method shown in FIG.

[0387] 28.5 Incorporating the problem geometry into the PDN There are several different approaches to constructing a PDN, which can benefit from considering the geometry of the problem.

[0388] For example, when using an analog PDN, the problem geometry can be incorporated into the circuit structure shown in Figures 42 and 43. Each affine transformation in Figures 42 and 43 is mathematically represented by a matrix A. The problem geometry can be incorporated into each A matrix associated with these affine transformations. For example, these A matrices can be sparse, with non-zero elements corresponding only to the problem connectivity. In other words, these A matrices can be chosen to have a structure similar to the adjacency matrix A discussed previously. Matching the zero elements of the A matrix to the zero elements of the adjacency matrix A provides a technique for incorporating the problem geometry into an analog PDN.

[0389] Physically speaking, this corresponds to limiting the number of wires used in the circuit structure of Figure 43. In other words, the wires in Figure 43 can be bundled if the adjacency matrix A has non-zero elements associated with those wires. Thus, the adjacency matrix can be used to guide the incorporation of the geometry of the problem in the construction of the circuit of Figure 43.

[0390] 29 Loss Estimator 29.1 ELBO loss function estimator Recall that the loss function can be written as

number

number

[0391] The loss terms are expressed as κ and λ. L ELBO This can be written as =κ-λ.

number

[0392] An unbiased estimator can compute the loss function (or its gradient). This estimator can be low precision to save resources.

[0393] Next, we consider some estimators for the loss function. We consider sampling from q(w). We consider K samples w (k) , k=1,...,K. Then, we can define an unbiased estimator of the loss function as

number

number

[0394] A dataset with M data points

number

number

number

[0395]

number

number

number

number

number

number

[0396] An unbiased estimator of the log-likelihood can be based on inverse binomial sampling (IBS), which repeats samples until the sample returns a value of 1.

number

number

number

number

[0397] This is an unbiased estimator,

number

number

number

[0398] Another suitable biased estimator is based on fixed sampling. Although biased,

number

number

number

number

number

[0399] Furthermore, to define an additional estimator for the ELBO loss function, one can sample some of the data points in D rather than including the sum of all data points in D. This is often known as the mini-batch approach.

[0400] Below we explain how to estimate the two terms κ and λ separately.

[0401] 29.2 Estimating the log-likelihood term, kappa The loss term κ is an unbiased estimator

number

number

[0402]

number

number

[0403] This is done using the simultaneous time evolution of the HLN device and the post-WD device. (m) is used to initialize the input h(0) of the HLN device. The HLN and post-WD devices then evolve simultaneously over time to a particular weight trajectory w (k) This generates the output value

number

number

number

number

[0404]

number

number

[0405] Using the inverse binomial sampling (IBS) approach involves repeatedly running the HLN device with the same weight trajectory. This seems like it would require storing the weight trajectory in memory. Therefore,

number

number

number

number

[0406] this

number

number

[0407] 29.3 Digital Time Integration of the λ Term The λ loss term can be estimated using the following estimator,

number

[0408] Figure 45 shows

number

[0409] Define a vector.

number

number

number

[0410] As shown in Figure 45, the output of the WD3310 is amplified (by a factor of 2) using amplifier 4510, and then a voltage adder 4520 is used to add 2 watts. t and s (Φ) (t,w t ), the latter coming from the output of PDN3330. d(t) is prepared as a voltage vector and can be passed through ADC4530, which feeds the resulting digital signal to digital processor 4540. Digital processor 4540 applies g to d(t). -1 , take the square of the norm of the resulting vector, then integrate over time, and finally sum over all samples (sum over k). The result of this calculation is

number

[0411] 29.4 Analog Time Integration of the λ Term

number

[0412] Figure 46 shows the results using analog time integration.

number

[0413] However, after d(t) is prepared, instead of passing it through an ADC, the system -1 46 has an analog circuit (e.g., mixer 4610) that multiplies by g -1 We show how to do this multiplication in the special case where is a diagonal matrix, and -1 =diag(γ1,γ2,...,γ W ) Generally, g -1 If is an arbitrary matrix, analog circuitry can perform a matrix-vector multiplication. The result of this multiplication is to prepare the voltage vector u(t) that appears in equation (149).

[0414] After preparing u(t), the time integral can be calculated using circuit 4620, which implements the thermal method in Figure 19. That is, consider the thermal method for RMS (root mean square) voltage measurement. In this method, a voltage source generates an unknown time-dependent voltage. This voltage source heats resistor R1. A digital device (e.g., a CPU) applies a DC voltage to resistor R2 until both resistors reach the same temperature. At this point, the DC voltage is equal to the RMS voltage of the voltage source. Temperature sensing can be done with two semiconductor diodes, which can be viewed as thermistors. The overall circuit diagram is shown in Figure 19, including an operational amplifier. This subroutine is used to measure the voltage source u i (t), which is the i-th component of the voltage vector u(t), gives the subroutine:

number

[0415] More circuitry 4630 executes the subroutine of FIG. 18 to vector a={a i} and returns the square of the norm of the vector, producing an output voltage proportional to:

number

[0416] Analog-to-digital converter 4640 converts this voltage to a digital value. The system can perform this same procedure for many different samples (i.e., many different k). Digital device 4650 averages the results over the different samples and then

number

[0417] 30 Training a Bayesian Neural Network The loss estimator described above can be used as a subroutine in the training process of a Bayesian neural network. This training process takes the form of a hybrid analog-digital process, with two-way communication between analog devices (including HLN, WD, PDN, and LE) and a digital processor. The analog devices evaluate the loss function for fixed values ​​of the parameters and send the loss function values ​​to the digital processor. The digital processor then updates the parameters to new values ​​based on an optimization routine. This optimization routine can be selected in a variety of ways, including gradient-free methods and other standard methods. The new values ​​of the parameters are fed back to the analog devices, after which a new loss function value is determined and the process is repeated. This process can use a DAC and an ADC, respectively, to convert the parameter values ​​to analog voltages and the loss function values ​​to digital signals.

[0418] An alternative approach is to implement a gradient descent optimization routine by using analog devices to compute gradients and feeding these gradients into a digital process. To compute gradients using analog devices, the adjoint sensitivity method can be employed. Details of the adjoint sensitivity method are described below. Furthermore, this adjoint sensitivity method can be implemented on analog devices in the context of neural ODEs. Therefore, the electrical circuit shown below provides inspiration for how to implement the adjoint sensitivity method in this Bayesian neural network. Specifically, the following method can be extended to the case of neural SDEs.

[0419] 31 By-product Technology: Analog Neural ODE 31.1 Overview A subroutine of the thermodynamic AI system for Bayesian deep learning is an analog neural ODE, which corresponds to or represents the hidden layer network described in Section 25.3.

[0420] Neural networks represented or implemented by neural ODEs have broad applications in supervised machine learning and time series fitting. In this sense, the analog neural ODE architecture has applications beyond Bayesian deep learning and is relevant to deep learning and time series fitting in general. Section 32 details the application to time series fitting.

[0421] Recall that the architecture for implementing analog neural ODEs is shown in Figures 32, 33, 10, and 34. Figure 32 shows the unit cell. Figure 32 shows how the unit cells are connected. Figure 10 shows the connection structure. Figure 34 shows how to create an extended analog neural network to implement the extended neural ODE by expanding the space, i.e., by increasing the number of unit cells.

[0422] The free parameters in an analog neural network of a neural ODE are the voltages α, β, and Γ, as shown in Figure 39. In general, these free parameters do not have to come from a weight spreader (as shown in Figure 39) but can instead be supplied by a digital processor such as a CPU or FPGA. These parameters can also be supplied by an analog device that evolves the weights over time but does not add stochastic noise, in which case the device functions as a weight evolver (rather than a weight spreader).

[0423] 31.2 Adjoint Sensitivity Methods for Computing Gradients The adjoint sensitivity method provides one way to compute the gradient of a neural ODE, where the gradient refers to the derivative of the loss function L with respect to the parameters θ,

number

[0424] This method is Running a neural network in reverse time, describing the evolution of the hidden layer values ​​h(t), · Implement the second differential equation for the adjoint variable a(t), inversely in time, where

number

number

[0425] The variables h(t), a(t), and g(t) evolve together according to a system of simultaneous differential equations, which have the following form:

number

[0426] where the variable t runs backward, say from t=1 to t=0. If we want a differential equation that runs forward in time, we can perform a variable transformation τ=1-t. After this transformation, we get

number

[0427] 31.3 Analog Approach to Gradient Computation Consider an analog approach to the adjoint sensitivity method: an analog physical device can implement the system of differential equations in Equation (166), where r corresponds to the physical time over which the analog system evolves.

[0428] Achieving this goal involves introducing electrical circuits for the time evolution of a and g. This also involves reversing the direction of the drift of h, since a negative sign appears in the upper part of equation (166), unlike equation (117). The following describes these three subroutines: 1. Reverse the sign of the drift in h, 2. Using analog circuits to evolve a over time; 3. Calculating g using analog circuitry.

[0429] Drift reversal at 31.4 h Having the flexibility to apply a negative sign to the drift of h involves changing the unit cell of the analog neural network. The unit cell in Figure 32 does not allow this possibility. This can be seen, for example, from the differential equation in Eq. (122), since the coefficient J can only have a specific sign.

[0430] This additional flexibility (applying a minus sign to the drift) can be achieved with a circuit such as that shown in Figure 47, where a voltage mixer is used to mix the voltage source k(t): k(t)=k 順方向If (t) is a neural network operating in forward time, then k(t)=-k if it is a neural network operating in reverse time. 順方向 (1-t), which therefore provides a means of inverting the sign of the drift term in the analog neural ODE for analog neural networks.

[0431] 31.5 Companion Devices Next, we consider introducing a circuit for the time evolution of a. The circuit for evolving a is called an adjoint device.

[0432] Figure 48 shows a possible architecture for the companion device. Each unit cell consists of a series-connected capacitor

number

number

[0433] Furthermore, as shown in Figure 48, the unit cell has a resistance

number

[0434] The voltage across the capacitor is the vector v a and represents the state variable. In other words, the adjoint variable a is expressed as v a Then, the state variables evolve over time according to the differential equation

number

number

[0435] Furthermore, the matrix J a The elements of are given as follows:

number

[0436] Equation (167) is a matrix A.

number

number

[0437] One approach is to

number

number

number

number

number

number

[0438] Another approach is to use an analytical model for f in an analog fashion.

number

number

number

[0439] 31.6 g calculation circuit To calculate the gradient g, first

number

number

number

number

number

[0440] Then, during the time evolution of the adjoint device, the vector a can be read by obtaining the voltage across the capacitors of the adjoint device. This vector can be fed to an analog circuit (e.g., using a layer of resistors) that performs a matrix-vector multiplication. This matrix-vector multiplication is then converted into the matrix obtained in the previous step.

number

[0441] matrix-vector product

number

[0442] Although the above approach is analog, some operations can be performed digitally, e.g., matrix-vector multiplication.

number

[0443] 31.7 Training Analog Neural Networks The above method can be used to calculate the gradient with respect to the parameters θ of an analog neural ODE that represents an analog neural network. In practice, the values ​​of these parameters can be stored digitally in or connected to the memory of a CPU or FPGA and then fed to the analog circuitry as needed.

[0444] The training process can thus correspond to an optimization routine involving a hybrid analog-digital feedback loop, where a digital device supplies values ​​for the parameters θ, an analog device calculates the gradient, and then the digital device performs a gradient descent step before providing new values ​​for the parameters, and the process is repeated.

[0445] A DAC and an ADC can be used to convert the parameter values ​​into analog voltages and the gradient values ​​into digital signals, respectively, thus corresponding to using a gradient descent approach to training an analog neural network.

[0446] In a gradient-free approach to training an analog neural network, there is no need for adjoint sensitivity methods. Instead, the analog neural network can be trained using a loss function based on the output of the analog neural network. The value of this loss function can then be used in the context of a gradient-free optimization routine facilitated by a digital device.

[0447] 32 By-product application: Time series data fitting 32.1 Overview Time series data offer important applications related to financial analysis, market forecasting, epidemiology, and medical data analysis. Often, data for a particular point in time may exist at irregular time intervals. In such cases, it is useful to have a model that can always make predictions, thus interpolating between data points and extrapolating beyond the data, for example, to make predictions into the future where no data exists.

[0448] Discrete neural networks, such as recurrent neural networks, have been used in the past to interpolate and extrapolate time series data, but neural networks that implement or follow latent ODEs have been shown to outperform recurrent neural networks in these tasks.

[0449] Latent ODEs are essentially the same continuous-time models as neural ODEs, but different names are used to distinguish between their applications. Specifically, neural ODEs (latent ODEs) represent neural networks used in supervised machine learning (fitting time series data). Latent ODEs can be viewed as parameterized ODEs, with parameters trained to fit the corresponding neural network to time series data (according to some loss function).

[0450] 32.2 Analog Latent ODEs Section 31 presents an analog architecture for implementing neural ODEs, which can be used to execute subroutines of latent ODE models for fitting time series data.

[0451] Figure 50 provides a schematic diagram for an analog implementation of a latent ODE. The analog latent ODE implementation of Figure 50 has three components or subsystems: 1. Encoder, 2. Analog Neural ODE Processor, 3. It is a decoder.

[0452] Training data is provided as observations from several time series. These time series observations are fed into an encoder. The encoder has free parameters that can be trained. For example, the encoder could be a recurrent neural network. The output of the encoder could be an initial vector of hidden layer values ​​h(0), or the output could be a probability distribution from which h(0) is sampled. If the encoder is stored on a digital device, its output passes through a DAC and then is fed as an analog signal to the analog neural ODE.

[0453] The Analog Neural ODE Processor presented in Section 31 provides a latent space for the latent ODE. This latent space is initialized to h(0) by the encoder. The values ​​of the hidden layer then evolve over time according to the differential equations that describe the Analog Neural ODE. Recall that Figures 32, 33, 10, and 34 provide details on how to construct the Analog Neural ODE Processor.

[0454] The hidden layer value h(t k ) is calculated by measuring the voltage of a capacitor in an analog neural ODE processor at various times {t k}. This set of values ​​{h(t k)} can be fed to a decoder, which could be a neural network stored on a digital device. In this case, the ADC k )}, and feed the resulting digital signal to a decoder. The decoder may have free parameters to be trained. The decoder may be stochastic, so that the output predicted by the decoder depends on the hidden layer values ​​{h(t k )}. The advantage of making the decoder stochastic is that it can be used to simulate noisy or stochastic processes.

[0455] The decoder output corresponds to the predictions that the latent ODE model makes for the true time series. These predictions may extend beyond the time interval associated with the observations, in which case the predictions correspond to extrapolated values.

[0456] The parameters of the encoder, decoder, and analog neural ODE processor are optimized, and a training process is performed to minimize or maximize a loss function, which essentially corresponds to fitting to the time series data. For example, the training process can use a loss function based on the evidence-based lower bound (ELBO), although other loss functions are possible. A gradient-based approach can be taken, and the adjoint sensitivity method discussed in Section 31 can be used to calculate the gradient.

[0457] 32.3 Extension to analog latent SDEs The framework and architecture shown in Figure 50 can be extended as follows: An analog neural SDE processor can be substituted in place of the analog neural ODE processor for the latent space used in Figure 50. Replacing the analog neural ODE with an analog neural SDE results in an analog latent SDE. Analog latent SDEs can be used to model time series data generated by stochastic processes. This has useful applications in financial and market analysis.

[0458] Figure 51 shows an analog neural SDE processor with a stochastic noise source in each unit cell of the analog neural ODE processor architecture. Voltage sources Bj represent the stochastic noise sources. Apart from this additional voltage source, the rest of the architecture of the analog neural SDE processor can be the same as the analog neural ODE processor.

[0459] An analog neural SDE processor inserted into the latent space shown in Figure 50 in place of the analog neural ODE processor can be used to fit and extrapolate time series data from sources with stochastic properties.

[0460] 33 General Framework Here, we provide a general framework that encompasses multiple applications, including those previously discussed (diffusion models and Bayesian deep learning) as well as others.

[0461] 33.1 Unifying Thermodynamic AI Algorithms The goal here is to present a thermodynamic hardware paradigm relevant to multiple AI applications. One step towards this goal is to mathematically unify different AI applications under the same framework. The applications considered are shown in Figure 52 and listed below: 1. Generative Diffusion Model 2. Bayesian Neural Networks 3. Monte Carlo Inference 4. Annealing 5. Time Series Forecasting

[0462] With careful consideration, we can formulate a mathematical framework that encompasses all of the above algorithms as special cases, and say that these algorithms belong to a class called thermodynamic AI algorithms.

[0463] At a conceptual level, a thermodynamic AI algorithm can be defined as an algorithm consisting of two subroutines: 1. A subroutine that evolves a stochastic differential equation (SDE) over time. 2. A subroutine in which Maxwell's demon observes the state variables of the SDE and applies the drift term accordingly.

[0464] At a mathematical level, we propose that a thermodynamic AI algorithm simulates or implements the following set of equations (or a subset of them):

number

number

number

[0465] These can be seen to correspond to Newton's laws of motion with the addition of diffusion and friction. In these equations, p, x, and f represent momentum, position, and force, respectively. The matrices M, D, and B are hyperparameters, with M being the mass matrix and D being the diffusion matrix. The dw term is the Wiener process. Finally, U θ is a (trainable) potential energy function. In general, much application-specific information about the task to be solved is needed to determine the potential energy function U θFor readability, the above equation omits the dependence of the variables on time t and space x.

[0466] This integration is crucial for developing a hardware paradigm that is broadly applicable to a variety of AI algorithms. Below we describe the fundamental building blocks of this hardware paradigm.

[0467] 33.2 Basic Building Blocks [Table 1] We now discuss the fundamental building blocks of thermodynamic AI hardware. As the name "thermodynamics" suggests, thermodynamic systems are inherently dynamic in nature. Therefore, the fundamental building blocks should also be dynamic. This contrasts with classical or qubits, where the state of the system ideally remains fixed and does not change unless actively modified by gates. Thermodynamic building blocks should passively and naturally evolve over time without the application of gates.

[0468] But what kind of dynamic process should we observe? A reasonable suggestion is a stochastic Markov process. Naturally, this should be continuous in time, since no point in time is more special than any other. Thus, the discrete building blocks, which we call s-bits, follow a continuous-time Markov chain (CTMC), where the "s" in s-bits stands for stochastic.

[0469] For the continuous structural blocks we call s-modes, a natural analogue is Brownian motion (also called Wiener process). This process can be assumed to be a martingale (as is usually assumed in Brownian motion), meaning it is unbiased. We use s-units as a general term to encompass both s-bits and s-modes.

[0470] For comparison, we can consider p-bits (p-modes), the basic building blocks of stochastic systems in the discrete (continuous) case. p-bits can be thought of as random number generators that randomly produce 0 or 1. The analogue to this in the continuous case, namely p-modes, can be a random number generator that produces real numbers according to a Gaussian distribution with zero mean and variance.

[0471] While p-bits (p-modes) are clearly different from s-bits (s-modes), there is some connection between them: the latter can be thought of as the time integral of the former. For example, s-modes can be seen as the I-to integral of p-modes. In this sense, if one has access to both p-bits (p-modes) and a time integration device, one can experimentally construct s-bits (s-modes).

[0472] 33.3 Physical Realization of s-mode 33.3.1 Individual s-modes s-modes represent continuous random variables whose evolution is governed by drift, diffusion, or other physical processes (similar to particles undergoing Brownian motion). At the heart of the physical implementation of such variables is a source of stochasticity. A natural starting point for implementing thermodynamic AI hardware is analog electrical circuits, because these circuits have inherent fluctuations that can be exploited for computation.

[0473] The most common source of noise in electrical circuits is thermal noise. Thermal noise, also called Johnson-Nyquist noise, arises from the random thermal oscillations of charge carriers in conductors, resulting in fluctuations in the voltage or current in the conductor. Thermal noise is Gaussian distributed, with a flat frequency spectrum (white noise) that fluctuates with the standard deviation of the voltage.

number

[0474] Another type of electrical noise is shot noise. Shot noise arises from the discrete nature of charge carriers and the fact that the probability of a charge carrier passing a point in a conductor at any given time is random. This effect is particularly important in semiconductor junctions, where charge carriers must overcome a potential barrier to conduct current. The probability of a charge carrier crossing the potential barrier is an independent random event, which causes fluctuations in the current through the junction. Shot noise is Gaussian distributed and has a flat frequency spectrum (white noise) with a standard deviation of the current fluctuations.

number

[0475] In general, the physical implementation of any s-mode should be such that the amplitude of its stochasticity is controllable independently of other system parameters. For example, electrical thermal noise and shot noise are known to have tuning knobs that provide some control over the noise amplitude.

[0476] Furthermore, it is necessary to ensure that the amplitude of the fluctuations is compatible with the rest of the system, i.e., measurable. Thermal noise and shot noise sources typically have voltage fluctuations on the order of a few microvolts or less. To make these fluctuations measurable with on-chip analog-to-digital converters, which measure voltages on the order of hundreds of millivolts, amplification is required. This amplification can be done using single or multi-stage voltage amplifiers. It is also possible to independently control the amplitude of the fluctuations by using variable gain amplifiers.

[0477] The s-mode can be expressed through the transition of any degree of freedom of an electrical circuit.

[0478] If we choose the voltage at a particular node in the circuit as the degree of freedom, a simple stochastic voltage noise source can play the role of the s-mode. This can be achieved by using a noisy resistor with a non-zero temperature. The circuit diagram in Figure 53 shows a typical equivalent noise model for a noisy resistor, consisting of a stochastic voltage noise source δv(t) in series with an ideal (noiseless) resistor of resistance R. The resistor's intrinsic terminal capacitance C is also added to the equivalent resistor model. We will use the voltage at node 1 (here simply labeled v(t)) as the s-mode. The transition of the s-mode in this case follows the SDE model:

number

[0479] The voltage fluctuations in (174) have the characteristics of ordinary Gaussian white noise, with 〈δv(t)〉=0 and 〈δv(t)δv(t′)〉=2k B TRδ(tt′), where < > denotes the statistical mean and δ(tt′) denotes the Dirac delta function. The form of the SDE can be seen to include a drift term proportional to v(t) and a diffusion or stochastic term proportional to δv(t).

[0480] For s-modes to be more computationally useful, their inherent stochastic evolution needs to be constrained by algorithmic properties. For example, adding a capacitor in series with a noise resistor can add a drift term to the s-mode evolution. In general, other electrical components such as inductors or nonlinear elements can be added to further constrain the evolution of s-modes.

[0481] 33.3.2 s-mode connection When building a system of many s-modes, it is most desirable to introduce some form of connection between them to express the correlations and geometric constraints. Again, the medium of analog electrical circuits provides a natural choice for connecting s-modes.

[0482] As a first example, two circuits of the type described in Section 33.3.1 can be connected through a resistor as depicted in the top panel of Figure 54. The connected s-modes are represented by the voltages at node 1 and node 2, connected through their drift terms (time dependence omitted for readability).

number

number

[0483] This system of simultaneous equations can be simplified and expressed in matrix form as follows:

number

number

number

[0484] A second way to connect two s-modes is to use a capacitor as the connecting element, as shown in the bottom panel of Figure 54. In this configuration, the two connected s-modes, represented by the voltages at Node 1 and Node 2, have a connection with drift and diffusion (time dependence omitted for readability).

number

number

[0485] For simplicity of notation, using the same notation as in (177), this system of simultaneous equations can be expressed in matrix form as follows:

number

number

[0486] 33.4 Maxwell's Demon Maxwell's demon is a thermodynamic concept similar to cooling, as it can decrease the entropy of an external system over time by interacting with it. Here, the demon acts as an intelligent observer that periodically collects data (i.e., measures) from the system and takes some action on the system based on the information collected. Classic examples involve gas mixtures and physical barriers, as shown in Figure 11, but can also be implemented by a variety of physical means, including electrical circuits.

[0487] We argue that Maxwell's demon is (1) a key ingredient in thermodynamic AI systems due to the complex entropy transitions required for AI applications, and (2) can be easily implemented in practice on several different hardware architectures.

[0488] Regarding the first point, AI applications such as Bayesian inference aim to approximate posterior distributions, and it is known that such posterior distributions can be highly complex and multimodal. Similarly, generative models aim to handle arbitrary data distributions. Therefore, generating only Gaussian distributions, as an isolated s-mode system does, is insufficient for these applications. Regarding the second point, we describe below how to implement Maxwell's demon in hardware.

[0489] 33.4.1 Digital Maxwell's Demon We consider Maxwell's demon (MD) as the hardware component of a thermodynamic AI system. MD devices can be constructed in various ways, using digital or analog approaches. A digital MD system is very simple to explain; it can correspond to a neural network stored in a digital processor such as a CPU or FPGA. In this case, the state vector v must be transmitted to the digital processor (which serves as the input to the neural network), and then the proposed action of Maxwell's demon (i.e., the output of the neural network) must be transmitted back to the thermodynamic hardware. Therefore, a means of interconverting signals between the thermodynamic hardware and the digital processor is required. This is illustrated in a diagram such as Figure 12 or Figure 41, where the interconversion is shown as an analog-to-digital converter (ADC) and a digital-to-analog converter (DAC).

[0490] 33.4.2 Analog Maxwell's Demon Analog MD devices may allow for closer integration with other thermodynamic hardware. Furthermore, this may avoid signal cross-conversion. Here, we describe a force-based approach for MD devices; alternative approaches (such as the total derivative approach discussed in the previous section) are also possible.

[0491] Recall from Newtonian mechanics that the force f(x) on a system at position x is given by the gradient of the potential energy function U(x), and the momentum p is proportional to the time derivative of the position.

number

[0492] We will now discuss how the MD device performs the mapping from input to output in equation (182). We assume that the MD device has a latent or hidden variable corresponding to the position vector x(t). The latent variable x(t) is stored in the MD device's memory and evolves over time. Specifically, it evolves over time according to the differential equation in equation (181). Therefore, for the latent variable:

number

[0493] The MD device also stores a potential energy function U0(t, x(t)). In general terms, this potential energy function can be time-dependent. This time-dependence is important for certain applications, such as annealing, where we want the potential energy function to change over time. Furthermore, we allow the potential energy function to depend on a set of parameters θ. These parameters are trainable and are trained during a training process, such as a diffusion model. Thus, the potential energy function represents the trainable part of the MD device.

number

[0494] Internally, the MD device generates a force by combining Equation (183) and Equation (184). Specifically, as shown in Figure 55, the MD device uses Equation (181) to calculate the gradient of the potential energy function at time t and position x(t). The result of this calculation is the output of the MD device.

number

[0495] 33.5 Fitting diffusion models to a general framework Figure 52 shows how diffusion models and other applications fit into this general framework.

[0496] For the diffusion model, by changing the variable τ=Tt in the backward process, the SDE equation can be rewritten as follows:

number

number

[0497] In this case, both t and τ proceed forward in time, i.e., dt and dτ are positive increments in these equations.

number

[0498] A diffusion model can be fitted into this framework using the following mapping: (Diffusion process) (s-mode device) (188) (Score Network) (Maxwell's Demon Device) (189)

[0499] The mathematical diffusion process in the diffusion model can be mapped to the physical diffusion process in the s-mode device. Similarly, the score vector output by the score network corresponds to the vector d(t,v(t)) output by the MD device. Therefore, the diffusion model fits into the general framework of thermodynamic AI systems.

[0500] 33.6 Fitting Bayesian Deep Learning to a General Framework Bayesian deep learning, as described in Section 24 above, fits into this general framework as follows: To fit it into this framework, we make the following mapping: (Weight Diffusion Device)(s-mode Device) (190) (Post-Drift Network) (Maxwell's Demon Device) (191)

[0501] The weight diffuser corresponds to the s-mode device, and the posterior drift network corresponds to the Maxwell's demon device. The weight diffuser uses the s-mode transition to generate the weight trajectory w tis sampled and imported into HLN. Maxwell's demon is used to generate complex s-mode transitions that generate weight trajectories according to the posterior distribution.

[0502] 33.7 Fitting Annealing to a General Framework Annealing processes, such as simulated annealing, fit into this general framework as follows: Suppose we have an optimization problem, where the loss function of interest, L(x), is a continuous function of the N-dimensional state variables, x. In this case, optimizing x solves the optimization problem in the context of simulated annealing. In this setting, we consider proposing a system of equations for the state variables, x(t), and the auxiliary variables, p(t).

number

number

[0503] where W is the N-dimensional Brownian motion, S is the N×N-dimensional lower triangular matrix, i.e., D=SS T The first differential equation is called the optimization ODE, and equation (193) is called the auxiliary SDE. The evolution of the state variable x(t) is effectively stochastic because it is coupled with the auxiliary variables that evolve through the auxiliary SDE. Equations (192) and (193) represent the simulated annealing process.

[0504] Equations (192) and (193) are special cases of equations (169) and (170), which are a general framework for thermodynamic AI hardware. Specifically, there is the following mapping to this hardware: (Auxiliary SDE)(s-mode device) (194) (Optimization ODE) (Evolution of latent variables in Maxwell's demon device) (195)

[0505] The idea is that an auxiliary SDE that describes the evolution of p can be implemented on the s-mode device. Furthermore, -∇L(x) corresponds to the vector d output from the Maxwell's demon on this hardware. The optimization ODE is then mapped to the evolution of the latent variables in the Maxwell's demon device. This uses a force-based Maxwell's demon, as discussed above and shown in Figure 55. Note also that the mass matrix that appears in this framework is set equivalent for this application, i.e., M=I.

[0506] 34 Conclusion While various inventive embodiments have been described and illustrated herein, those skilled in the art will readily conceive of other means and / or structures for performing the functions and / or obtaining the results and / or one or more advantages described herein, and these variations and / or modifications are deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, or configurations will depend on the specific application in which the inventive teachings are used. Those skilled in the art will recognize or be able to find, using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. Accordingly, it should be understood that the foregoing embodiments are presented by way of example only, and that, within the scope of the appended claims and their equivalents, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, or method described herein. Furthermore, where not mutually inconsistent, any combination of these features, systems, articles, materials, kits or methods, or any two or more thereof, is within the inventive scope of this disclosure.

[0507] Also, various inventive concepts may be embodied as one or more methods, examples of which are provided. Acts performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed such that acts are performed in an order different from that shown, and may include some acts being performed simultaneously despite being shown as sequential acts in the illustrated embodiments.

[0508] All definitions defined and used herein should be understood to supersede any dictionary definition, definition in any document incorporated by reference, or ordinary meaning of the defined term, or more than one.

[0509] The indefinite articles "a" and "an," as used in this specification and claims, unless clearly indicated to the contrary, should be understood to mean "at least one."

[0510] The term "and / or," as used in the specification and claims, should be understood to mean "either or both" of the associated components, i.e., components that are present together in some cases and separately in other cases. Multiple components listed with "and / or" should be interpreted similarly, i.e., meaning "one or more" of the associated components. Other components other than the components specifically identified in the "and / or" clause may optionally be present, whether related to the components specifically identified in the clause. Thus, as a non-limiting example, a reference to "A and / or B," when used in conjunction with the open-ended language of "comprising," can refer in one embodiment to A only (optionally including components other than B), in another embodiment to B only (optionally including components other than A), in yet another embodiment to both A and B (optionally including other components), etc.

[0511] As used in this specification and in the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when distinguishing items in a list, "or" or "and / or" should be construed as inclusive, i.e., including at least one element of any number or list of elements, and also including more elements, and optionally including additional, unlisted items. Terms clearly indicating the contrary, such as "only one of" or "exactly one of," or, when used in the claims, "consisting of," refer to the inclusion of exactly one element from the list of elements. Generally, as used herein, the word "or" shall only be construed as indicating exclusive alternatives (i.e., "one or the other but not both") when preceded by terms of exclusivity such as "either," "one of," "only one of," or "exactly one of." As used in the field of patent law, the term "consisting essentially of" when used in the claims shall have its ordinary meaning.

[0512] The phrase "at least one," as used in the specification and claims, in connection with a list of one or more ingredients, should be understood to mean at least one ingredient selected from any one or more ingredients in the list, and does not necessarily imply the inclusion of at least one of every ingredient specifically listed in the list of ingredients, nor does it exclude any combination of ingredients in the list of ingredients. This definition also permits the optional presence of other ingredients beyond those specifically identified, whether or not they are related to the ingredients specifically identified in the list of ingredients to which the phrase "at least one" refers. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B"), or, equivalently, "at least one of A and / or B" can refer in one embodiment to at least one, and optionally more than one, A, and no B (and optionally including components other than B); in another embodiment to at least one, and optionally more than one, B, and no A (and optionally including components other than A); in yet another embodiment to at least one, and optionally more than one, A, and at least one, and optionally more than one, B (and optionally including other components); etc.

[0513] In the claims and specification above, all transitional phrases such as "comprising," "including," "carrying," "having," "containing," "involving," "holding," and "composed of" are to be understood to be open-ended, i.e., meaning "including, but not limited to." Only the transitional phrases "consisting of" and "consisting essentially of" are closed or semi-closed transitional phrases as defined in the U.S. Patent and Trademark Office Manual of Patent Examining Procedure, Section 2111.03.

Claims

1. 1. A system for running a generative diffusion model on a dataset, comprising: a physical system having a plurality of degrees of freedom, each degree of freedom of the plurality of degrees of freedom having a continuous state variable associated with a corresponding feature of the data set and evolving according to a corresponding differential equation having an adjustable diffusion term and an adjustable drift term; system.

2. The system of claim 1 , further comprising a processor operatively connected to the physical system to reduce the entropy of the continuous state variables to generate an output of the generative diffusion model.

3. The system of claim 2 , wherein the processor is configured to decrease the entropy of the continuous state variables by reading the continuous state variables and modifying at least one of the adjustable drift terms.

4. The system of claim 2 , wherein the processor is configured to reduce the entropy of the continuous state variables by executing a score network.

5. The system of claim 2 , wherein the processor is configured to be trained by an optimization routine that reduces a loss function.

6. The system of claim 5 , wherein the physical system is configured to assist in estimating the loss function.

7. The system of claim 6 , wherein the processor includes analog circuitry configured to assist in estimating the loss function.

8. The system of claim 7 , wherein the estimation of the loss function includes a simultaneous time evolution of the physical system and the analog circuitry within the processor.

9. The system of claim 8 , wherein the loss function quantifies the degree of score matching.

10. The system of claim 1 , wherein the degrees of freedom are physically connected to each other according to a geometry of a problem associated with the dataset.

11. 10. The system of claim 1, further comprising a function generator operatively connected to the physical system for multiplying the adjustable drift term and the adjustable diffusion term by an arbitrary time-dependent function to modify the differential equation governing the evolution of the continuous state variable.

12. 10. The system of claim 1, further comprising a digital device operatively connected to the physical system to upload the data set to the degrees of freedom and to download new data corresponding to measurements of values ​​of the continuous state variables after evolution of the continuous state variables.

13. The system of claim 1 , wherein the physical system includes a network of electrical circuits, each electrical circuit in the network of electrical circuits providing a corresponding degree of freedom in the plurality of degrees of freedom.

14. Each electrical circuit in the network of electrical circuits is a capacitor having a charge or voltage encoding the continuous state variable; a stochastic noise source in series with the capacitor that generates the adjustable diffusion term; a resistor in series with the capacitor to generate the adjustable drift term; The system of claim 13 , comprising:

15. The system of claim 14 , wherein the stochastic noise source is a thermal noise source, a shot noise source, or both.

16. Each electrical circuit in the network of electrical circuits is a first adjustable voltage source operatively connected to the stochastic noise source to adjust the adjustable diffusion term of the corresponding differential equation; a second adjustable voltage source operatively connected to the resistor for adjusting the adjustable drift term of the corresponding differential equation; The system of claim 14 further comprising:

17. Each electrical circuit in the network of electrical circuits is a variable resistor in parallel with the capacitor, the variable resistor being controlled by an adjustable voltage source, for adjusting the adjustable drift term of the differential equation; an amplifier operatively connected to the stochastic noise source for amplifying the output of the stochastic noise source to adjust the adjustable diffusion term of the differential equation; The system of claim 14 further comprising:

18. 20. The system of claim 17, wherein the amplifier has a variable gain determined by an additional variable resistor, the resistance of which is controlled by an additional adjustable voltage source.

19. The system of claim 13 , further comprising switches that connect the electrical circuits in the network of electrical circuits according to a problem geometry associated with the data set.

20. 14. The system of claim 13, further comprising a processor operatively connected to the network of electrical circuits to reduce the entropy of the continuous state variables to generate an output of the generative diffusion model.

21. 21. The system of claim 20, wherein the processor is configured to decrease the entropy of the continuous state variables by reading the continuous state variables and modifying at least one of the adjustable drift terms.

22. 21. The system of claim 2 or claim 20, wherein the processor includes analog circuitry configured to evolve over time simultaneously with the physical system.

23. 23. The system of claim 22, wherein the analog circuitry is configured to continuously output a score value prediction as an analog signal for use as an input to the physical system.

24. 24. The system of claim 23, wherein the score value is generated by physically modeling partial derivatives of the score value with respect to time and the continuous state variable using a trainable physical device.

25. 25. The system of claim 24, wherein the trainable physical device used to model the partial derivatives is an artificial neural network.

26. 25. The system of claim 24, further comprising an integration circuit operatively connected to the trainable physical device that generates the score value by integrating an output of the trainable physical device over time.

27. 23. The system of claim 22, wherein the processor further comprises a field programmable gate array (FPGA).

28. the analog circuit includes a network of electrical circuits, each electrical circuit in the network of electrical circuits providing a corresponding degree of freedom; A capacitor; a resistor in series with the capacitor; a voltage source in series with the capacitor; 23. The system of claim 22, comprising:

29. 30. The system of claim 28, wherein at least one electrical circuit in the network of electrical circuits is capacitively coupled to at least one other electrical circuit in the network of electrical circuits according to a problem geometry associated with the data set.

30. 30. The system of claim 28, wherein the voltage source is a multi-layer neural network configured to receive the continuous state variables as inputs and to output a voltage value for each electrical circuit in the network of electrical circuits.

31. The system of claim 1 , wherein the physical system includes a switch configured to operate between a setting for a forward diffusion process and a setting for a backward diffusion process.

32. 1. A system for implementing a Bayesian neural network, comprising: a physical system having a plurality of degrees of freedom, each degree of freedom having a continuous state variable associated with a corresponding weight in the Bayesian neural network and evolving according to a corresponding differential equation having a diffusion term and a drift term; system.

33. 33. The system of claim 32, wherein the physical system comprises a network of electrical circuits, each electrical circuit in the network of electrical circuits providing a corresponding degree of freedom.

34. Each electrical circuit in the network of electrical circuits is a capacitor having a charge or voltage encoding the continuous state variable; a stochastic noise source in series with the capacitor generating the diffusion term; a resistor in series with the capacitor generating the drift term; 34. The system of claim 33, comprising:

35. 35. The system of claim 34, wherein the stochastic noise source is a thermal noise source, a shot noise source, or both.

36. Each electrical circuit in the network of electrical circuits is 35. The system of claim 34, further comprising an amplifier operatively connected to the stochastic noise source to amplify the output of the stochastic noise source to adjust the diffusion term of the corresponding differential equation.

37. 34. The system of claim 33, further comprising switches that connect the electrical circuits in the network of electrical circuits according to a problem geometry associated with the Bayesian neural network.

38. 33. The system of claim 32, further comprising a processor operatively connected to the physical system to modify the evolution of the continuous state variables of the differential equation to generate a posterior distribution of the weights of the Bayesian neural network.

39. 39. The system of claim 38, wherein the processor is configured to read the continuous state variables from the physical system and modify at least one of the drift terms.

40. 39. The system of claim 38, wherein the processor is configured to be trained by an optimization routine that reduces a loss function.

41. 41. The system of claim 40, wherein the physical system is configured to assist in estimating the loss function.

42. 41. The system of claim 40, further comprising analog circuitry operatively connected to the processor to assist in calculating a time integral to estimate the loss function.

43. 41. The system of claim 40, wherein the physics system is configured to assist in estimating a gradient of the loss function.

44. 40. The system of claim 38, wherein the processor comprises a field programmable gate array (FPGA).

45. 40. The system of claim 38, wherein the processor comprises analog circuitry configured to evolve over time simultaneously with the physical system.

46. 46. ​​The system of claim 45, wherein the analog circuitry is configured to continuously output the prediction of the drift term as an analog signal for use as an input to the physical system.

47. 47. The system of claim 46, wherein the predictions are generated by physically modeling partial derivatives of the drift term with respect to time and the continuous state variables using a trainable physical device.

48. 48. The system of claim 47, wherein the trainable physical device used to model the partial derivatives implements an artificial neural network.

49. 48. The system of claim 47, further comprising at least one integrator circuit operatively connected to the trainable physical device that generates the drift term by integrating an output of the trainable physical device over time.

50. 33. The system of claim 32, further comprising a processor that implements a model having weights corresponding to continuous state variables of the physical system.

51. 51. The system of claim 50, wherein the processor comprises a field programmable gate array (FPGA).

52. 51. The system of claim 50, wherein the processor includes a network of electrical circuits, each electrical circuit in the network of electrical circuits providing a corresponding latent variable of the model.

53. Each electrical circuit in the network of electrical circuits is a capacitor having a charge or voltage encoding the corresponding latent variable; a nonlinear element providing a nonlinear activation function for said model; 53. The system of claim 52, comprising:

54. 54. The system of claim 53, wherein the nonlinear element comprises at least one of a transistor or a diode.

55. 53. The system of claim 52, wherein the network of electrical circuits includes a variable resistor having a resistance controlled by an external voltage source.

56. 53. The system of claim 52, wherein the network of electrical circuits includes a number of unit cells equal to or greater than the dimension of a feature space of inputs to the Bayesian neural network.

57. 1. A system for implementing neural ordinary differential equations (ODEs), comprising: a physical system having a plurality of degrees of freedom, each degree of freedom having a corresponding continuous state variable associated with a hidden layer unit of said neural ODE and evolving according to a differential equation having a nonlinear term and an affine term; system.

58. 58. The system of claim 57, wherein the physical system comprises a network of electrical circuits, each electrical circuit in the network of electrical circuits providing a corresponding degree of freedom.

59. Each electrical circuit in the network of electrical circuits is a capacitor having a charge or voltage encoding the corresponding continuous state variable; a nonlinear element that generates the corresponding nonlinear term; 59. The system of claim 58, comprising:

60. 59. The system of claim 58, wherein the network of electrical circuits includes a variable resistor that generates the affine term, the variable resistor having a trainable resistance.

61. 58. The system of claim 57, wherein the physical system has a reversible transition.

62. 62. The system of claim 61, further comprising an additional physical system that evolves an adjoint variable simultaneously with the evolution of the continuous state variables.

63. 58. The system of claim 57, wherein the physical system is configured to perform fitting or extrapolation of time series data.

64. 60. The system of claim 58, wherein each electrical circuit in the network of electrical circuits includes a corresponding stochastic noise source.

65. 1. A processor configured to implement a Bayesian neural network, comprising: an analog weight spreader with stochastic transitions that generates a trajectory of weight values ​​for the Bayesian neural network; a hidden layer network operatively connected to the analog weight spreader for generating an output vector based on an input vector and the trajectory of the weight values; a posterior drift network operatively connected to the analog weight spreader and the hidden layer network, the posterior drift network guiding the trajectory of the weight values ​​based on the output vector; 12. A processor comprising:

66. 66. The processor of claim 65, wherein the trajectory of weight values ​​is a solution of a stochastic differential equation describing the stochastic evolution of the analog weight spreader.

67. The analog weight spreader includes a plurality of interconnected unit cells, each interconnected unit in the plurality of interconnected unit cells comprising: a resistor representing the drift term of the stochastic differential equation; a stochastic noise source in series with the resistor, the stochastic noise source generating a diffusion term in the stochastic differential equation; a capacitor in series with the resistor and the stochastic noise source, the capacitor encoding a continuous state variable that evolves according to the stochastic differential equation; 67. The processor of claim 66, comprising:

68. 68. The processor of claim 67, wherein the stochastic noise source includes at least one of a shot noise source or a thermal noise source.

69. 66. The processor of claim 65, wherein the analog weight spreader is configured to be switched between a first configuration in which the analog weight spreader generates a posteriori weight distribution and a second configuration in which the analog weight spreader generates a pre-emptive weight distribution.

70. 66. The processor of claim 65, wherein the hidden layer network comprises at least one of a processing unit or a field programmable gate array.

71. 66. The processor of claim 65, wherein the hidden layer network comprises a network of analog unit cells.

72. Each analog unit cell in the network of analog unit cells comprises: a capacitor encoding continuous state variables that evolve according to a neural ordinary differential equation (ODE); a resistor in series with the capacitor, the resistor providing a time derivative of the continuous state variable; a voltage source in parallel with the capacitor that at least partially provides a linear drift term for the continuous state variable; a nonlinear element in parallel with the capacitor, the nonlinear element providing a nonlinear drift term for the continuous state variable; 72. The processor of claim 71, comprising:

73. 73. The processor of claim 72, wherein the non-linear element comprises at least one of a transistor or a diode.

74. 72. The processor of claim 71, wherein the analog unit cells in the network of analog unit cells are connected by switches and resistor bridges in a geometry based on the input vector.

75. the post-drift network includes at least one of a processing unit or a field programmable gate array; an analog-to-digital converter operatively connecting the analog weight spreader to the post-drift network and converting an analog output from the analog weight spreader into a digital input to the at least one of the processing unit or the field programmable gate array; a digital-to-analog converter operatively connecting the post-drift network to the analog weight spreader and converting a digital output from the at least one of the processing unit or the field programmable gate array into an analog input to the analog weight spreader; 66. The processor of claim 65, further comprising:

Images