Bistable Resistive Coupling System

Abstract

The bistable resistive coupling system includes a plurality of visible nodes, a plurality of hidden nodes, and a plurality of coupling elements electrically connected to a visible node of the plurality of visible nodes and a hidden node of the plurality of hidden nodes, respectively, each of the plurality of coupling elements including a programmable resistor. Also described are coupling apparatus for first and second nodes in the network, and a method of training the bistable resistive coupling system.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 176,247, filed April 17, 2021, which is incorporated by reference in its entirety.

[0002] It is clear that nature is constantly performing a lot of computations, such as solving differential equations and random sampling. For example, transistors can be turned on and off based on natural laws, and are the basis of most computers today. However, using these laws is different from harnessing nature's computational power at a higher level, for example to solve an entire problem. In fact, some very powerful algorithms are inspired by nature. It is not difficult to imagine that if a computational substrate is based on nature, it can be used to solve a particular set of problems more quickly and efficiently than mapping that problem to a von Neumann interface. In such efforts, Ising machines have recently made rapid progress.

[0003] Simply put, Ising machines exploit nature to seek low energy states for systems of coupled spins. A wide variety of problems (indeed, all NP-complete problems) can be expressed as equivalent optimization problems in Ising formulas. Existing Ising machines, mostly in prototype and conceptual form, have already shown the promise of remarkably good performance and energy efficiency for optimization problems. However, the real appeal of these systems lies in their future potential. First, with design iterations, their computational power and efficiency will continue to improve. Second, novel hardware will allow the design of algorithms (especially those inspired by nature) to evolve along with the hardware, resulting in a richer mix of problem-solving modalities.

[0004] The Ising model is used to describe the Hamiltonian of spin-coupled systems. Spin has one degree of freedom and can take one of two values ​​(+1, -1). The energy of the system is determined by the spin pair coupling (J ij ) and the interaction of some external field (μ) with each spin (h i ) The resulting Hamiltonian is shown in Equation 1.

number

[0005] A physical system with such a Hamiltonian naturally tends towards low-energy states. It is as if nature were trying to solve an optimization problem with Equation 1 as the objective function, which is far from trivial. In fact, the cardinality of the state space grows exponentially with the number of spins, and the optimization problem is NP-complete, which can be easily transformed into a generalized max-cut problem, which is part of the original list of NP-complete problems.

[0006] Therefore, the physical system of spins somehow requires programmable coupling parameters (J in Eq. 1). ij and μh i ), this can be used as a dedicated computer to solve optimization problems that can be expressed in Ising equations (Equation 1). In fact, all problems in the Karp NP-complete set have their Ising equations derived. Furthermore, if the problem already has a quadratic unconstrained binary optimization (QUBO) equation, then mapping it to an Ising equation is as simple as substituting bits for spins, i.e., σ i =2b i The result is -1.

[0007] Due to the wide variety of problems that can be mapped to the Ising equations, building nature-based computational systems to solve these problems has attracted considerable attention. Roughly speaking, the design of an Ising machine involves four steps:

[0008] 1) Identify a physical variable that represents spin (such as a quantum bit, the phase of a light pulse, or the polarity of a voltage on a capacitor).

[0009] 2) Identify the coupling mechanism and how to program the coefficients.

[0010] 3) Demonstrate problem-solving capabilities both in terms of their theory of operation (revealing nature's "invisible hand") and in terms of satisfactory practical results.

[0011] 4) Demonstrate superior machine metrics (time to solution, energy consumption, and build cost).

[0012] Importantly, different approaches may offer different fundamental tradeoffs and may undergo different gestation rates, so it is premature to evaluate general approaches based on observed prototypic cases.

[0013] Some of the earliest and perhaps best known Ising machines are the quantum annealers sold by D-Wave. Quantum annealing (QA) differs from adiabatic quantum computing (AQC) in that it relaxes the adiabaticity requirement. While QA technically includes AQC as a subset, the current D-Wave system is not adiabatic, so there is no guarantee that the ground state will theoretically be reached. Without the guarantee of a ground state, the Ising physics of qubits has no other known advantages over alternatives. It can be argued that using quantum devices to represent spin is probably suboptimal. First, these devices are much more sensitive to noise and therefore require cryogenic operating conditions that consume a lot of power (25KW for the D-Wave 2000q). Second, coupling large numbers of qubits can be difficult compared to other spins, which explains why current machines use local coupling networks. As a result, for common graph topologies, the number of nodes required for these locally coupled machines grows quadratically, with the nominal 2000 nodes on a D-Wave 2000q corresponding to only about 64 effective nodes.

[0014] Coherent Ising Machines (CIMs) can be considered as a second-generation design that addresses several issues. In T. Inagaki et al. (Science, vol. 354, no. 6312, pp. 603-606, 2016), all 2000 nodes can be connected to each other, and it is believed to be the most powerful Ising machine to date. Because CIMs use special light pulses as spins, they can operate at room temperature and consume only about 200 W of power. However, the pulses must be contained in a 1 km long optical fiber, and the system requires strict temperature stability, making it difficult to maintain a stable operating state for many spins.

[0015] The working principle of the CIM can be observed by the Kuramoto model, so theoretically, similar goals can be achieved using other oscillators. This has led to the development of a number of electronic oscillator-based Ising machines (OIMs), which can be considered the third generation. These systems use LC tanks for spin and (programmable) resistors as coupling units. These electronic oscillator-based Ising machines are a significant improvement over previous designs in terms of mechanical metrics. Indeed, their exact power consumption and operating speed can range over orders of magnitude, depending on the exact inductances and capacitances chosen. However, it is not difficult to achieve desktop sizes with power consumption of around 1-10 W, which is a significant improvement over cabinet-sized machines with power consumption of 200 W-25 kW. However, in on-chip integration, inductors often cause practical challenges. They are area intensive and have undesirable parasitics such as reduced quality factor and increased phase noise, all of which cause practical challenges in maintaining frequency uniformity and phase synchrony among thousands of on-chip oscillators.

[0016] Another electronic design with a different architecture is the Bistable Resistively-coupled Ising Machine (BRIM). In BRIM, spin is implemented as a capacitor voltage controlled by a feedback circuit, making it bistable. This design is CMOS compatible and uses voltage (rather than phase) to represent spin, allowing for easy interfacing to additional architectural support for computational tasks. Thus, the system disclosed herein uses a similar baseline substrate as BRIM. It should be noted that the same principles described here can be directly applied to all Ising machines with different amounts of glue logic. More detailed information regarding BRIM systems can be found in International Application No. PCT / US2021 / 070402, filed April 16, 2021, the entire contents of which are incorporated herein by reference.

[0017] The concept of energy is used in classical optimization algorithms as well as in many machine learning algorithms collectively known as energy-based models (EBM). A system typically consists of two sets of variables X and Y (in a concrete example, X represents the pixels of an image and Y represents a Boolean variable that classifies the image). If the energy of a state, E(X,Y), is low, then the classification is good. In many models, the energy is similar to the Ising equation. For example, in the well-known model of the Boltzmann machine, the distinction between the two sets of variables is ignored and each variable is denoted by σ i where the energy corresponds to the Ising model.

number

[0018] When using a Boltzmann machine for inference, the system is similar to using an Ising machine, but with an important difference. In both cases, the weights (W ij ) is input to the system, and the output is a low-energy state (σ i ) The difference comes from the meaning of the variables / spins. In a Boltzmann machine, the spins contain two sets of variables called visible and hidden units. During inference, the visible units are "clamped" to the input (e.g., an image) and only the hidden units are allowed to change in the search for low-energy states.

[0019] Unlike optimization problems where the weights are part of the problem formulation, EBM requires training to obtain an optimal set of weights. As with many machine learning algorithms, this is done by using gradient descent, which lowers the loss function while iterating over a set of training examples. An important point to highlight here is that the main challenge in such gradient descent algorithms is that they often involve terms that are difficult to compute, requiring approximation algorithms. Again, natural computational substrates allow for convenient or efficient approaches for the substrate without the need to faithfully follow the general von Neumann algorithm.

[0020] In this disclosure, a physical Ising machine is presented that helps accelerate EBMs in many different ways, both in training and inference. For this purpose, we have chosen a special case of Boltzmann machines called Restricted Boltzmann Machines (RBMs), a widely used algorithm highly optimized for von Neumann architectures. RBMs (and their multi-layer variants) have applications in expert learning and unsupervised learning. An exemplary RBM is shown in Figure 1.

[0021] The RBM only has connections between visible nodes 101 and hidden nodes 102, but no connections between two visible nodes or two hidden nodes, as shown in Figure 1. As a result, the energy function is shown in Equation 3.

number

number

number

[0022] Like other neural networks, RBMs can be stacked in multi-layer configurations to form deep networks. Specifically, two common variants are deep belief networks (DBNs) and deep Boltzmann machines (DBMs). There are subtle differences between these variants and simpler RBMs. For simplicity, this disclosure focuses on RBMs and follows the traditional approach when stacking multiple layers. Summary of the Invention

[0023] In one aspect, a bistable resistive coupling system includes a plurality of visible nodes, a plurality of hidden nodes, and a plurality of coupling elements electrically connected to a visible node of the plurality of visible nodes and a hidden node of the plurality of hidden nodes, each of the plurality of coupling elements including a programmable resistor. In one embodiment, each of the plurality of coupling elements includes two programmable resistors. In one embodiment, each programmable resistor includes a field effect transistor having a source, a gate, and a drain, with a gate capacitor connected between the source and the gate. In one embodiment, each of the plurality of coupling elements includes an analog counter having an overflow signal and an underflow signal, the overflow signal configured to increase a value of the programmable resistor, and the underflow signal configured to decrease a value of the programmable resistor.

[0024] In one embodiment, at least one of the plurality of visible nodes or the plurality of hidden nodes includes a sigmoid element including an inverter having an input, an output, and a load resistor connected between the output and a common mode reference. In one embodiment, at least one of the plurality of visible nodes or the plurality of hidden nodes includes a random noise generator including a binary random number generator having an output and a low pass filter connected to the output.

[0025] In one embodiment, the system further includes a comparator having a first input and a second input, the first input being connected to an output of the sigmoid element and the second input being connected to a filtered output of the binary random number generator. In one embodiment, at least one of the plurality of visible nodes or the plurality of hidden nodes includes a capacitor and a feedback unit connected across the capacitor configured to make a voltage across the capacitor bistable. In one embodiment, at least one of the plurality of visible nodes or the plurality of hidden nodes includes a buffer.

[0026] In one aspect, a coupling device for connecting first and second nodes in a network includes an inverting input and a non-inverting input, first and second field effect transistors having a drain, a gate, and a source, respectively, where the drain of the first field effect transistor is connected to the non-inverting input and the drain of the second field effect transistor is connected to the inverting input, first and second gate capacitors connected between the gates and sources of the first and second field effect transistors, respectively, a summing output connected to the sources of the first and second field effect transistors, and a voltage adjusting element connected to the gates of the first and second field effect transistors, the voltage adjusting element configured to adjust the gate voltages of the first and second field effect transistors in response to a control signal.

[0027] In one embodiment, the voltage regulating element includes an analog counter. In one embodiment, the device further includes at least one current source switchably connected to the gate of the first or second field effect transistor. In one embodiment, the device further includes four current sources, one current source switchably connected to each gate of the first and second field effect transistors and connected to a positive voltage or ground. In one embodiment, the voltage regulating element further includes overflow and underflow outputs of the analog counter configured to increase or decrease the amount of charge on the first and second gate capacitors. In one embodiment, the first and second field effect transistors are N-channel field effect transistors.

[0028] In one aspect, a method for training a bistable resistive coupling system includes initializing a set of weighting elements and a set of biasing elements in the bistable resistive coupling system, initializing a set of visible nodes of the bistable resistive coupling system to a first set of initial values, clamping the set of visible nodes to the first set of initial values ​​for a period of time and allowing the set of hidden nodes to settle to the first set of hidden values, increasing a counter of at least one weighting element based on a product of the first set of initial values ​​and the first set of hidden values, initializing a set of hidden nodes of the bistable resistive coupling system to a set of random values ​​selected from a table of hidden values, annealing the visible nodes and hidden nodes for a second period of time, decreasing the counter of the at least one weighting element based on the annealed values ​​of the visible nodes and hidden nodes, increasing or decreasing a weighting value of the at least one weighting element if the counter of the at least one weighting element overflows or underflows, and repeating a programmable number of learning steps from the step of initializing the set of visible nodes.

[0029] In one embodiment, the set of values ​​used to initialize the set of hidden nodes is obtained from a corresponding set of hidden values ​​from a previous annealing step. In one embodiment, the method further includes reading the combined values ​​from the system using at least one analog-to-digital converter. In one embodiment, the fixed period of time is in the range of 1 nanosecond or less. In one embodiment, the second fixed period of time is in the range of 1 nanosecond or less. In one embodiment, the method further includes storing the annealed values ​​of the hidden nodes in a table of annealed hidden values.

[0030] The above and other objects and features will become apparent from the following description and by reference to the accompanying drawings, which are included to provide an understanding of the invention and are incorporated in and constitute a part of this specification, and in which like numerals represent like elements. [Brief description of the drawings]

[0031] [Figure 1] This is a restricted Boltzmann machine. [Diagram 2] 1 is an exemplary high-level BRIM implementation showing a bistable capacitive node with programmable resistive coupling and programming logic. [Diagram 3] FIG. 1 is a diagram of a high-level RBM implementation showing visible and hidden nodes, with clamping units that drive node biases, a connection mesh, and programming logic. [Figure 4] Pseudocode contrastive divergence algorithm for training RBM. [Diagram 5] 1 is a graph showing the distribution of energy states of BRIM. [Figure 6] Architectural diagram of a complex design in which a machine follows a gradient. [Figure 7] FIG. 2 is a high-level block diagram of a node. [Figure 8] FIG. 2 is a detailed diagram of certain circuit elements of an exemplary node. [Figure 9] 1 is an exemplary binding unit. [Figure 10] 13 is a graph showing the speedup of the Boltzmann sampler over the TPU and Gibbs sampler for training different RBMs when the batch size is 500 images. [Figure 11] 13 is a graph showing the energy consumption of the TPU and Gibbs sampler for various benchmarks normalized with the Boltzmann sampler for a batch size of 500 images. [Figure 12] 1 is a graph showing the average log probability of the conventional algorithms (CD-1 and CD-10) and the disclosed modified algorithm used for the Boltzmann Gradient Follower (BGF). [Figure 13A] 1 is a graph showing the average log probability of the conventional algorithms (CD-1 and CD-10) and the disclosed modified algorithm used for the Boltzmann Gradient Follower (BGF). [Figure 13B]1 is a graph showing the average log probability of the conventional algorithms (CD-1 and CD-10) and the disclosed modified algorithm used for the Boltzmann Gradient Follower (BGF). [Figure 13C] 1 is a graph showing the average log probability of the conventional algorithms (CD-1 and CD-10) and the disclosed modified algorithm used for the Boltzmann Gradient Follower (BGF). [Figure 13D] 1 is a graph showing the average log probability of the conventional algorithms (CD-1 and CD-10) and the disclosed modified algorithm used for the Boltzmann Gradient Follower (BGF). [Figure 14A] 13 is a graph showing the moving average of the mean log probability for different models with varying amounts of injected noise and fluctuations. The data is smoothed using a 10-point moving average. [Figure 14B] 13 is a graph showing the moving average of the mean log probability for different models with varying amounts of injected noise and fluctuations. The data is smoothed using a 10-point moving average. [Figure 14C] 13 is a graph showing the moving average of the mean log probability for different models with varying amounts of injected noise and fluctuations. The data is smoothed using a 10-point moving average. [Figure 14D] 13 is a graph showing the moving average of the mean log probability for different models with varying amounts of injected noise and fluctuations. The data is smoothed using a 10-point moving average. [Figure 15A] 13 is a graph showing the mean absolute error (MAE) of different models as the amount of injected noise and variation is varied. The final MAE ranges from 0.771 to 0.765. [Figure 15B] 13 is a graph showing the Roc curves of different models with varying amounts of injected noise and variance, with final AUC range of 0.964 - 0.967. [Figure 16] We show the cumulative probability distribution of the KL divergence against the ground truth for the training results of CD and BGF. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] It should be understood that the drawings and description of the present invention have been simplified to illustrate elements relevant to a clear understanding of the present invention, but for clarity, many other elements found in related systems and methods have been omitted. Those skilled in the art may recognize that other elements and / or steps are desirable and / or necessary in implementing the present invention. However, because such elements and steps are well known in the art and do not facilitate a better understanding of the present invention, descriptions of such elements and steps are not provided herein. The disclosure herein covers all variations and modifications to such elements and methods known to those skilled in the art.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, exemplary methods and materials are described below.

[0034] As used herein, each of the following terms has the meaning associated with it in this section.

[0035] As used herein, the articles "a" and "an" refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, "an element" means one or more elements.

[0036] As used herein, "about" when referring to a measurable value, such as an amount, a temporal duration, and the like, is meant to encompass variations of ±20%, ±10%, ±5%, ±1%, and ±0.1% from the particular value, as such variations are appropriate.

[0037] Throughout this disclosure, various aspects of the invention can be expressed in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Thus, the description of a range should be considered to have specifically disclosed all possible narrower ranges and individual numerical values ​​within that range. For example, the description of a range such as 1-6 should be considered to have specifically disclosed the narrower ranges such as 1-3, 1-4, 1-5, 2-4, 2-6, 3-6, etc., as well as the individual numerical values ​​contained within that range, such as 1, 2, 2.7, 3, 4, 5, 5.3, 6, and all whole and partial increments therebetween. This applies regardless of the breadth of the range.

[0038] In one aspect, disclosed herein is an Ising machine substrate that provides underlying additional architectural support. The present disclosure provides a new class of extended Ising machines that are suitable for both training and inference in machine learning applications.

[0039] Some Ising machines have already shown better performance and energy efficiency in optimization problems. Through design iteration and co-evolution between hardware and algorithms, more benefits are expected from nature-based computational systems. One embodiment of the disclosed device is an extended Ising machine suitable for both training and inference with energy-based machine learning algorithms. In one embodiment, the Ising substrate accelerates key parts of the algorithm, achieving significant speedups and efficiency gains. In some embodiments, more drastic modifications make the machine a self-sufficient gradient follower that essentially completes training in hardware. This can achieve approximately 29x speedup and approximately 1000x reduction in energy consumption compared to the TPU host.

[0040] In one embodiment of the present disclosure, the training step of the energy-based machine learning system is implemented in an integrated circuit by using programmable resistive elements in each coupling unit connecting the nodes of the machine, where the resistance value of the resistive elements is stored, controlled and updated during the training process by utilizing electronic circuitry located within the boundaries of a chip area substantially dedicated to a single coupling unit. This saves energy, reduces the overall chip area and increases the training speed. In another embodiment of the present invention, a random number generator with a single-bit output low-pass filtered by a continuous-time analog filter is used to generate random analog values ​​over a predetermined range, realizing random sampling of the probability distribution of the node voltages. In another embodiment of the present invention, the trajectories of a physical Ising machine are used to generate nature random samples that follow a (quasi-)Boltzmann distribution. This allows the entire Markov chain Monte Carlo algorithm to be efficiently embedded in the control circuitry of the coupling unit.

[0041] Ising machine substrate As disclosed herein, some physical substrates can exploit nature to perform optimization of the Ising equation. In principle, any such substrate can be used to accelerate machine learning algorithms. One such system using an integrated electronic design, referred to herein as BRIM, is disclosed herein. Figure 2 shows a high-level view of the BRIM architecture.

[0042] The architecture shown in FIG. 2 shows, for example, bistable capacitive nodes 201, 204 with programmable resistive coupling 202 and associated programming logic 203. Between each pair of nodes (e.g., nodes 201 and 204), one bidirectional coupling unit 202 is shown, resulting in an upper triangular coupling network. In some implementations, the coupling units may be composed of two unidirectional parts, forming a symmetric network. In some embodiments, the programming logic may include, for example, one or more memory elements 206 configured to store values ​​provided to the coupling elements. In some embodiments, multiple coupling units, for example an entire column of coupling units, may be connected to the same programming configuration circuitry via a multiplexer 207 configured to select one or more of the coupling units to program in a given cycle. In some embodiments, the programming logic 203 may include one or more digital-to-analog converters, for example for reading analog values ​​stored as bit strings in the memory 206 for use in the configuration of the coupling units 202.

[0043] In the illustrated machine, each node 201, 204 includes a capacitor and a feedback unit that makes the capacitor bistable (1 or -1). In some embodiments, the node capacitors in visible or hidden nodes can have capacitances in the range of 10-100 femtofarads (fF), or 1-10 fF, or 1-50 fF, or about 50 fF. A mesh of all-to-all programmable resistors (e.g., 202) helps to represent the Ising equation that the system is trying to optimize. When treated as a dynamic system, Lyapunov analysis can be applied to the differential equations governing the node voltages. It can be shown that all local minima in the Ising energy landscape are stable states of the system. In simpler terms, starting from a given initial state, this system of coupled nodes searches for a local minimum without explicit guidance from a controller. In some embodiments, an additional annealing control is used to avoid local minima by injecting random "spin flips". This is similar to accepting a state with low energy with nonzero probability during simulated annealing.

[0044] In an RBM, nodes are separated into a bipartite graph. For this special graph, the architecture for the Ising machine substrate can be slightly modified to have nodes at two edges of the connection network. Referring to FIG. 3, in this structure, visible nodes (e.g., 301) can only be connected to hidden nodes (e.g., 302). This layout significantly improves space efficiency compared to allowing connections between all nodes. As a specific example, one disclosed benchmark uses 784 visible nodes (28×28 pixels) and 500 hidden nodes. Mapping them to a general all-to-all Ising substrate requires four times as many connection units ((784+500) 2 vs784×500) is required.

[0045] Finally, the nodes of an Ising machine can be extended to support the operation of the RBM algorithm. In both training and inference, it is common to clamp visible or hidden nodes to a particular value. In some embodiments, the apparatus may further include a clamp unit 303 configured to hold the values ​​of hidden nodes (e.g., 302) and / or visible nodes (e.g., 301) during a particular phase of the process as detailed herein. In one embodiment, the clamp unit 303 can be implemented with one or more digital-to-analog converters (DACs) connected to the nodal capacitors of the node(s) whose analog voltage output(s) are to be clamped. In one embodiment, the clamp unit has a set of 1-bit DACs that clamp the capacitor voltages to either a ground voltage or a power supply voltage. One important detail is that the inputs to the visible nodes are multi-bit values ​​in some embodiments. In such implementations, multi-bit digital-to-analog and / or analog-to-digital converters may be required. In a baseline Ising machine, the number of required combination units is N 2 Since most of the area is proportional to (N is the number of nodes), the area is allocated to the coupling unit, so that changing the structure of individual nodes has little impact on the system complexity and chip area. In some embodiments, the system may include one or more additional combining units that hold bias values, such as combining units 304a and 304b in FIG. 3. For example, in Equation 4 below:

number

number

[0046] Gibbs Sampler Architecture One variant of the RBM accelerator is more conventional and simply leverages the Ising substrate to accelerate portions of software algorithms that are naturally suited to hardware. For convenience, such accelerator designs are referred to herein as Gibbs samplers, since they follow the conventional Gibbs sampling-based algorithm shown in FIG.

[0047] As shown in the algorithm in Figure 4, the training loop (lines 7 to 18) pos , h pos and v neg , h neg In a nutshell, the algorithm is stochastic gradient descent, and the training loop iteratively iterates over all the weights W ij In the so-called positive phase (lines 8 and 9), we calculate the stochastic gradient for the training sample v pos is clamped to the visible nodes, and the hidden nodes h pos The corresponding samples of are generated based on the conditional probability formula:

number

number

[0048] In the negative phase (lines 12 and 13 in Fig. 4), k-steps of Gibbs sampling are performed. While preserving the values ​​of the current hidden nodes, a new set of visible nodes is generated with probability, where the probability is shown in Equation 5.

number

[0049] From there, the updated visible nodes are projected back to generate updated hidden nodes, forming one complete step of a Markov Chain Monte Carlo (MCMC) algorithm. In principle, performing such a step in an MCMC algorithm would result in rather poor sampling. In practice, a small number of k steps are chosen to balance cost and sampling quality. Such k-step contrastive divergence algorithms are often referred to as CD-k.

[0050] At each learning step, the current weight matrix [W ij ] M×N is each unit R ij The coupling array is programmed so that the resistance at is proportional to the following formula:

number

[0051] This step is similar to programming an optimization formula in a standalone Ising machine. If a set of nodes (say visible) is further clamped to a fixed value, each coupling unit generates a current equal to the voltage of the visible node divided by the resistance of the programmable coupling unit, which is equivalent to multiplying the corresponding weight in the matrix. Thus, each hidden node sees the sum of the currents across the column. Used in this way, the coupling array effectively generates a vector-matrix multiplication operation.

[0052] At this stage, rather than reading out the resulting current, we pass it through a nonlinear circuit that produces the effect of a logistic function. In fact, if properly constructed, a simple inverter can provide a good approximation of this function. Finally, the output of the logistic function is the probability that the node is 1. This can also be assisted by a relatively simple circuit: a comparator with the other input fed by a pseudorandom voltage level. A high-level block diagram is shown in Figure 8, and the details of the circuit implementation are described below.

[0053] In some embodiments, the described architectural support offloads much of the training loop of the algorithm of Figure 4 to hardware. The remaining operations may be performed on a digital functional unit such as a TPU in some embodiments, while in other embodiments, the hardware implementation disclosed herein may perform all of the functions of the algorithm of Figure 4. One exemplary sequence of operations is as follows:

[0054] 1) Initialize the TPU.

[0055] 2) Programming the random coupling matrix and bias to the Ising substrate.

[0056] 3) We divide the visible units into training samples (v pos ) to clamp.

[0057] 4) After the operation of the Ising board is completed, the hidden unit (h pos ) is read.

[0058] 5) Perform k steps of Gibbs sampling by alternating clamping of hidden and visible units to generate opposite-side samples.

[0059] 6) Visible Units (v neg ) and hidden units (h neg ) to read the final value.

[0060] 7) (On the TPU) Compute the new connection matrix and biases.

[0061] 8) Repeat from step 2 for subsequent learning steps.

[0062] Boltzmann Gradient Follower Architecture The systems mentioned above are improvements over digital units. However, this is mainly due to the efficiency gains of the approximate analog implementation. The advantages of nature-based computational systems are often much greater when the entire algorithm can leverage some natural process. This requires a deeper understanding of the intent of the algorithm.

[0063] In RBM, the goal is to capture the training data with a probability distribution model, where the probability is exponentially related to the energy of a state, such as in the Boltzmann distribution (hence the name).

number

[0064] Obviously, these probabilities need to sum to one, so the formula for the probability of a particular state (v, h) is:

number

[0065] "Capturing" the training data means that the machine model (weights and biases) maximizes the probability of all T training samples. In other words,

number

number

number

number

[0066] For clarity of notation, we use a single training sample (u = v (t) ) contributions are used and the focus is on the first part of the molecule.

number

[0067] where < > data The notation refers to the expectation over the data, i.e., holding the data constant (u) and averaging over all possible h.

[0068] Following the same steps, the second part of the gradient

number

number

[0069] where < > model The notation denotes the expectation over the entire state space given by the current model (joint parameters and biases).

[0070] As shown, to compute the gradient of any parameter, one would need to compute the expectation of a large number of states, which would be impractical. A common solution is the MCMC algorithm (e.g., CD-k), which is quite similar to simulated annealing in solving Ising formulation problems. In fact, in both cases, the Markov chain is time-inhomogeneous.

[0071] An Ising machine substrate can be used to think of it as a special Markov chain, which in effect performs a kind of sampling of the state space. Figure 5 shows the cumulative distribution of the energies of the states visited by the disclosed Ising substrate, and a fit curve of the Boltzmann distribution to the same set of energies. This agreement suggests that this Ising substrate can be thought of as a Boltzmann sampler. Using such a sampler, we generate negative-phase samples (lines 11-14 of the algorithm in Figure 4) to obtain the model expectation <v neg h neg > can be calculated.

[0072] The Ising substrate, once initialized to some initial condition, proceeds to traverse an energy landscape dictated by both the system's governing differential equations and the annealing control. This has the effect of "sampling" the state space, arguably producing better samples than the random walk of CD-k's algorithm. However, the samples are produced much faster than a host computer can typically access and post-process to obtain the expected value. Thus, disclosed herein is a more direct approach, where the sampled expected value ( <v i h j > data or <v i h j > model ) is a model parameter inside the Ising substrate (e.g., W ij ) is added or subtracted directly from

[0073] In the CD-k algorithm, accumulation is expressed as follows, where <·> s denotes the expectation for the set of samples s.

number

[0074] Here, the expectation value is accumulated over mini-batches of samples (e.g., n=100) before being used to update the parameters to the next value. The choice of n is usually a matter of convenience and some trial and error. For implementation convenience, samples are accumulated in different mini-batch configurations, and a purely digital counter would be significantly larger and consume more power than the disclosed combination unit. Fortunately, such a counter can be made using analog circuitry that takes much less area and energy, at the expense of noise-induced errors. In some embodiments, an analog up-down counter is used. Any increment or decrement acts on the counter, and only if the counter overflows or underflows, the counter is decremented by charging or discharging the appropriate capacitor. ij Actually adjust the following.

[0075] In summary, the net result of one embodiment of the disclosed design is that instead of using a fixed mini-batch size, the disclosed machine effectively uses a variable mini-batch size. The mini-batch is data dependent and therefore parameter-specific. Furthermore, the circuit implementation of the counter adds effective noise in addition to the noise due to stochastic sampling of the gradient. All non-idealities are faithfully modeled during evaluation of the system. Now, a fine grained indication can be made as to whether the disclosed learning algorithm produces biased estimators or not. Empirical analysis indicates that the estimation bias does not appear to affect the final accuracy measurement. In fact, in some embodiments, the disclosed modifications appear to reduce bias from commonly used algorithms. This is explained in more detail in Experimental Example #2 below.

[0076] For negative-phase samples, we use hidden units (h (k) , k=1..p), p different initial conditions are used. These are often called p-particles. After each positive phase, one of the particles (e.g., h (3)) is loaded and an Ising machine annealing is performed (corresponding to the random walk of the Von Neumann algorithm). In some embodiments, the annealing time is less than 5 ns, less than 4 ns, less than 3 ns, less than 2 ns, less than 1 ns, less than 500 ps, ​​about 1 ns, or any other suitable range. Then, v neg h neg A sample of is taken and the resulting hidden unit value is applied to the loaded p particles (in this case h (3) ) position. In some embodiments, the result may be stored in a different position, e.g., in a different p-particle position or in a different position than the existing p-particles.

[0077] The parameters are physically represented by the conductance of a resistor that can be set. The resistors are implemented by transistors with variable gate / source voltages. Increasing or decreasing the parameters can be achieved by raising or lowering the gate voltage. This can be achieved by briefly turning on a charge / discharge circuit connected to the effective gate capacitor. This turns out to be not as easy as one might think, as there are multiple nonlinearity issues in the circuit elements. The result is that the gradient descent is slower when the values ​​are close to zero. Although it does not affect the overall effectiveness of the disclosed machine, in some embodiments a slightly related version is used (more on this below) in which this issue is significantly reduced.

[0078] Finally, since now the entire learning takes place within the (augmented) Ising substrate, the combination units are larger and more complex. Moreover, the trained results need to be read out at the end of the learning process, which requires additional analog-to-digital converters (ADCs) with high cost and surface area. Nevertheless, they are only used once at the end of the algorithm.

[0079] In summary, this architecture requires some significant new circuitry and minor modifications to the operating algorithms, but it executes the intent of a traditional software implementation, and as demonstrated herein, the resulting quality is comparable to the software implementation, even when significant noise is taken into account.

[0080] Figure 6 shows one embodiment of the disclosed Boltzmann gradient follower architecture. The key addition is a coupling unit (e.g. 601) where the programmable resistors 602 acting as weights can be adjusted to the appropriate positions. This is achieved by controlled charging and discharging pulses that adjust the gate voltages of the transistors acting as programmable resistors. In this architecture, the digital (host) computer mainly takes on a more peripheral role, such as setting up the system, providing data at a fixed frequency, and reading the final results. The operation can be described as follows:

[0081] 1) Initialize the weights and biases. (In some embodiments, the weights are initialized to small random values. This can certainly be implemented by the hardware itself. However, programmable initial conditions may be useful for special purposes (e.g., research).

[0082] 2) The host then sends the training samples to the visibility unit's latch.

[0083] 3) The machine clamps the data and waits a given time (e.g., 1 ns or less) for the hidden units to settle.

[0084] 4) The obtained sample (v pos h pos ) is W ij It is used to increment the counter of v pos and pos If both are 1, the counter W ij is incremented by a constant value.

[0085] 5) The machine then loads one of the p particles and begins the annealing process, for example, less than 1 ns.

[0086] 6) After annealing, the resulting sample (v neg h neg ) is W ij It is used to decrement an analog counter of v neg and neg If both are 1, the counter W ij is decremented by a constant value. Thus, an increment or decrement of the counter occurs only if the product of v and h is equal to 1, otherwise the counter is not incremented or decremented. The increment or decrement depends on the learning rate.

[0087] 7) When any counter overflows or underflows, the charge / discharge circuitry is activated and the W ij is incremented or decremented by a fixed amount and the (analog) counter is reset.

[0088] 8) The process of steps (2-7) is repeated for a programmable number of learning steps. Then the ADC reads out the combined voltages column by column.

[0089] circuit In addition to the circuitry required to implement the baseline Ising substrate, additional circuitry is required at the nodes to make them stochastic according to the RBM algorithm, summarized in Figure 7. Specifically, a current summation circuit, a sigmoid function, and a random number generator. Finally, a coupling unit is required for the Boltzmann gradient follower architecture. All of these are discussed in more detail below.

[0090] The current summing circuit 701 runs on two non-overlapping clock phases (φ and φ′). During the reset phase (φ′), the hidden node capacitor 711 and the visible node are connected to the hidden node h j Both the column buses that connect to V CM =V ddDuring the subsequent integration phase (φ), the capacitor 711 is connected to the column bus and precharged for a fixed time interval t int (e.g., <5ns, <4ns, <3ns, <2ns, <1ns, <500ps, etc.) and integrate all currents

number

[0091] In general, sigmoid functions are monotonic and have a bell-shaped first derivative. The simplest circuit that exhibits similar properties is an inverter, as shown in FIG. 8 for a detailed view of a sigmoid unit 702. Before using an inverter as a sigmoid unit, several issues must be addressed. First, while the transfer function of a typical inverter used as a logic gate consists of three distinct regions (i.e., two triode regions close to the power rails and a steep region in the middle due to the high gain near the inverter threshold), a sigmoid function typically has a smoother transfer function. Increasing the transistor channel length and reducing the gain that can be achieved by adding a load resistor 801 at the output can bring the inverter's transfer function closer to a sigmoid function. In various embodiments, the load resistor 801 may have a resistance in a range of, for example, 3 kΩ to 30 GΩ, or 3 kΩ to 100 kΩ, or 3 kΩ to 1 μΩ, or 1 μΩ to 100 μΩ, or 1 μΩ to 1 GΩ, or 1 GΩ to 30 GΩ, or any other suitable range.

[0092] The second problem is that the transfer function of the inverter is a vertically inverted image of a typical sigmoid function, which is mitigated by introducing an additional inversion at a later stage. As an example, this effect can be mitigated by using an inverting comparator, as shown in element 802 of FIG.

[0093] Thermal noise from electronic devices can be used to generate randomness. The circuit shown in element 803 of FIG. 8 is a type of random number generator (RNG) whose operation is as follows: When φ is low, node A (831) and node B (832) are at a metastable point V dd As φ becomes higher, the circuit enters the evaluation phase, where both nodes discharge towards the inverter switching point, and the large gain at this point pulls one of nodes A or B to V. dd The RNG circuit itself generates a binary random sequence (i.e., the output is V dd Since the RNG 803 generates a binary random sequence (either V or 0), it is not directly suitable for achieving stochastic sampling at the output of the sigmoid function. However, the binary random sequence at the output of the RNG 803 can be filtered from 0 to V by applying an RC low-pass filter 804, as shown in FIG. dd The resulting white noise is then compared to the output of a sigmoid function in a standard dynamic comparator 802 to achieve stochastic sampling. In various embodiments, the low pass filter 804 can have a passband of less than 5 GHz, less than 2 GHz, less than 1 GHz, less than 500 MHz, less than 100 MHz, or any other suitable passband.

[0094] Referring to Figure 9, two programmable resistors (

number

number

number

number

number

number

number

number

number

[0095] This is achieved by turning on only one pair of diagonal current sources (e.g., 901a and 901d). During the initialization phase, the digital-to-time converter (DTC) in the programming logic (see Fig. 3) initializes the clock by charging the gate capacitors:

number

number

number

number

number

number

[0096] Experimental Example The present invention will be described in more detail with reference to the following experimental examples.These examples are provided for illustrative purposes only and are not intended to be limiting unless otherwise specified.Therefore, the present invention should not be construed as being limited to the following examples in any way, but should be construed as embracing any and all variations that become evident as a result of the teachings provided herein.

[0097] Without further elaboration, it is believed that one of ordinary skill in the art can, using the preceding description and the following examples, make and use the systems and methods of the present invention. Accordingly, the following examples specifically illustrate exemplary embodiments of the present invention, and are not to be construed as limiting in any way the remainder of the disclosure.

[0098] To better understand the characteristics of the disclosed extended Ising substrate for EBM, we compared various metrics between the disclosed design and our implementation using the TPU. It is worth noting that an evolutionary perspective is necessary when looking at the results. Both digital-numerical architectures and nature-based computational systems evolve. Design improvements can induce significant changes in those metrics. More importantly, the boundaries between the two groups continue to blur, and mutual influence is a very intended outcome.

[0099] In the following experimental example, we describe the experimental setup, including benchmark RBM and DBN models, then provide a quantitative comparison between the TPU and a noiseless model of the disclosed analog design in terms of energy efficiency and throughput, and finally present some detailed analyses that help understand the behavior of the system.

[0100] Experiment #1 - Experimental Setup To evaluate the considered system, the system was trained on various applications such as image classification, recommender system, and anomaly detection. For image classification, various datasets were used including handwritten images (MNIST), Japanese characters (KMNIST), fashion images (FMNIST), extended handwritten alphabet images (EMNIST), color images (CIFAR10), and a toy dataset (SmallNorb). The images range from 28x28 grayscale (NIST) to 96x96 grayscale (SmallNorb), and 32x32 color (CIFAR10). RBM and DBN algorithms were used to train all NIST datasets, and convolutional RBM algorithm was used for CIFAR10 and SmallNorb datasets. The 100k MovieLens dataset and the "European Credit Card Fraud Detection" dataset were used to train the RBM as a recommender system and anomaly detector, respectively. The learning rate used to train these models was 0.1, and the sizes of the RBM and DBN configurations are shown in Table 1. Since RBMs are unsupervised models, one way to quantify the quality of training is the average log probability of the training samples, which can be measured using annealed importance sampling. Several common metrics have also been reported, such as classification accuracy using a logistic regression layer at the end for image classification, mean absolute error (MAE) on test data and projection data for recommender systems, and area under the receiver operating characteristic (ROC) curve for anomaly detection. [Table 1]

[0101] Modeling the disclosed design is relatively straightforward. The system is assumed to have enough nodes to fit the largest problem in the set. The execution time is therefore the product of the number of iterations and the number of cycles per iteration. Anything not run on the disclosed hardware system is run on a host machine, which is assumed to contain the same TPU as the baseline.

[0102] A first-order analysis of the disclosed system is desired because many variables are involved and it would be deceptive to pretend that all parameters have been accurately estimated. Roughly speaking, it is believed that the reported results should be accurate to within an order of magnitude. In other words, it is highly unlikely that an actual implementation following the exact design would produce results that are more than about 3 times better or more than 3 times worse. Not all parameters have the same width of confidence region. Execution time, for example, has tight constraints since it is simply the result of repeated iterations, and circuits can be reliably designed to meet relatively modest cycle times. Area and power (especially elements not customized for the disclosed design, such as DTC) have a reasonably large design space, so the numbers used are only best estimates of the metrics that could have been chosen with the current off-the-shelf products. The size of the transistors has been determined to keep noise at a reasonable level. The biggest concern in such a study is whether the proposed system will definitely work. The biggest unknown is the collective effect of noise, for which some sensitivity analysis is provided herein.

[0103] Execution speed First, we compare the execution speed of the two proposed architectures, the Gibbs sampler and the Boltzmann gradient follower (BGF). The operating frequency of the Gibbs sampler and the BGF is 1 GHz. For comparison, we use the TPU (v1) as a baseline. The TPU was modeled to execute 8-bit fixed-point operations at a clock frequency of 700 MHz. The same frequency was also assumed for the disclosed digital control.

[0104] Figure 10 shows a graph of the speedup for different benchmarks. As can be seen, for larger networks, most of the hardware is utilized and the speed advantage increases. Of course, if the problem is larger than can be mapped on a single chip, a multi-chip solution is required or part of the computation needs to be performed elsewhere, such as on the host. The data sets used in these examples all fit comfortably inside a small die. Therefore, this example focuses only on a single-chip analysis.

[0105] Overall, to a first approximation, adding a Gibbs sampler or a Boltzmann gradient follower to a TPU array increases speed by a factor of 2 (Gibbs) or 29 (BGF) on geometric mean, with only a small increase in area.

[0106] Energy and Area Next, we consider the energy consumption, which has a higher degree of uncertainty. In the following, we use a simplified first-principles analysis as a starting point, before discussing the extrapolation of the results for the whole system.

[0107] The main source of the fundamental efficiency of the Ising substrate is that while many algorithms mimic nature, the disclosed hardware directly embodies that nature. For example, in a typical step of MCMC, flipping one node requires about O(N) multiply-accumulate (MAC) operations followed by some probability sampling. Ignoring probability sampling, each MAC operation costs about pJ. For the problem discussed, N≈1000. Such an inversion would therefore require about nJ of inversions using conventional digital computation. In contrast, in BRIM, flipping a node requires a distributed network of currents that charge and discharge the node capacitors. With a node capacitor of about 50 fF and a voltage of about 1 V, a node flip would cost about 50 fJ. Thus, the Ising substrate can be about four orders of magnitude more efficient than conventional computational substrates.

[0108] As for the whole system, the energy savings depend on multiple factors that are not included in the simple analysis above. To obtain the power / energy estimates, the circuit was designed on Cadence 45nm using the Generic Process Design Kit (GPDK045), which is the latest technology for which a design kit was available for this analysis. The area and power consumed by the TPU unit were taken from X. He, et al., Proceedings of the 34th ACM International Conference on Supercomputing, 2020, pp. 1-12, which employs 28nm technology.

[0109] Taking into account the differences in techniques, we can compare the energy consumption of different benchmarks, as shown in Figure 11. All energy results were normalized to the energy results of the Boltzmann gradient follower. Figure 11 shows the energy consumption of the TPU implementation and the Gibbs sampler for different benchmarks, normalized to the Boltzmann sampler, for a batch size of 500 images.

[0110] The disclosed accelerator is more efficient at performing effective operations than algorithmically defined digital TPU operations. Overall, the disclosed accelerator demonstrates an improvement of approximately 1000x.

[0111] Two qualitative points are worth highlighting: on the one hand, the digital circuitry still has room for improvement, and more efficient designs could further reduce the cost per operation; on the other hand, the types of short random walks performed in the disclosed Boltzmann gradient follower architecture are not the most efficient use of the architecture. With further algorithmic innovation, applications may be found that fully exploit the capabilities of the disclosed and similar nature-based systems.

[0112] Next, we estimated the chip area. Again, this is not a direct comparison given the technology differences. Nevertheless, the baseline 28nm TPU is about 330mm 2 , of which 24% is the MAC array. For comparison, the area and power of the BGF and its building blocks are shown in Table 2. The coupling units are overwhelmingly more numerous (relative to the other units, they are roughly O(N 2 ) vs. O(N), with the area mostly determined by the join unit. Assuming a 1024 × 1024 array, the Gibbs sampler architecture costs 1 mm 2 The Boltzmann gradient follower has an area of ​​about 16 mm 2 ,which occupies about 4.8% of the total area of ​​the,host TPU.,As shown in Table 3, BGF is much more efficient than TPU or,state-ofthe-art computational accelerators in this particular algorithm. [Table 2] [Table 3]

[0113] Impact of algorithm changes According to a first principle analysis, the disclosed Boltzmann gradient follower architecture simply implements a different style of stochastic gradient descent. It should provide similar solution quality to the end user, rather than providing exactly the same trained weights. In the next section, we numerically analyze the changes resulting from the algorithm change. As mentioned before, we use two metrics: the average log probability of the training samples and the classification accuracy. Figure 12 shows the average log probability of the models obtained using the different methods (CD-1, CD-10, and the disclosed modification to the Boltzmann gradient follower) during the training period with different datasets. Note that the log probability is difficult to calculate, so we approximate it with AIS as mentioned before. Therefore, the results should not be read too precisely. In fact, in some datasets (e.g., CIFAR10), the same AIS mechanism completely fails to produce a finite estimate for the partition function. One example dataset (MNIST) is shown in Figure 12, and other datasets are shown in Figures 13A, 13B, 13C, and 13D. Finally, because the raw data has high frequency noise, the data was smoothed using a moving average of 10 data points for better clarity. The same legend as in Figure 12 applies to Figures 13A-13D.

[0114] As shown, in general, the log probability trajectory increases over time, often quite significantly. This means that the trained model better approximates the probability distribution of the training data over time. However, the exact trajectory varies significantly even under common CD-k practice with different k values. The disclosed modified algorithm naturally produces unique trajectories. Compared to CD-10, the difference in trajectories is often less pronounced than the result of choosing a small k for convenience. In addition to the uncertainty of the log probability estimate itself, it has been pointed out that beyond a certain point, the improvement in log probability can simply be the result of overfitting. Thus, the classification error is shown in Table 4. Benchmarks where rounding is applied by the tool show that there is no observable difference between common practice and the disclosed algorithm. In benchmarks where little accuracy is reported, this difference is negligibly small. [Table 4]

[0115] The data presented in Table 4 is the rounding test accuracy obtained by different types of neural network models for each data set using different algorithms.

[0116] Overall, the main takeaway is that the disclosed systems and algorithms include minor modifications of popular algorithms for implementation convenience. From first principles, this is no different from, for example, choosing CD-k over impractical maximum likelihood learning, or choosing just a random k value in CD-k for convenience. Empirical observations suggest that these changes do not actually affect the effectiveness of the learning approach.

[0117] Effect of noise on training Finally, we investigated the impact of noise and variations on the quality of the solutions. The results so far assume that the analog hardware is free of noise and variations. To simulate process variations and circuit noise, we injected static variations in the resistance values ​​of the coupling units and dynamic noise at both the nodes and the coupling units. The noise and variations were generated by a Gaussian distribution with root-mean-square (RMS) values ​​between 3% and 30%. Therefore, different results can be seen for a pair of (RMS variation , R.M.S. noise ) Figures 14A, 14B, 14C, and 14D show the same smoothing results of the average log-probabilities for the 25 combinations for the different models. The data is smoothed using a 10-point moving average.

[0118] When the combination of noise and variance is not too extreme (e.g., <10% each), the impact on the log probability is negligible. In many cases, the loss is smaller than that obtained by modifying the algorithm to fit the disclosed Boltzmann gradient follower. However, even for more extreme variance and noise configurations, the impact on the log probability does not seem to be significant. The final estimation accuracy remains unchanged for all image-based benchmarks. For the recommender system and anomaly detection benchmarks, the final results show slight changes, as shown in Figures 15A and 15B.

[0119] In short, analog circuits, although subject to noise, can perform gradient descent functions well even in the presence of significant noise without affecting the quality of the overall training process.

[0120] discussion Overall, the experimental analysis of the two architectures provides some evidence that significant performance and energy benefits can be obtained in a very small additional chip footprint, even without new algorithms that specifically exploit the hardware capabilities. On the other hand, computational substrates such as the TPU have a clearly more general purpose. At this stage, the disclosed analysis does not yet suggest that it is a convincing design in itself. However, the general direction shows promise for several reasons.

[0121] First, the Ising substrate is already quite useful in accelerating a wide variety of optimization problems. In the future, it will become even more versatile and widely used. Thus, the cost of additional architectural support will be further reduced. Second, the disclosed design may be improved by further research on generality, performance, reconfigurability, and support for exploiting parallelism in the training set. Finally, energy-based models have many advantages, but they are notoriously expensive to train. With the advent of prototype Ising machines and systems such as the disclosed Boltzmann gradient follower, better algorithms may emerge.

[0122] Experiment #2 - Bias Unlike maximum likelihood (ML) learning algorithms, contrastive divergence (CD) algorithms are known to be biased. That is, the fixed points of the CD algorithm are not identical to the fixed points of the ML algorithm. Although this topic has generated many publications, it is not a problem in practice. First, the bias has been shown to be small. Second, the ultimate goal of the algorithm is to capture the training data sufficiently to be useful. Nevertheless, what is presented here is an empirical observation of the bias with respect to the disclosed modified training algorithm. In this experiment, the same method was used as used by Carreira-Perpinan and Hinton in their original investigation. The size of the system was small enough to allow the ground truth to be obtained by enumeration. The system consists of 12 visible and 4 hidden units, all binary. Sixty different distributions were randomly generated from 100 training images. Then, ML, CD, and the disclosed training algorithm (BGF) were run for the same 1000 iterations to obtain the weights. Finally, the obtained probability distributions were compared to the ground truth by measuring the KL divergence. Each algorithm run produced one KL divergence measurement. For each algorithm, 400 runs were performed from different random initial conditions. The resulting 400 different measurements are plotted as a cumulative probability distribution. Figure 16 shows the results.

[0123] First, it is shown that ML learning, although impractical, does not bias the final estimate. In contrast, the remaining algorithms have fairly similar bias properties. This is not surprising, since the BGF is in fact a modification of the CD algorithm. However, based on the inherent speed of exploring the phase space, the BGF can be thought of as CD-k with very large k. As k → ∞, CD-k effectively becomes ML. As a result, the BGF is indeed less likely to provide a larger KL divergence compared to the traditional CD-k. Overall, the main point is clear: the disclosed BGF does not pose the problem of biased estimates. If anything, it improves the bias properties of the commonly used von Neumann algorithm.

[0124] The graph in Figure 16 shows the cumulative probability distribution of the KL divergence against the ground truth for the training results of CD and BGF. In other words, every point (x, y) on the curve indicates that y% of the training distribution has a final KL divergence less than or equal to x from the ground truth.

[0125] conclusion Ising machines can exploit nature to perform very fast, energy-efficient, and effective computations. Ising machines can also be used to perform operations on energy-based models, such as restricted Boltzmann machines (RBMs) and other differential algorithms. In this disclosure, two different designs were presented that extend the Ising substrate with additional circuitry to support RBM training. With some small modifications, it was shown that the Ising machine can easily act as a Gibbs sampler to speed up parts of the RBM algorithm, resulting in about a 2x speedup and 2.3x energy improvement compared to the TPU host. With more significant modifications, the substrate can act as a Boltzmann sampler, essentially training the RBM while following the gradient, without any additional host computation. Compared to a TPU host with a significantly larger chip area, such a Boltzmann gradient follower can achieve a 29x speedup and 1000x energy savings. With further research, hardware-software co-designed nature-based computing systems could become an important new architectural modality.

[0126] The disclosures of any and all patents, patent applications, and publications cited herein are incorporated herein by reference in their entirety. Although the present invention has been disclosed with reference to specific embodiments, it will be apparent to those skilled in the art that other embodiments and variations of the present invention may be devised without departing from the true spirit and scope of the present invention. It is intended that the appended claims be construed to include all such embodiments and equivalent variations.

[0127] References The following publications are incorporated herein by reference in their entireties:

[0128] S. Kirkpatrick, CD Gelatt, and MPVecchi, “Optimization by simulated annealing,” Science, vol. 220, no. 4598, pp. 671-680, 1983.

[0129] G.Zames,N.Ajlouni,N.Ajlouni,N.Ajlouni,J.Holland,W.Hills,and D.Goldberg,“Genetic algorithms in search,optimization and machine learning,” Information Technology Journal,vol.3,no.1,pp.301-302,1981.

[0130] D.H.Ackley,G.E.Hinton,and T.J.Sejnowski,“A learning algorithm for boltzmann machines,” Cognitive science,vol.9,no.1,pp.147-169,1985.

[0131] A.Narayanan and M.Moore,“Quantum-inspired genetic algorithms,” in Proceedings of IEEE International Conference on Evolutionary Computation,1996,pp.61-66.

[0132] R.M.Karp,Reducibility among Combinatorial Problems.Boston,MA: Springer US,1972,pp.85-103.

[0133] A.Lucas,“Ising formulations of many np problems,” Frontiers in Physics,vol.2,p.5,2014.

[0134] K.Kim,M.-S.Chang,S.Korenblit,R.Islam,E.E.Edwards,J.K.Freericks,G.-D.Lin,L.-M.Duan,and C.Monroe,“Quantum simulation of frustrated ising spins with trapped ions,” Nature,vol.465,no.7298,pp.590-593,2010.

[0135] N.G.Berloff,M.Silva,K.Kalinin,A.Askitopoulos,J.D.Topfer,P.Cilibrizzi,W.Langbein,and P.G.Lagoudakis,“Realizing the classical xy hamiltonian in polariton simulators,” Nature materials,vol.16,no.11,pp.1120-1126,2017.

[0136] R.Barends,A.Shabani,L.Lamata,J.Kelly,A.Mezzacapo,U.Las Heras,R.Babbush,A.G.Fowler,B.Campbell,Y.Chen et al.,“Digitized adiabatic quantum computing with a superconducting circuit,” Nature,vol.534,no.7606,pp.222-226,2016.

[0137] M.Yamaoka,C.Yoshimura,M.Hayashi,T.Okuyama,H.Aoki,and H.Mizuno,“A 20k-spin ising chip to solve combinatorial optimization problems with cmos annealing,” IEEE Journal of Solid-State Circuits,vol.51,no.1,pp.303-309,2015.

[0138] P.I.Bunyk,E.M.Hoskinson,M.W.Johnson,E.Tolkacheva,F.Altomare,A.J.Berkley,R.Harris,J.P.Hilton,T.Lanting,A.J.Przybysz et al.,“Architectural considerations in the design of a superconducting quantum annealing processor,” IEEE Transactions on Applied Superconductivity,vol.24,no.4,pp.1-10,2014.

[0139] A.D.King,J.Carrasquilla,J.Raymond,I.Ozfidan,E.Andriyash,A.Berkley,M.Reis,T.Lanting,R.Harris,F.Altomare et al.,“Observation of topological phenomena in a programmable lattice of 1,800 qubits,” Nature,vol.560,no.7719,pp.456-460,2018.

[0140] F.Bohm,G.Verschaffelt,and G.Van der Sande,“A poor man’s coherent ising machine based on opto-electronic feedback systems for solving optimization problems,” Nature communications,vol.10,no.1,pp.1-9,2019.

[0141] R.Hamerly,A.Sludds,L.Bernstein,M.Prabhu,C.Roques-Carmes,J.Carolan,Y.Yamamoto,M.Soljacic,and D.Englund,“Towards large-scale photonic neural-network accelerators,” in 2019 IEEE International Electron Devices Meeting(IEDM).IEEE,2019,pp.22-8.

[0142] D.Pierangeli,G.Marcucci,and C.Conti,“Large-scale photonic ising machine by spatial light modulation,” Physical review letters,vol.122,no.21,p.213902,2019.

[0143] R.Harris,M.W.Johnson,T.Lanting,A.J.Berkley,J.Johansson,P.Bunyk,E.Tolkacheva,E.Ladizinsky,N.Ladizinsky,T.Oh,F.Cioata,I.Perminov,P.Spear,C.Enderud,C.Rich,S.Uchaikin,M.C.Thom,E.M.Chapple,J.Wang,B.Wilson,M.H.S.Amin,N.Dickson,K.Karimi,B.Macready,C.J.S.Truncik,and G.Rose,“Experimental investigation of an eight-qubit unit cell in a superconducting optimization processor,” Phys.Rev.B,vol.82,p.024511,Jul 2010.

[0144] T.Inagaki,Y.Haribara,K.Igarashi,T.Sonobe,S.Tamate,T.Honjo,A.Marandi,P.L.McMahon,T.Umeki,K.Enbutsu et al.,“A coherent ising machine for 2000-node optimization problems,” Science,vol.354,no.6312,pp.603-606,2016.

[0145] R.Afoakwa,Y.Zhang,U.K.R.Vengalam,Z.Ignjatovic,and Michael,“Brim: Bistable resistively coupled ising machine,” 2021.

[0146] S.Boixo,T.F.Ronnow,S.V.Isakov,Z.Wang,D.Wecker,D.A.Lidar,J.M.Martinis,and M.Troyer,“Evidence for quantum annealing with more than one hundred qubits,” Nature physics,vol.10,no.3,pp.218-224,2014.

[0147] R.Hamerly,T.Inagaki,P.L.McMahon,D.Venturelli,A.Marandi,T.Onodera,E.Ng,C.Langrock,K.Inaba,T.Honjo et al.,“Experimental investigation of performance differences between coherent ising machines and a quantum annealer,” Science advances,vol.5,no.5,p.eaau0823,2019.

[0148] R.Hamerly,T.Inagaki,P.L.McMahon,D.Venturelli,A.Marandi,T.Onodera,E.Ng,C.Langrock,K.Inaba et al.,“Scaling advantages of all-to-all connectivity in physical annealers: the coherent ising machine vs.d-wave 2000q,” arXiv:arXiv:1807.00089,2018.

[0149] Y.Yamamoto,K.Aihara,T.Leleu,K.-i.Kawarabayashi,S.Kako,M.Fejer,K.Inoue,and H.Takesue,“Coherent ising machines-optical neural networks operating at the quantum limit,” npj Quantum Information,vol.3,no.1,pp.1-15,2017.

[0150] P.L.McMahon,A.Marandi,Y.Haribara,R.Hamerly,C.Langrock,S.Tamate,T.Inagaki,H.Takesue,S.Utsunomiya,K.Aihara,R.L.Byer,M.M.Fejer,H.Mabuchi,and Y.Yamamoto,“A fully programmable 100-spin coherent ising machine with all-to-all connections,” Science,vol.354,no.6312,pp.614-617,2016.

[0151] K.Takata,A.Marandi,R.Hamerly,Y.Haribara,D.Maruo,S.Tamate,H.Sakaguchi,S.Utsunomiya,and Y.Yamamoto,“A 16-bit coherent ising machine for one-dimensional ring and cubic graph problems,” Scientific Reports,vol.6,no.1,p.34089,2016.

[0152] F.Bohm,G.Verschaffelt,and G.Van der Sande,“A poor man’s coherent ising machine based on opto-electronic feedback systems for solving optimization problems,” Nature Communications,vol.10,no.1,p.3538,2019.

[0153] Y.Takeda,S.Tamate,Y.Yamamoto,H.Takesue,T.Inagaki,and S.Utsunomiya,“Boltzmann sampling for an XY model using a non-degenerate optical parametric oscillator network,” Quantum Science and Technology,vol.3,no.1,p.014004,Nov 2017.

[0154] K.Cho,A.Ilin,and T.Raiko,“Improved learning of gaussian-bernoulli restricted boltzmann machines,” in International conference on artificial neural networks.Springer,2011,pp.10-17.

[0155] Y.LeCun,S.Chopra,M.Ranzato,and F.-J.Huang,“Energy-based models in document recognition and computer vision,” in Ninth International Conference on Document Analysis and Recognition(ICDAR 2007),vol.1.IEEE,2007,pp.337-341.

[0156] S.Ding,J.Zhang,N.Zhang,and Y.Hou,“Boltzmann machine and its applications in image recognition,” in Intelligent Information Processing VIII,Z.Shi,S.Vadera,and G.Li,Eds.Cham: Springer International Publishing,2016,pp.108-118.

[0157] G.E.Dahl,M.Ranzato,A.r.Mohamed,and G.Hinton,“Phone recognition with the mean-covariance restricted boltzmann machine,” in Proceedings of the 23rd International Conference on Neural Information Processing Systems -Volume 1,ser.NIPS’10.Red Hook,NY,USA: Curran Associates Inc.,2010,p.469-477.

[0158] R.D.Hjelm,V.D.Calhoun,R.Salakhutdinov,E.A.Allen,T.Adali,and S.M.Plis,“Restricted boltzmann machines for neuroimaging: An application in identifying intrinsic networks,” NeuroImage,vol.96,pp.245-260,2014.

[0159] A.Al-Waisy,M.A.Mohammed,S.Al-Fahdawi,M.Maashi,B.Garcia-Zapirain,K.H.Abdulkareem,S.Mostafa,N.M.Kumar,and D.N.Le,“Covid-deepnet: Hybrid multimodal deep learning system for improving covid-19 pneumonia detection in chest x-ray images,” CMC-COMPUTERS MATERIALS & CONTINUA,vol.67,no.2,pp.2409-2429,2021.

[0160] Q.Zhang,Y.Xiao,W.Dai,J.Suo,C.Wang,J.Shi,and H.Zheng,“Deep learning based classification of breast tumors with shear-wave elastography,” Ultrasonics,vol.72,pp.150-157,2016.

[0161] R.Salakhutdinov and G.Hinton,“Deep boltzmann machines,” in Artificial intelligence and statistics.PMLR,2009,pp.448-455.

[0162] H.Wang,Y.Cai,and L.Chen,“A vehicle detection algorithm based on deep belief network,” The scientific world journal,vol.2014,2014.

[0163] A.M.Abdel-Zaher and A.M.Eldeib,“Breast cancer classification using deep belief networks,” Expert Systems with Applications,vol.46,pp.139-144,2016.

[0164] R.Salakhutdinov,A.Mnih,and G.Hinton,“Restricted boltzmann machines for collaborative filtering,” in Proceedings of the 24th international conference on Machine learning,2007,pp.791-798.

[0165] A.Gouveia and M.Correia,“A systematic approach for the application of restricted boltzmann machines in network intrusion detection,” in International Work-Conference on Artificial Neural Networks.Springer,2017,pp.432-446.

[0166] U.Fiore,F.Palmieri,A.Castiglione,and A.De Santis,“Network anomaly detection with the restricted boltzmann machine,” Neurocomputing,vol.122,pp.13-23,2013.

[0167] Y.Wang,Z.Pan,X.Yuan,C.Yang,and W.Gui,“A novel deep learning based fault diagnosis approach for chemical process with extended deep belief network,” ISA transactions,vol.96,pp.457-467,2020.

[0168] J.Xie,G.Du,C.Shen,N.Chen,L.Chen,and Z.Zhu,“An end-to-end model based on improved adaptive deep belief network and its application to bearing fault diagnosis,” IEEE Access,vol.6,pp.63 584-63 596,2018.

[0169] N.Srivastava,R.R.Salakhutdinov,and G.E.Hinton,“Modeling documents with deep boltzmann machines,” arXiv:1309.6865,2013.

[0170] M.Fatemi and M.Safayani,“Joint sentiment / topic modeling on text data using a boosted restricted boltzmann machine,” Multimedia Tools and Applications,vol.78,no.15,pp.20 637-20 653,2019.

[0171] G.Hinton and R.Salakhutdinov,“Discovering binary codes for documents by learning deep generative models,” Topics in Cognitive Science,vol.3,no.1,pp.74-91,2011.

[0172] J.Hernandez and A.G.Abad,“Learning from multivariate discrete sequential data using a restricted boltzmann machine model,” in 2018 IEEE 1st Colombian Conference on Applications in Computational Intelligence(ColCACI).IEEE,2018,pp.1-6.

[0173] H.Larochelle,Y.Bengio,J.Louradour,and P.Lamblin,“Exploring strategies for training deep neural networks.”Journal of machine learning research,vol.10,no.1,2009.

[0174] D.Erhan,A.Courville,Y.Bengio,and P.Vincent,“Why does unsupervised pre-training help deep learning?” in Proceedings of the thirteenth international conference on artificial intelligence and statistics.JMLR Workshop and Conference Proceedings,2010,pp.201-208.

[0175] Y.Hua,J.Guo,and H.Zhao,“Deep belief networks and deep learning,” in Proceedings of 2015 International Conference on Intelligent Computing and Internet of Things.IEEE,2015,pp.1-4.

[0176] E.Qin,A.Samajdar,H.Kwon,V.Nadella,S.Srinivasan,D.Das,B.Kaul,and T.Krishna,“Sigma: A sparse and irregular gemm accelerator with flexible interconnects for dnn training,” in 2020 IEEE International Symposium on High Performance Computer Architecture(HPCA),2020,pp.58-70.

[0177] S.Levantino,G.Marzin,and C.Samori,“An adaptive pre-distortion technique to mitigate the dtc nonlinearity in digital plls,” IEEE Journal of Solid-State Circuits,vol.49,no.8,pp.1762-1772,2014.

[0178] M.R.Li,C.H.Yang,and Y.L.Ueng,“A 5.28-gb / s ldpc decoder with time-domain signal processing for ieee 802.15.3c applications,” IEEE Journal of Solid-State Circuits,vol.52,no.2,pp.592-604,2016.

[0179] D.Miyashita,R.Yamaki,K.Hashiyoshi,H.Kobayashi,S.Kousai,Y.Oowaki,and Y.Unekawa,“An ldpc decoder with time-domain analog and digital mixed-signal processing,” IEEE Journal of Solid-State Circuits,vol.49,no.1,pp.73-83,2013.

[0180] K.Madani,P.Garda,E.Belhaire,and F.Devos,“Two analog counters for neural network implementation,” IEEE Journal of Solid-State Circuits,vol.26,no.7,pp.966-974,1991.

[0181] N.P.Jouppi,C.Young,N.Patil,D.Patterson,G.Agrawal,R.Bajwa,S.Bates,S.Bhatia,N.Boden,A.Borchers et al.,“In-datacenter performance analysis of a tensor processing unit,” in Proceedings of the 44th Annual International Symposium on Computer Architecture,2017,pp.1-12.

[0182] L.Deng,“The mnist database of handwritten digit images for machine learning research [best of the web],” IEEE Signal Processing Magazine,vol.29,no.6,pp.141-142,2012.

[0183] T.Clanuwat,M.Bober-Irizar,A.Kitamoto,A.Lamb,K.Yamamoto,and D.Ha,“Deep learning for classical japanese literature,” arXiv:1812.01718,2018.

[0184] H.Xiao,K.Rasul,and R.Vollgraf,“Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” arXiv:1708.07747,2017.

[0185] G.Cohen,S.Afshar,J.Tapson,and A.Van Schaik,“Emnist: Extending mnist to handwritten letters,” in 2017 International Joint Conference on Neural Networks(IJCNN).IEEE,2017,pp.2921-2926.

[0186] A.Krizhevsky,G.Hinton et al.,“Learning multiple layers of features from tiny images,”2009.

[0187] Y.LeCun,F.J.Huang,and L.Bottou,“Learning methods for generic object recognition with invariance to pose and lighting,” in Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition,2004.CVPR 2004.,vol.2.IEEE,2004,pp.II-104.

[0188] A.Coates,A.Ng,and H.Lee,“An analysis of single-layer networks in unsupervised feature learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics.JMLR Workshop and Conference Proceedings,2011,pp.215-223.

[0189] S.Verma,P.Patel,and A.Majumdar,“Collaborative filtering with label consistent restricted boltzmann machine,” in 2017 Ninth International Conference on Advances in Pattern Recognition(ICAPR).IEEE,2017,pp.1-6.

[0190] A.Pumsirirat and L.Yan,“Credit card fraud detection using deep learning based on auto-encoder and restricted boltzmann machine,” International Journal of Advanced Computer Science and Applications,vol.9,no.1,2018.

[0191] F.M.Harper and J.A.Konstan,“The movielens datasets: History and context,” Acm transactions on interactive intelligent systems(tiis),vol.5,no.4,pp.1-19,2015.

[0192] R.Salakhutdinov and I.Murray,“On the quantitative analysis of deep belief networks,” in Proceedings of the 25th international conference on Machine learning,2008,pp.872-879.

[0193] W.Li,P.Xu,Y.Zhao,H.Li,Y.Xie,and Y.Lin,“Timely: Pushing data movements and interfaces in pim accelerators towards local and in time domain,” in 2020 ACM / IEEE 47th Annual International Symposium on Computer Architecture(ISCA).IEEE,2020,pp.832-845.

[0194] X.He,S.Pal,A.Amarnath,S.Feng,D.-H.Park,A.Rovinski,H.Ye,Y.Chen,R.Dreslinski,and T.Mudge,“Sparse-tpu: Adapting systolic arrays for sparse matrices,” in Proceedings of the 34th ACM International Conference on Supercomputing,2020,pp.1-12.

[0195] M.A.Carreira-Perpinan and G.Hinton,“On contrastive divergence learning,” in International workshop on artificial intelligence and statistics.PMLR,2005,pp.33-40.

Claims

1. 1. A bistable resistive coupling system, comprising: A plurality of visible nodes; A number of hidden nodes; a plurality of coupling elements electrically connected to a visible node among the plurality of visible nodes and a hidden node among the plurality of hidden nodes, 11. A bistable resistive coupling system, wherein each of the plurality of coupling elements comprises a programmable resistor.

2. The system of claim 1 , wherein each of the plurality of coupling elements includes two programmable resistors.

3. 3. The system of claim 1, wherein each programmable resistor comprises a field effect transistor having a source, a gate, and a drain, with a gate capacitor connected between the source and the gate.

4. 3. The system of claim 1, wherein each of the plurality of coupling elements includes an analog counter having an overflow signal and an underflow signal, the overflow signal being configured to increase the value of the programmable resistor and the underflow signal being configured to decrease the value of the programmable resistor.

5. 3. The system of claim 1, wherein at least one of the plurality of visible nodes or the plurality of hidden nodes comprises a sigmoid element, the sigmoid element comprising an inverter having an input, an output, and a load resistor connected between the output and a common mode reference.

6. 3. The system of claim 1, wherein at least one of the plurality of visible nodes or the plurality of hidden nodes comprises a random noise generator, the random noise generator comprising a binary random number generator having an output and a low pass filter connected to the output.

7. 7. The system of claim 6, further comprising a comparator having a first input and a second input, the first input being connected to an output of a sigmoid element and the second input being connected to the filtered output of the binary random number generator.

8. 3. The system of claim 1, wherein at least one of the plurality of visible nodes or the plurality of hidden nodes comprises a capacitor and a feedback unit connected across the capacitor, the feedback unit being configured to make a voltage across the capacitor bistable.

9. The system of any one of claims 1 to 2, wherein at least one of the plurality of visible nodes or the plurality of hidden nodes comprises a buffer.

10. 1. A coupling device for connecting a first and a second node in a network, comprising: an inverting input and a non-inverting input; first and second field effect transistors each having a drain, a gate, and a source, the drain of the first field effect transistor being connected to the non-inverting input and the drain of the second field effect transistor being connected to the inverting input; first and second gate capacitors respectively connected between the gate and the source of the first and second field effect transistors; a sum output connected to the sources of the first and second field effect transistors; a voltage adjustment element connected to the gates of the first and second field effect transistors, the voltage adjustment element configured to adjust a voltage of the gates of the first and second field effect transistors in response to a control signal.

11. The coupling device of claim 10 , wherein the voltage regulation element comprises an analog counter.

12. A coupling device according to any one of claims 10 to 11, further comprising at least one current source switchably connected to the gate of the first or second field effect transistor.

13. 12. The coupling device of claim 10, further comprising four current sources, one current source switchably connected to each of the gates of the first and second field effect transistors and connected to a positive voltage or ground.

14. 12. The coupling device of claim 11, wherein the voltage adjustment element further comprises overflow and underflow outputs of the analog counter configured to increase or decrease the amount of charge on the first and second gate capacitors.

15. The coupling device according to any one of claims 10 to 11, wherein the first and second field effect transistors are N-channel field effect transistors.

16. 1. A method for training a bistable resistive coupling system, comprising: initializing a set of weighting elements and a set of biasing elements in the bistable resistive coupling system; initializing a set of visible nodes of the bistable resistive coupling system to a first set of initial values; clamping the set of visible nodes to the first set of initial values ​​for a period of time and allowing the set of hidden nodes to settle to a first set of hidden values; incrementing a counter of at least one weighting factor based on a product of the first set of initial values ​​and the first set of hidden values; initializing a set of hidden nodes of the bistable resistive coupling system to a set of random values ​​selected from a table of hidden values; annealing the visible and hidden nodes for a second period of time; decrementing the counter of at least one weighting factor based on the annealed values ​​of the visible and hidden nodes; increasing or decreasing a weighting value of the at least one weighting factor when the counter of the at least one weighting factor overflows or underflows; and repeating a programmable number of learning steps from the step of initializing the set of visible nodes.

17. 17. The method of claim 16, wherein the set of values ​​used to initialize the set of hidden nodes is obtained from a set of corresponding hidden values ​​from a previous annealing step.

18. 17. The method of claim 16, further comprising the step of reading a combined value from the system using at least one analog-to-digital converter.

19. The method of any one of claims 16 to 17, wherein the period of time is in the range of 1 nanosecond or less.

20. The method of any one of claims 16 to 18, wherein the second fixed period of time is in the range of 1 nanosecond or less.

21. The method of any one of claims 16 to 18, further comprising the step of storing the annealed values ​​of the hidden nodes in a table of the hidden values ​​after annealing.

Images