Neuromorphic analog signal processor for predictive maintenance of machines

Abstract

Systems, methods, and apparatus are provided for predictive maintenance of a machine. An example apparatus includes a vibration sensor configured to sense vibration of a vibration source and an analog circuit. The analog circuit includes a plurality of operational amplifiers and a plurality of resistors. The analog circuit is coupled to the vibration sensor and is configured to: receive an analog signal from the vibration sensor; and calculating an output based on the analog signal by executing a portion of the trained neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation-of-and claims priority to U.S. patent application No. 17 / 902,757, filed on September 2, 2022, entitled “Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines,” which is a part-continuation-of-U.S. application No. 17 / 744,565, filed on May 13, 2022, entitled “Systems and Methods for Human Activity Recognition Using Analog Neuromorphic Computing Hardware,” which is a part-continuation-of-U.S. application No. 17 / 189,109, filed on March 1, 2021, entitled “Analog Hardware Realization of Neural Networks,” which is a continuation-of PCT application No. PCT / RU2020 / 000306, entitled “Analog Hardware Realization of Neural Networks,” filed on June 25, 2020, each of which is incorporated herein by reference in its entirety. U.S. Application No. 17 / 189,109 is also a continuation-in-part of PCT Application No. PCT / EP2020 / 067800, filed on June 25, 2020, entitled “Analog Hardware Realization of Neural Networks,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The disclosed implementations relate generally to neural networks, and more particularly to systems and methods for predictive maintenance of machines using neuromorphic analog signal processors.

[0004] background

[0005] Traditional hardware has failed to keep up with innovations in neural networks and the growing popularity of machine learning-based applications. As the development of digital microprocessors has stagnated, the complexity of neural networks continues to exceed the computational power of CPUs and GPUs. Neuromorphic processors based on spike neural networks (such as Loihi and True North) are limited in their applications. For GPU-like architectures, the power and speed of such architectures are limited by the speed of data transmission. Data transmission can consume up to 80% of chip power and can significantly affect the speed of calculations. Edge applications require low power consumption, but there are currently no known high-performance hardware implementations that consume less than 50 milliwatts of power.

[0006] Memristor-based architectures using crossbar technology are still impractical for making recurrent and feed-forward neural networks. For example, memristor-based crossbars have many disadvantages, including current leakage and high latency during operation, which make them impractical. In addition, there are reliability issues when making memristor-based crossbars, especially when the neural network has negative and positive weights. For large neural networks with many neurons, in high dimensions, memristor-based crossbars cannot be used for simultaneous propagation of different signals, which in turn complicates the summation of signals when neurons are represented by operational amplifiers. In addition, memristor-based analog integrated circuits have many limitations, such as the small number of resistance states, the first cycle problem when forming a memristor, the complexity of channel formation when training a memristor, the unpredictable dependence on the size of the memristor, the slow operation of the memristor, and the drift of the resistance state.

[0007] In addition, the training process required for neural networks presents unique challenges to the hardware implementation of neural networks. The trained neural network is used for specific inferencing tasks, such as classification. Once the neural network is trained, the hardware equivalent is manufactured. When the neural network is retrained, the hardware manufacturing process is repeated, driving up costs. Although there are some reconfigurable hardware solutions, such hardware is not easily mass-produced and costs much more than non-reconfigurable hardware (e.g., five times the cost). In addition, edge environments (such as smart home applications) do not require reprogrammability themselves. For example, 85% of all applications of neural networks do not require any retraining during operation, so on-chip learning is not that useful. In addition, edge applications include noisy environments that can make reprogrammable hardware unreliable.

[0008] One such application is predictive maintenance of machines. Vibration sensors are often used to measure vibrations in machinery, tracks, railway vehicles, wind turbines, and oil and gas pumps. Sensors that monitor rotating or reciprocating parts of machinery typically generate signals in the 0 to 20kHz range. This makes the sensors consume a lot of power, making it almost impossible to use batteries. The signals generated due to vibrations can be transmitted wirelessly to a device for analysis. A broadband connection to the control unit is often necessary because the signal can include a large data stream. This data stream may reduce the battery life of the operating sensor node.

[0009] Overview

[0010] Therefore, there is a need for methods, circuits and / or interfaces that address at least some of the deficiencies noted above. Analog circuits that model trained neural networks and are manufactured according to the techniques described herein can provide advantages of improved performance per watt, can be useful in implementing hardware solutions in edge environments, and can address a variety of applications, such as drone navigation and self-driving cars. The cost advantages provided by the proposed manufacturing methods and / or analog network architectures are more pronounced for larger neural networks. In addition, analog hardware implementations of neural networks provide increased parallelism and neuromorphism. In addition, neuromorphic analog components are insensitive to noise and temperature variations when compared to digital counterparts.

[0011] Chips made according to the techniques described herein offer orders of magnitude improvements in size, power, and performance over conventional systems and are ideal for edge environments, including for retraining purposes. Such analog neuromorphic chips can be used to implement edge computing applications or in Internet of Things (IoT) environments. Due to analog hardware, preliminary processing that can consume more than 80%-90% of power (e.g., formation of descriptors for image recognition) can be moved onto the chip, thereby reducing energy consumption and network load, which can open up new application markets.

[0012] Various edge applications can benefit from the use of such analog hardware. For example, for video processing, the technology described in this article can be used to include direct connections to CMOS sensors without digital interfaces. Various other video processing applications include road sign recognition for cars, real depth and / or synchronous positioning and map building based on cameras for robots, room access control without server connections, and always-on solutions for security and healthcare. Such chips can be used for data processing of radars and lidars and for low-level data fusion. Such technology can be used to implement battery management functions for large battery packs, sound / voice processing without connections to data centers, voice recognition on mobile devices, wake-up voice commands for IoT sensors, translators that translate one language into another, large sensor arrays of IoT with low signal strength, and / or configurable process control with hundreds of sensors. In particular, this technology can be used to implement human activity recognition. Replacing conventional software algorithms with simulated neural network implementations can help unload calculations from device microcontrollers, so that the total power consumption of the device is significantly reduced (e.g., power consumption is reduced by more than 40%). In addition, the technology described in this article can be used to distinguish a large number of human activities, and this technology can be used to develop personalized solutions for different users or user categories. For example, neural network determination or generation of descriptors of human activities allows recognition of more than fifty different human activities. The neural network allows personalization of activities and descriptors can be generated for a specific user as a unique digital fingerprint.

[0013] According to some implementations, neuromorphic simulation chips can be mass-produced after standard software-based neural network simulation / training. Client neural networks can be easily ported with custom chip design and production, regardless of the structure of the neural network. In addition, according to some implementations, libraries are provided that are ready to make on-chip solutions (network simulators). Such solutions only require training once with a lithography mask change, after which the chip can be mass-produced. For example, during chip production, only part of the lithography mask needs to be changed.

[0014] The techniques described herein can be used to design and / or manufacture analog neuromorphic integrated circuits that are mathematically equivalent to trained neural networks (feedforward or recurrent neural networks). According to some implementations, the process begins with a trained neural network, which is first converted into a transformed network composed of standard elements. Software with a known model representing the standard elements is used to simulate the operation of the transformed network. Software simulation is used to determine the individual resistance value of each resistor in the transformed network. The photolithography mask is arranged based on the arrangement of the standard elements in the transformed network. Each standard element is arranged in a mask using an existing library of circuits corresponding to the standard elements to simplify and speed up the process. In some implementations, the resistors are arranged in one or more masks separated from masks including other elements (e.g., operational amplifiers) in the transformed network. In this way, if the neural network is retrained, only the mask containing the resistors or other types of fixed resistance elements representing the new weights in the retrained neural network needs to be regenerated, which simplifies and speeds up the process. The photolithography mask is then sent to a fab for manufacturing analog neuromorphic integrated circuits.

[0015] In one aspect, according to some implementations, a method for hardware implementation of a neural network is provided. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology into an equivalent analog network of analog components. The method also includes calculating a weight matrix of the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection between analog components of the equivalent analog network. The method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​of the analog components.

[0016] In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix and represents a resistance value.

[0017] In some implementations, the method further includes obtaining new weights for the trained neural network, calculating a new weight matrix for the equivalent analog network based on the new weights, and generating a new resistance matrix for the new weight matrix.

[0018] In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons calculates a corresponding output based on a corresponding mathematical function, and transforming the neural network topology into an equivalent simulated network of simulated components includes: for each layer of the one or more layers of neurons: (i) identifying one or more function blocks for the corresponding layer based on the corresponding mathematical function. Each function block has a corresponding schematic implementation, and the corresponding schematic implementation has a block output that conforms to the output of the corresponding mathematical function; and (ii) generating a corresponding multi-layer network of simulated neurons based on arranging the one or more function blocks. Each simulated neuron implements a corresponding function of the one or more function blocks, and each simulated neuron of a first layer of the multi-layer network is connected to one or more simulated neurons of a second layer of the multi-layer network.

[0019] In some implementations, the one or more function blocks include one or more basic function blocks selected from the group consisting of: (i) having a block output V out =reLU(∑w i .V i in + bias), ReLU is the rectified linear unit (ReLU) activation function or a similar activation function, V i represents the i-th input, w i represents the weight corresponding to the i-th input, and bias represents the bias value, and ∑ is a summation operator; (ii) with block output V out =Coeff.V i .V j The signal multiplier block of . i represents the ith input, and V j represents the jth input, and coeff is a predetermined coefficient; (iii) has a block output , V represents the input, and A and B are the predetermined coefficient values ​​of the sigmoid activation block; (iv) having a block output V out =A*tanh(B*V in ) of the hyperbolic tangent activation block, V in represents input, and A and B are predetermined coefficient values; and (v) a signal delay block having a block output U(t)=V(t-dt), t represents the current time period, V(t-dt) represents the output of the signal delay block of the previous time period t-dt, and dt is the delay value.

[0020] In some implementations, identifying the one or more functional blocks includes selecting the one or more functional blocks based on a type of the corresponding layer.

[0021] In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons calculates a corresponding output based on a corresponding mathematical function, and transforming the neural network topology into an equivalent analog network of the analog component includes: (i) decomposing a first layer of the neural network topology into a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks for the corresponding sub-layer based on the corresponding intermediate mathematical function; and (b) generating a corresponding multi-layer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a corresponding function of the one or more sub-function blocks, and each analog neuron of the first layer of the multi-layer analog sub-network is connected to one or more analog neurons of the second layer of the multi-layer analog sub-network.

[0022] In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting the one or more weights such that combining the one or more intermediate functions produces the mathematical function.

[0023] In some implementations, the method further includes: (i) generating an equivalent digital network of digital components for one or more output layers of the neural network topology; and (ii) connecting outputs of one or more layers of the equivalent analog network to the equivalent digital network of digital components.

[0024] In some implementations, the analog component includes a plurality of operational amplifiers and a plurality of resistors, each operational amplifier representing an analog neuron of the equivalent analog network, and each resistor representing a connection between two analog neurons.

[0025] In some implementations, selecting component values ​​for the analog components includes performing a gradient descent method to identify possible resistance values ​​for the plurality of resistors.

[0026] In some implementations, the neural network topology includes one or more GRU or LSTM neurons, and transforming the neural network topology includes generating one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons.

[0027] In some implementations, one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency of the neural network topology.

[0028] In some implementations, the neural network topology includes one or more layers of neurons that implement infinite activation functions, and transforming the neural network topology includes applying one or more transformations selected from the group consisting of: (i) replacing the infinite activation function with a finite activation; and (ii) adjusting connections or weights of the equivalent simulated network so that for predetermined one or more inputs, the difference in output between the trained neural network and the equivalent simulated network is minimized.

[0029] In some implementations, the method further includes generating one or more photolithography masks for fabricating a circuit implementing an equivalent analog network of the analog components based on the resistance matrix.

[0030] In some implementations, the method further includes: (i) obtaining new weights for the trained neural network; (ii) calculating a new weight matrix for the equivalent analog network based on the new weights; (iii) generating a new resistance matrix for the new weight matrix; and (iv) generating a new photolithography mask for manufacturing a circuit that implements the equivalent analog network of the analog component based on the new resistance matrix.

[0031] In some implementations, the trained neural network is trained using software simulation to generate weights.

[0032] In another aspect, according to some implementations, a method for hardware implementation of a neural network is provided. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes calculating one or more connection constraints based on analog integrated circuit (IC) design constraints. The method also includes transforming the neural network topology into an equivalent sparsely connected network of analog components that satisfies the one or more connection constraints. The method also includes calculating a weight matrix of the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection between analog components of the equivalent sparsely connected network.

[0033] In some implementations, transforming the neural network topology into an equivalent sparsely connected network of analog components includes deriving possible input connectivity N according to one or more connection constraints. i and output connectivity N o .

[0034] In some implementations, the neural network topology includes at least one densely connected layer having K inputs and L outputs and a weight matrix U. In such a case, transforming the at least one densely connected layer includes constructing a neural network topology having K inputs, L outputs, and The equivalent sparsely connected network of layers makes the input connectivity no more than N i , and the output connectivity does not exceed N o .

[0035] In some implementations, the neural network topology includes at least one densely connected layer having K inputs and L outputs and a weight matrix U. In such a case, transforming the at least one densely connected layer includes constructing a neural network topology having K inputs, L outputs, and Each layer m is represented by the corresponding weight matrix U m Represented by , where missing connections are represented by zeros, so that the input connectivity does not exceed N i , and the output connectivity does not exceed N o . Equation U=Π m＝1..M U m is met with a predetermined accuracy.

[0036] In some implementations, the neural network topology includes a neural network with K inputs and L outputs, with a maximum input connectivity P i , maximum output connectivity P o and a single sparsely connected layer with a weight matrix U, where missing connections are represented by zeros. In this case, transforming the single sparsely connected layer involves constructing a network with K inputs, L outputs, The equivalent sparse connection network of layers, each layer m is represented by the corresponding weight matrix U m Represented by , where missing connections are represented by zeros, so that the input connectivity does not exceed N i , and the output connectivity does not exceed N o . Equation U=Π m＝1..M U m is met with a predetermined accuracy.

[0037] In some implementations, the neural network topology includes a convolutional layer with K inputs and L outputs. In such a case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes decomposing the convolutional layer into a network with K inputs, L outputs, a maximum input connectivity P, and a maximum input connectivity P. i and the maximum output connectivity P o A single sparsely connected layer of P i ≤N i And P o ≤N o .

[0038] In some implementations, a weight matrix is ​​used to generate a schematic model for implementing an equivalent sparsely connected network.

[0039] In some implementations, the neural network topology includes a recurrent neural layer. In such a case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes transforming the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections.

[0040] In some implementations, the neural network topology includes a recurrent neural layer. In such a case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes decomposing the recurrent neural layer into several layers, wherein at least one of the layers is equivalent to a densely or sparsely connected layer having K inputs and L outputs and a weight matrix U, wherein missing connections are represented by zeros.

[0041] In some implementations, the neural network topology includes K inputs, a weight vector U∈R K and a single-layer perceptron with computational neurons having an activation function F. In such a case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving a connectivity N of the equivalent sparsely connected network based on one or more connectivity constraints; (ii) using equation Calculate the number of layers m of the equivalent sparsely connected network; and (iii) construct an equivalent sparsely connected network with K inputs, m layers and connectivity N. The equivalent sparsely connected network includes one or more corresponding simulated neurons in each of the m layers, each simulated neuron of the first m-1 layers implements the identity transformation, and the simulated neuron of the last layer implements the activation function F of the computational neuron of the single-layer perceptron. In addition, in such a case, calculating the weight matrix of the equivalent sparsely connected network includes calculating the weight vector W of the connection of the equivalent sparsely connected network by solving the system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S uses equations is calculated.

[0042] In some implementations, the neural network topology includes K inputs, a single-layer perceptron with L computational neurons, and a weight matrix V including a row of weights for each of the L computational neurons. In such a case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving a connectivity N of the equivalent sparsely connected network based on one or more connectivity constraints; (ii) using equation Calculate the number of layers m of the equivalent sparsely connected network; (iii) decompose the single-layer perceptron into L single-layer perceptron networks. Each single-layer perceptron network includes a corresponding computational neuron among the L computational neurons; (iv) for each single-layer perceptron network in the L single-layer perceptron networks: (a) construct a corresponding equivalent pyramidal subnetwork with K inputs, m layers and connectivity N for the corresponding single-layer perceptron network. The equivalent pyramidal subnetwork includes one or more corresponding simulated neurons in each of the m layers, each simulated neuron in the first m-1 layers implements the identity transformation, and the simulated neuron in the last layer implements the activation function corresponding to the corresponding computational neuron of the corresponding single-layer perceptron; and (b) construct an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork together (including connecting the inputs of each equivalent pyramidal subnetwork of the L single-layer perceptron networks together to form an input vector with L*K inputs). Furthermore, in such a case, computing the weight matrix of the equivalent sparsely connected network includes, for each of the L single-layer perceptron networks: (i) setting the weight vector U = V i , the i-th row of the weight matrix V corresponds to the corresponding computational neuron, and the corresponding computational neuron corresponds to the corresponding single-layer perceptron network; and (ii) calculating the weight vector W of the connection of the corresponding equivalent pyramidal subnetwork by solving the system of equations based on the weight vector U i The system of equations consists of K equations with S variables, and S uses equations is calculated.

[0043] In some implementations, the neural network topology includes a K-input, multilayer perceptron having S layers, where each layer i of the S layers includes a corresponding set of computational neurons L i And the corresponding weight matrix V i , weight matrix V i Including L i In this case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving the connectivity N of the equivalent sparsely connected network according to one or more connection constraints; (ii) decomposing the multilayer perceptron into Q = ∑ i＝1,S (L i ) single-layer perceptron networks. Each single-layer perceptron network includes a corresponding computational neuron among the Q computational neurons. Decomposing the multilayer perceptron includes duplicating one or more of the K inputs shared by the Q computational neurons; (iii) for each of the Q single-layer perceptron networks: (a) using equation Calculate the number of layers m of the corresponding equivalent pyramidal subnetwork. K i,jis the number of inputs to the corresponding computational neuron in the multilayer perceptron; and (b) constructing a K-threshold for the corresponding single-layer perceptron network i,j The equivalent pyramidal subnetwork has a number of inputs, m layers, and a connectivity of N. The equivalent pyramidal subnetwork includes one or more corresponding simulated neurons in each of the m layers, each simulated neuron in the first m-1 layers implements the identity transformation, and the simulated neuron in the last layer implements the activation function corresponding to the corresponding computational neuron of the corresponding single-layer perceptron network; and (iv) constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork together, including connecting the inputs of each equivalent pyramidal subnetwork of the Q single-layer perceptron networks together to form a sparsely connected network with Q*K i,j In addition, in such a case, calculating the weight matrix of the equivalent sparsely connected network includes: for each of the Q single-layer perceptron networks: (i) setting the weight vector U = V i j , the i-th row of the weight matrix V corresponds to the corresponding computational neuron, the corresponding computational neuron corresponds to the corresponding single-layer perceptron network, where j is the corresponding layer of the corresponding computational neuron in the multilayer perceptron; and (ii) calculating the weight vector W of the connection of the corresponding equivalent pyramidal subnetwork by solving the system of equations based on the weight vector U i The system of equations consists of K with S variables i,j equations, and S uses the equation is calculated.

[0044] In some implementations, the neural network topology includes a convolutional neural network (CNN) having K inputs, S layers, and each layer i of the S layers includes a set of corresponding computational neurons L i And the corresponding weight matrix V i , weight matrix V i Including for L i In this case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving the connectivity N of the equivalent sparsely connected network according to one or more connection constraints; (ii) decomposing the CNN into Q = ∑ i＝1，S (L i ) single-layer perceptron networks. Each single-layer perceptron network includes a corresponding computational neuron among the Q computational neurons. Decomposing the CNN includes replicating one or more of the K inputs shared by the Q computational neurons; (iii) for each of the Q single-layer perceptron networks: (a) using equation Calculate the number of layers m of the corresponding equivalent pyramidal subnetwork. j is the corresponding layer of the corresponding computational neuron in the CNN, and Ki,j is the number of inputs to the corresponding computational neuron in the CNN; and (b) constructing a K-threshold for the corresponding single-layer perceptron network i,j The equivalent pyramidal subnetwork has a number of inputs, m layers, and a connectivity of N. The equivalent pyramidal subnetwork includes one or more corresponding simulated neurons in each of the m layers, each simulated neuron in the first m-1 layers implements the identity transformation, and the simulated neuron in the last layer implements the activation function corresponding to the corresponding computational neuron of the corresponding single-layer perceptron network; and (iv) constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork together, including connecting the inputs of each equivalent pyramidal subnetwork of the Q single-layer perceptron networks together to form a sparsely connected network with Q*K i,j In addition, in such a case, calculating the weight matrix of the equivalent sparsely connected network includes: for each of the Q single-layer perceptron networks: (i) setting the weight vector U = V i j , the i-th row of the weight matrix V corresponds to the corresponding computational neuron, the corresponding computational neuron corresponds to the corresponding single-layer perceptron network, where j is the corresponding layer of the corresponding computational neuron in the CNN; and (ii) calculating the weight vector W of the connection of the corresponding equivalent pyramidal subnetwork by solving the system of equations based on the weight vector U i The system of equations consists of K with S variables i,j equations, and S uses the equation is calculated.

[0045] In some implementations, the neural network topology includes K inputs, a layer L with K neurons p , a layer L with L neurons n and the weight matrix W∈R L×K , where R is the set of real numbers, layer L p Each neuron is connected to layer L n Each neuron in layer L n Each neuron in layer L performs the activation function F, so that n Output of the equation Y using input x o =F(Wx) is calculated. In such a case, transforming the neural network topology into an equivalent sparsely connected network of analog components includes performing a trapezoidal transformation, which includes: (i) deriving possible input connectivity N according to one or more connection constraints I >1 and possible output connectivity N O >1; (ii) According to K·L <L·N I +K·N O Determine, construct a layer LA including K simulated neurons with the same activation functionp , with the same activation function Layer LA of simulated neurons h and a layer LA with L simulated neurons implementing the activation function F o The three-layer simulation network makes the LA p Each simulated neuron in has N O output, in layer LA h Each simulated neuron in has no more than N I input and N O outputs, and at layer LA o Each simulated neuron in has N I input. Furthermore, in such a case, computing the weight matrix of the equivalent sparsely connected network involves applying the matrix equation W o .W h =W is solved to generate the sparse weight matrix W o and W h , the matrix equation W o .W h =W includes the use of K·N O +L·N I K·L equations of variables, so that the layer LA o The total output of Y is calculated using the equation o =F(W o .W h .x) is calculated. The sparse weight matrix W o ∈R K×M Indicates that in layer LA p and LA h The connection between them, and the sparse weight matrix W h ∈R M×L Indicates that in layer LA h and LA o The connection between.

[0046] In some implementations, performing the trapezoidal transformation further comprises: according to K·L≥L·N I +K·N O Determination of: (i) Make layer L p Split to obtain a sublayer L with K' neurons p1 and a sublayer L with (K-K') neurons p2 , so that K′·L ≥ L·N I +K′·N O ; (ii) For a sublayer L with K' neurons p1 , performing the construction and generation steps; and (iii) for a sublayer L with K-K' neurons p2 , cyclically executing the split, build, and generate steps.

[0047] In some implementations, the neural network topology includes a multilayer perceptron network. In such a case, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing a trapezoidal transformation and calculating a weight matrix of an equivalent sparsely connected network.

[0048] In some implementations, the neural network topology includes a recurrent neural network (RNN) that includes (i) computation of a linear combination of two fully connected layers, (ii) element-wise addition, and (iii) nonlinear function computation. In such a case, the method also includes performing a trapezoidal transformation and computing a weight matrix of an equivalent sparsely connected network for (i) two fully connected layers, and (ii) nonlinear function computation.

[0049] In some implementations, the neural network topology includes a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) computation of linear combinations of multiple fully connected layers, (ii) element-wise addition, (iii) Hadamard product, and (iv) multiple nonlinear function computations. In such a case, the method also includes performing a trapezoidal transformation and computing a weight matrix of an equivalent sparsely connected network for (i) multiple fully connected layers, and (ii) multiple nonlinear function computations.

[0050] In some implementations, the neural network topology includes a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers and (ii) one or more fully connected layers. In such a case, the method further includes: (i) transforming the plurality of partially connected layers into equivalent fully connected layers by inserting dropout connections with zero weights; and (ii) for each consecutive pair of the equivalent fully connected layer and the one or more fully connected layers, iteratively performing a trapezoidal transformation and calculating a weight matrix of the equivalent sparsely connected network.

[0051] In some implementations, the neural network topology includes K input neurons, L output neurons, and a weight matrix U∈R L×K , where R is a set of real numbers, and each output neuron performs an activation function F. In such a case, transforming the neural network topology into an equivalent sparsely connected network of the analog component includes performing an approximate transformation, which includes: (i) deriving possible input connectivity N according to one or more connection constraints I >1 and possible output connectivity N O >1; (ii) from the set (iii) based on the determination of p>0, a pyramid neural network is constructed to form the first p layers of the equivalent sparsely connected network, so that the pyramid neural network has in its output layer Each neuron in the pyramid neural network performs the same function; and (iv) constructs a p Inputs and L outputs of the trapezoidal neural network. Each neuron in the last layer of the trapezoidal neural network performs the activation function F, and all other neurons perform the identity function. In this case, computing the weight matrix of the equivalent sparsely connected network includes: (i) generating the weights of the pyramid neural network, including (a) setting the weights of each neuron i in the first layer of the pyramid neural network according to the following rule: (a) C is a non-zero constant, and k i =(i-1)N I +1; and (b) for all k i All weights j of neurons other than and (b) setting all other weights of the pyramid neural network to 1; and (ii) generating the weights of the ladder neural network including (a) according to equation Setting the weight of each neuron i of the first layer of the ladder neural network; and (b) setting the other weights of the ladder neural network to 1.

[0052] In some implementations, the neural network topology includes a network having K inputs, S layers, and L in the i-th layer. i＝1,S computational neurons and the weight matrix for the i-th layer A multilayer perceptron, where L 0 = K. In this case, transforming the neural network topology into an equivalent sparsely connected network of analog components includes: for each layer j of the S layers of the multilayer perceptron: (i) by j-1 Input, L j output neurons and weight matrix U j The corresponding single-layer perceptron composed of PTNNX performs approximate transformation to construct the corresponding pyramid-ladder network PTNNX j ; and (ii) constructing an equivalent sparsely connected network by stacking each pyramid ladder network.

[0053] In another aspect, according to some implementations, a method for hardware implementation of a neural network is provided. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes calculating a weight matrix of the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection. The method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix and represents a resistance value.

[0054] In some implementations, generating a resistance matrix for a weight matrix includes: (i) obtaining a predetermined range of possible resistance values ​​{R min , R max} and select an initial base resistance value R within a predetermined range 基本 (ii) selecting a finite length set of resistance values ​​within a predetermined range, the resistance value for {R i ,R j} provides all combinations in the range [-R 基本 ,R 基本 ] possible weights within (iii) the maximum weight and bias w of the incoming connections of each neuron or equivalently each layer of the simulated network max , for each simulated neuron or for each layer of the equivalent simulated network, a resistor value R is selected from a finite length set of resistor values + =R - , so that R + =R - is closest to R 基本 *w max and (iv) for each element of the weight matrix, R within a predetermined range of possible resistance values 1 and R 2 Among all possible values ​​of , choose The corresponding first resistance value R that minimizes the error 1 and the corresponding second resistance value R 2 . w is the corresponding element of the weight matrix, and r err is the predetermined relative tolerance value of the resistor.

[0055] In some implementations, the predetermined range of possible resistance values ​​includes resistances according to the nominal series E24 in the range 100 KΩ to 1 MΩ.

[0056] In some implementations, R + and R - is chosen independently for each layer of the equivalent simulation network.

[0057] In some implementations, R + and R - is chosen independently for each simulated neuron of the equivalent simulated network.

[0058] In some implementations, the first one or more weights and the first one or more inputs of the weight matrix represent one or more connections to a first operational amplifier of the equivalent analog network. In such a case, the method further includes, before generating the resistor matrix: (i) modifying the first one or more weights by a first value; and (ii) configuring the first operational amplifier to multiply a linear combination of the first one or more weights and the first one or more inputs by the first value before performing the activation function.

[0059] In some implementations, the method further includes: (i) obtaining a predetermined range of weights; and (ii) updating the weight matrix according to the predetermined range of weights so that for the same input, the equivalent simulated network produces an output similar to the trained neural network.

[0060] In some implementations, the trained neural network is trained such that each layer of the neural network topology has quantized weights.

[0061] In some implementations, the method further includes retraining the trained neural network to reduce sensitivity to errors in weight or resistor values ​​that cause the equivalent analog network to produce different outputs compared to the trained neural network.

[0062] In some implementations, the method further includes retraining the trained neural network to minimize a weight in any layer that is greater than a predetermined threshold value for the average absolute weight of the layer.

[0063] In another aspect, according to some implementations, a method for hardware implementation of a neural network is provided. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology into an equivalent analog network of analog components including multiple operational amplifiers and multiple resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes calculating a weight matrix of the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection. The method also includes generating a resistor matrix for the weight matrix. Each element of the resistor matrix corresponds to a corresponding weight of the weight matrix. The method also includes pruning the equivalent analog network based on the resistor matrix to reduce the number of multiple operational amplifiers or multiple resistors to obtain an optimized analog network of analog components.

[0064] In some implementations, pruning the equivalent analog network includes replacing resistors corresponding to one or more elements of the resistor matrix having resistance values ​​below a predetermined minimum threshold resistance value with conductors.

[0065] In some implementations, pruning the equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that are above a predetermined maximum threshold resistance value.

[0066] In some implementations, pruning the equivalent simulated network includes removing one or more connections of the equivalent simulated network corresponding to one or more elements in the weight matrix that are approximately zero.

[0067] In some implementations, pruning the equivalent simulated network further includes removing one or more simulated neurons of the equivalent simulated network that do not have any input connections.

[0068] In some implementations, pruning the equivalent simulated network includes: (i) ranking simulated neurons of the equivalent simulated network based on detected use of the simulated neurons when performing calculations on one or more data sets; (ii) selecting one or more simulated neurons of the equivalent simulated network based on the ranking; and (iii) removing the one or more simulated neurons from the equivalent simulated network.

[0069] In some implementations, detecting the use of the simulated neurons includes: (i) using modeling software to build a model of the equivalent simulated network; and (ii) measuring propagation of the simulated signal by calculation using the model to generate one or more data sets.

[0070] In some implementations, detecting the use of the simulated neuron includes: (i) using modeling software to build a model of the equivalent simulated network; and (ii) measuring an output signal of the model by using the model to generate calculations of one or more data sets.

[0071] In some implementations, detecting usage of the simulated neurons includes: (i) using modeling software to build a model of the equivalent simulated network; and (ii) measuring power consumed by the simulated neurons by using the model to generate calculations of one or more data sets.

[0072] In some implementations, the method further includes recalculating a weight matrix of the equivalent analog network after pruning the equivalent analog network and before generating one or more photolithography masks for fabricating a circuit implementing the equivalent analog network, and updating the resistance matrix based on the recalculated weight matrix.

[0073] In some implementations, the method further includes: for each simulated neuron of the equivalent simulated network: (i) calculating a corresponding bias value of the corresponding simulated neuron based on the weights of the trained neural network while calculating the weight matrix; (ii) removing the corresponding simulated neuron from the equivalent simulated network based on a determination that the corresponding bias value is higher than a predetermined maximum bias threshold; and (iii) replacing the corresponding simulated neuron with a linear junction in the equivalent simulated network based on a determination that the corresponding bias value is lower than a predetermined minimum bias threshold.

[0074] In some implementations, the method further includes reducing the number of neurons of the equivalent simulated network by increasing the number of connections from one or more simulated neurons of the equivalent simulated network before generating the weight matrix.

[0075] In some implementations, the method further includes, before transforming the neural network topology, pruning the trained neural network using a neural network pruning technique to update the neural network topology and weights of the trained neural network so that the equivalent simulated network includes less than a predetermined number of simulated components.

[0076] In some implementations, pruning is performed iteratively, taking into account the accuracy or level of match in output between the trained neural network and the equivalent simulated network.

[0077] In some implementations, the method further includes performing network knowledge extraction before transforming the neural network topology into an equivalent simulated network.

[0078] In another aspect, according to some implementations, an integrated circuit is provided. The integrated circuit includes an analog network of analog components manufactured by a method, the method comprising: (i) obtaining a neural network topology and weights of a trained neural network; (ii) transforming the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents a corresponding analog neuron, and each resistor represents a corresponding connection between a corresponding first analog neuron and a corresponding second analog neuron; (iii) calculating a weight matrix of the equivalent analog network based on the weights of the trained neural network, each element of the weight matrix representing a corresponding connection; (iv) generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix; (v) generating one or more photolithography masks for manufacturing a circuit that implements the equivalent analog network of analog components based on the resistance matrix; and (vi) manufacturing the circuit based on the one or more photolithography masks using a photolithography process.

[0079] In some implementations, the integrated circuit also includes one or more digital-to-analog converters configured to generate analog inputs to the equivalent analog network of analog components based on the one or more digital signals.

[0080] In some implementations, the integrated circuit further includes an analog signal sampling module configured to process the one-dimensional or two-dimensional analog input at a sampling frequency based on a number of inferences of the integrated circuit.

[0081] In some implementations, the integrated circuit also includes a voltage converter module that scales down or scales up the analog signal to match an operating range of the plurality of operational amplifiers.

[0082] In some implementations, the integrated circuit also includes a tact signal processing module configured to process one or more frames obtained from the CCD camera.

[0083] In some implementations, the trained neural network is a long short-term memory (LSTM) network. In such a case, the integrated circuit also includes one or more clock modules to synchronize the signal haptics and allow time series processing.

[0084] In some implementations, the integrated circuit also includes one or more analog-to-digital converters configured to generate digital signals based on outputs of the equivalent analog network of analog components.

[0085] In some implementations, the integrated circuit also includes one or more signal processing modules configured to process the one-dimensional or two-dimensional analog signals obtained from the edge application.

[0086] In some implementations, a trained neural network is trained using a training data set containing signals from an array of gas sensors for different gas mixtures for selectively sensing different gases in a gas mixture containing a predetermined amount of gas to be detected. In such a case, the neural network topology is a one-dimensional deep convolutional neural network (1D-DCNN) designed to detect 3 binary gas components based on measurements made by 16 gas sensors, and includes 16 sensor-based 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks, and 3 dense layers. In such a case, the equivalent simulated network includes: (i) up to 100 input and output connections per simulated neuron, (ii) a delay block for generating delays of any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 simulated neurons, and (vi) approximately 4,900,000 connections.

[0087] In some implementations, a trained neural network is trained using a training dataset containing thermal aging time series data for different MOSFETs for predicting the remaining useful life (RUL) of MOSFET devices. In such a case, the neural network topology includes 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively. In such a case, the equivalent simulated network includes: (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 simulated neurons, and (v) between 123,000 and 124,000 connections.

[0088] In some implementations, a trained neural network is trained using a training dataset containing time series data including discharge and temperature data during continuous use of different commercially available Li-ion batteries for monitoring the state of health (SOH) and state of charge (SOC) of lithium-ion batteries for use in a battery management system (BMS). In such a case, the neural network topology includes an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values. In such a case, the equivalent simulated network includes: (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 simulated neurons, and (v) between 51,000 and 52,000 connections.

[0089] In some implementations, a trained neural network is trained using a training dataset containing time series data including discharge and temperature data during continuous use of different commercially available Li-ion batteries, which is used to monitor the state of health (SOH) of lithium-ion batteries for use in a battery management system (BMS). In such a case, the neural network topology includes an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron. In such a case, the equivalent simulated network includes: (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 simulated neurons, and (v) between 2,200 and 2,400 connections.

[0090] In some implementations, a trained neural network is trained using a training data set containing voice commands for recognizing voice commands. In such a case, the neural network topology is a deep separable convolutional neural network (DS-CNN) layer with 1 neuron. In such a case, the equivalent simulated network includes: (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 simulated neurons, and (v) approximately 2.6 million connections.

[0091] In some implementations, a trained neural network is trained using a training dataset containing photoplethysmography (PPG) data, accelerometer data, temperature data, and galvanic skin response signal data for different individuals performing various physical activities over a predetermined time period, and reference heart rate data obtained from an ECG sensor, for determining pulse rate during physical exercise based on the PPG sensor data and the 3-axis accelerometer data. In such a case, the neural network topology includes: two Conv1D layers, each with 16 filters and a kernel of 20 that perform time series convolution; two LSTM layers, each with 16 neurons; and two dense layers with 16 neurons and 1 neuron, respectively. In such a case, the equivalent simulated network includes: (i) a delay block that produces any number of time steps, (ii) a maximum of 100 input and output connections per simulated neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 simulated neurons, and (vi) between 12,000 and 12,500 connections.

[0092] In some implementations, the trained neural network is trained to classify different objects based on pulse Doppler radar signals. In such a case, the neural network topology includes a multi-scale LSTM neural network.

[0093] In some implementations, the trained neural network is trained to perform human activity type recognition based on inertial sensor data. In such a case, the neural network topology includes three channel-wise convolutional networks, each channel-wise convolutional network has a convolutional layer with 12 filters and a kernel size of 64, and each convolutional layer is followed by a max pooling layer, and two common dense layers with 1024 neurons and N neurons, respectively, where N is the number of categories. In such a case, the equivalent simulated network includes: (i) a delay block that produces any number of time steps, (ii) a maximum of 100 input and output connections per simulated neuron, (iii) an output layer of 10 simulated neurons, (iv) a signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 simulated neurons, and (vi) between 20,000 and 21,000 connections.

[0094] In some implementations, the trained neural network is further trained to detect abnormal patterns of human activity based on accelerometer data merged with heart rate data using a convolution operation.

[0095] In another aspect, a method for generating a library of hardware implementations of neural networks is provided. The method includes obtaining a plurality of neural network topologies, each neural network topology corresponding to a respective neural network. The method also includes transforming each neural network topology into a respective equivalent analog network of analog components. The method also includes generating a plurality of photolithography masks for fabricating a plurality of circuits, each circuit implementing a respective equivalent analog network of analog components.

[0096] In some implementations, the method further includes obtaining a new neural network topology and weights for the trained neural network. The method further includes selecting one or more lithography masks from a plurality of lithography masks based on a comparison of the new neural network topology with the plurality of neural network topologies. The method further includes calculating a weight matrix of a new equivalent analog network based on the weights. The method further includes generating a resistance matrix for the weight matrix. The method further includes generating a new lithography mask for manufacturing a circuit implementing the new equivalent analog network based on the resistance matrix and the one or more lithography masks.

[0097] In some implementations, the new neural network topology includes a plurality of subnetwork topologies, and selecting the one or more lithography masks is further based on comparing each subnetwork topology to each network topology in the plurality of network topologies.

[0098] In some implementations, one or more of the plurality of subnet topologies fail to compare with any of the plurality of network topologies. In such a case, the method further comprises: (i) transforming each of the one or more subnet topologies into a corresponding equivalent analog subnet of the analog component; and (ii) generating one or more photolithographic masks for fabricating one or more circuits, each of the one or more circuits implementing a corresponding equivalent analog subnet of the analog component.

[0099] In some implementations, transforming a corresponding network topology into a corresponding equivalent simulated network includes: (i) decomposing the corresponding network topology into multiple subnet topologies; (ii) transforming each subnet topology into a corresponding equivalent simulated subnet of simulated components; and (iii) combining each equivalent simulated subnet to obtain a corresponding equivalent simulated network.

[0100] In some implementations, decomposing the respective network topology includes identifying one or more layers of the respective network topology as a plurality of subnetwork topologies.

[0101] In some implementations, each circuit is obtained by: (i) generating schematics of corresponding equivalent analog networks of analog components; and (ii) generating corresponding circuit layout designs based on the schematics.

[0102] In some implementations, the method further includes combining the one or more circuit layout designs prior to generating a plurality of photolithography masks for fabricating the plurality of circuits.

[0103] In another aspect, according to some implementations, a method for optimizing the energy efficiency of an analog neuromorphic circuit is provided. The method includes obtaining an integrated circuit that implements an analog network of analog components, the analog components including a plurality of operational amplifiers and a plurality of resistors. The analog network represents a trained neural network, each operational amplifier represents a corresponding analog neuron, and each resistor represents a corresponding connection between a corresponding first analog neuron and a corresponding second analog neuron. The method also includes using the integrated circuit to generate inferences for a plurality of test inputs, including simultaneously transmitting signals from one layer of the analog network to the next layer. The method also includes, when using the integrated circuit to generate inferences: (i) determining whether the signal output levels of the plurality of operational amplifiers are balanced; and (ii) based on the determination that the signal output levels are balanced: (a) determining an active set of analog neurons of the analog network that affects signal formation for propagation of the signal; and within a predetermined time period, turning off power to one or more analog neurons of the analog network that are different from the active set of analog neurons.

[0104] In some implementations, the active set of simulated neurons is determined based on calculating delays in signal propagation through the simulated network.

[0105] In some implementations, the active set of simulated neurons is determined based on detecting propagation of the signal through the simulated network.

[0106] In some implementations, the trained neural network is a feed-forward neural network, and the active set of simulated neurons belongs to an active layer of the simulated network, and shutting down power includes shutting down power for one or more layers before the active layer of the simulated network.

[0107] In some implementations, the predetermined time period is calculated based on simulating propagation of the signal through a simulated network, taking into account signal delays.

[0108] In some implementations, the trained neural network is a recurrent neural network (RNN), and the analog network further includes one or more analog components in addition to the plurality of operational amplifiers and the plurality of resistors. In such a case, the method further includes, based on a determination that the signal output levels are balanced, shutting down power for the one or more analog components for a predetermined period of time.

[0109] In some implementations, the method further includes turning on power for one or more simulated neurons of the simulated network after the predetermined time period.

[0110] In some implementations, determining whether the signal output levels of the plurality of operational amplifiers are balanced is based on detecting whether one or more operational amplifiers of the analog network output a signal level greater than a predetermined threshold.

[0111] In some implementations, the method further includes repeatedly turning off the set of activities of the simulated neurons for a predetermined period of time and turning on the set of activities of the simulated neurons for a predetermined period of time while generating the inference.

[0112] In some implementations, the method further includes: (i) based on a determination that the signal output levels are balanced, for each inference cycle: (a) during a first time interval, determining a first layer of simulated neurons of a simulated network that affects signal formation for propagation of the signal; and (b) shutting down power to the first one or more simulated neurons of the simulated network before the first layer for a predetermined time period; and (ii) during a second time interval after the first time interval, shutting down power to the second one or more simulated neurons for a predetermined time period, the second one or more simulated neurons including the first layer of simulated neurons of the simulated network and the first one or more simulated neurons.

[0113] In some implementations, the one or more simulated neurons consist of simulated neurons of a first one or more layers of a simulated network, and the active set of simulated neurons consists of simulated neurons of a second layer of the simulated network, and the second layer of the simulated network is different from a layer in the first one or more layers.

[0114] In another aspect, a method for identifying human activities is provided. The method includes tracking the activities of a user using one or more sensors, including obtaining multiple electrical signals from the one or more sensors. The method also includes forming a feature vector by extracting multiple features from the multiple electrical signals. The features correspond to inputs to a neural network model, which is trained to generate multiple descriptors for multiple predefined human activities. The method also includes applying a simulated neural computing hardware device to the feature vector to generate an embedding vector for a specified descriptor. The simulated neural computing hardware device implements the trained neural network model. The method also includes applying a trained machine learning classifier to the embedding vector to classify the user's activity as one of the predefined human activities.

[0115] In some implementations, the trained neural network model is an autoencoder including an encoder and a decoder.

[0116] In some implementations, the trained machine learning classifier is a KNN (K nearest neighbor) classifier. In some implementations, the number of neighbors of the KNN classifier is equal to five. In some implementations, the trained machine learning classifier is trained separately for each of the predefined human activities using binary classification.

[0117] In some implementations, the one or more sensors include one or more of an IMU, a camera, a microphone, and a biofeedback device.

[0118] In some implementations, the method further includes smoothing the output of the trained machine learning classifier to obtain the underlying categories of activities.

[0119] In some implementations, an analog neural computing hardware device is manufactured by steps including: obtaining a neural network topology and weights of a trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; calculating a weight matrix of the equivalent analog network based on the weights of the trained neural network model, wherein each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components; and manufacturing an integrated circuit according to the schematic model using a photolithography process.

[0120] In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.

[0121] In some implementations, a trained machine learning classifier is implemented using one or more digital components, and the trained machine learning classifier can be retrained for a new user.

[0122] In another aspect, a method for identifying human activities is provided. The method includes obtaining an electrical signal sequence from one or more sensors that track user activities. The method also includes forming a plurality of feature vectors by extracting features from the electrical signal sequence. The features correspond to inputs to a neural network model that is trained to generate a plurality of descriptors for a plurality of predefined human activities. The method also includes applying a simulated neural computing hardware device to the plurality of feature vectors to generate a plurality of embedding vectors, each embedding vector specifying a corresponding descriptor. The method also includes using the plurality of embedding vectors for classifying the user's activity as one of the predefined human activities.

[0123] In some implementations, the method further includes: receiving a set of descriptors describing a specific physical activity from the user; and classifying the user's activity as one of the specific physical activities using the set of descriptors and the plurality of embedding vectors. In some implementations, the method further includes generating statistics of the user's personal daily routine based on classifying the user's activity as one of the specific physical activities.

[0124] In some implementations, the method further includes: storing, for the user, a plurality of embedding vectors as describing a specific activity; and classifying a subsequent activity of the user as the specific activity using the plurality of embedding vectors.

[0125] In some implementations, the method further includes: receiving a set of descriptors describing the specific activity from a trainer different from the user; and providing feedback to the user if the activity matches the specific activity based on the plurality of embedding vectors and the set of descriptors.

[0126] In another aspect, a human activity recognition device is provided. The device includes an integrated circuit for human activity recognition. The integrated circuit includes an analog network of analog components, which are configured to implement a trained neural network model, which is trained to generate multiple descriptors for multiple predefined human activities based on multiple features extracted from multiple electrical signals from one or more sensors. The device also includes one or more digital components, which are configured to classify the human activity into one of the multiple predefined human activities according to the multiple descriptors generated by the integrated circuit.

[0127] In some implementations, the human activity recognition device also includes one or more sensors configured to collect a plurality of electrical signals during the human activity.

[0128] In some implementations, the trained neural network model is an autoencoder including an encoder and a decoder.

[0129] In some implementations, one or more digital components implement a trained machine learning classifier that is a KNN (K Nearest Neighbors) classifier that can be retrained.

[0130] In some implementations, the number of neighbors for the KNN classifier is equal to five.

[0131] In some implementations, the trained machine learning classifier is trained separately for each of a plurality of predefined human activities using binary classification.

[0132] In some implementations, the one or more digital components are further configured to smooth the output of the trained machine learning classifier to obtain the underlying categories of activity.

[0133] In some implementations, the one or more sensors include one or more of an IMU, a camera, a microphone, and a biofeedback device.

[0134] In some implementations, the integrated circuit is manufactured by steps including: obtaining a neural network topology and weights of a trained neural network model; transforming the neural network topology into an equivalent analog network of analog components; calculating a weight matrix of the equivalent analog network based on the weights of the trained neural network model. Each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components; and manufacturing the integrated circuit according to the schematic model using a photolithography process. In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix and represents a resistance value.

[0135] In another aspect, a system for predictive maintenance of a machine is provided according to some implementations. The system may include a hardware device. The hardware device may include a vibration sensor configured to sense vibration of a vibration source and an analog circuit including a plurality of operational amplifiers and a plurality of resistors. The analog circuit may be coupled to the vibration sensor and configured to (i) receive an analog signal from the vibration sensor and (ii) calculate an output based on the analog signal by executing a portion of a trained neural network.

[0136] In some implementations, the hardware device also includes a transceiver coupled to the analog circuit and configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN).

[0137] In some implementations, the vibration sensor is disposed adjacent to a movable component (e.g., a rotating or reciprocating component) of a machine (sometimes referred to as a vibration source), and the vibration sensor is configured to collect vibration signals from the rotating or reciprocating component. In some implementations, the rotating component includes a ball bearing of the machine.

[0138] In some implementations, the output of the simulation circuit represents an embedding for at least one of defining a vibration source, predicting a failure of a machine coupled to the vibration source, and generating a recommendation for maintaining the machine.

[0139] In some implementations, the vibration sensor is disposed in or on a tire and is configured to collect vibration data of the tire, which is an example of a vibration source.

[0140] In some implementations, the output represents an embedding for predicting at least one of: a road surface, a physical condition, a tire condition, a suspension system condition, or a time-to-failure of a vibration source.

[0141] In some implementations, the vibration sensor is configured to sample signals in the range of 0 to 20 kilohertz (kHz).

[0142] In some implementations, the vibration sensor is configured to sample signals up to 41 kHz.

[0143] In some implementations, the vibration sensor is configured to sample signals below the Nyquist sampling rate for fault detection and classification in the compressed domain.

[0144] In some implementations, a vibration sensor is configured to sample a signal for compression sensing (CS) for condition classification of a rolling element support in a rotating machine.

[0145] In some implementations, the trained neural network includes multiple neuron layers, the multiple neuron layers include a first set of layers and a second set of layers, and the analog circuit is configured to implement the first set of layers. In some implementations, the second set of layers consists of the last layer of the trained neural network.

[0146] In some implementations, the trained neural network includes a deep neural network for unsupervised learning based on a sparse autoencoder.

[0147] In some implementations, the trained neural network includes a ResNet convolutional neural network (CNN) with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings.

[0148] In some implementations, the vibration sensor is configured to sample and output a one-dimensional time domain signal of the rolling bearing fault signal.

[0149] In some implementations, the trained neural network includes stacked denoising autoencoders.

[0150] In some implementations, the analog circuit is configured to be powered by vibrations from the vibration source.

[0151] In some implementations, the hardware device also includes a power harvesting circuit configured to harvest power from the vibration of the vibration source and supply the power to the analog circuit.

[0152] In some implementations, the plurality of operational amplifiers are configured to implement neurons of a portion of a trained neural network, and the plurality of resistors are configured to implement axons or connections between neurons of a portion of the trained neural network.

[0153] In some implementations, the analog circuit is configured to implement an optimized neural network corresponding to the trained neural network.

[0154] In some implementations, the values ​​of the plurality of resistors are based on weights of the connections of the trained neural network.

[0155] In some implementations, the plurality of resistors are configured to connect to the plurality of operational amplifiers.

[0156] In some implementations, the analog circuit includes a resistor in backend-of-the-line (BEOL).

[0157] In some implementations, the trained neural network is an autoencoder including an encoder portion and a decoder portion. The encoder portion reconstructs the input vector at the output layer after a nonlinear transformation performed by a hidden layer. The analog circuit corresponds to the encoder portion of the autoencoder. The encoder portion includes a hidden layer. The analog circuit is configured to calculate a representation of the input vector with fewer dimensions than the input space of the input vector.

[0158] In some implementations, the analog circuit is configured to generate compressed data that encodes the vibration sensor data based on the vibration signature from the vibration sensor.

[0159] In another aspect, a system includes a hardware device, a transceiver, and a digital circuit. The hardware device includes a vibration sensor configured to sense vibration of a vibration source of a machine. The hardware device also includes an analog circuit, which includes a plurality of operational amplifiers and a plurality of resistors. The analog circuit is coupled to the vibration sensor and is configured to: receive an analog signal from the vibration sensor; and calculate an output based on the analog signal by executing a portion of a trained neural network. The transceiver is coupled to the analog circuit and is configured to receive an output from the analog circuit and transmit the output via a low power wide area network (LPWAN). The analog signal is converted before transmission. An analog-to-digital converter (ADC) may be included in the hardware device, or may be included as part of the transceiver. The digital circuit is communicatively coupled to the transceiver of the hardware device via the LPWAN. The digital circuit is configured to receive an output from the analog circuit (via the transceiver) and predict a state of the machine for maintenance based on the output.

[0160] In some implementations, the digital circuit includes one or more digital computing units selected from the group consisting of: a CPU, a GPU, a RISC, a FPGA, and an ASIC.

[0161] In some implementations, the digital circuit includes a processor configured to perform data classification. In some implementations, the data classification is performed by a neural network different from the trained neural network. In some implementations, the data classification is performed using k-nearest neighbors (k-NN).

[0162] In some implementations, the output of the analog circuit represents an embedding, and the digital circuit is configured to classify the analog signal using the embedding.

[0163] In some implementations, a method for vibration sensing is provided. The method includes sensing vibration of a vibration source using a vibration sensor to obtain an analog signal. The method also includes calculating an output based on the analog signal by executing a portion of a trained neural network using an analog circuit including a plurality of operational amplifiers and a plurality of resistors. The method also includes transmitting the output via a low power wide area network (LPWAN) using a transceiver.

[0164] In some implementations, a computer system has one or more processors, a memory, and a display.One or more programs include instructions for performing any of the methods described herein.

[0165] In some implementations, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by a computer system having one or more processors, a memory, and a display. The one or more programs include instructions for executing any of the methods described herein.

[0166] Thus, methods, systems, and apparatus for hardware implementation of trained neural networks are disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0168] For a better understanding of the aforementioned systems, methods, and graphical user interfaces, as well as additional systems, methods, and graphical user interfaces that provide data visualization analytics and data preparation, reference should be made to the following description of implementations in conjunction with the accompanying drawings, in which like reference numerals refer to corresponding parts throughout the drawings.

[0169] Figure 1A is a block diagram of a system for hardware implementation of a trained neural network using analog components according to some implementations. Figure 1B According to some implementations, a hardware implementation of a trained neural network using analog components is provided. Figure 1A Block diagram of an alternative representation of the system. Figure 1C According to some implementations, a hardware implementation of a trained neural network using analog components is provided. Figure 1A A block diagram of another representation of a system.

[0170] Figure 2A is a system diagram of a computing device according to some implementations. Figure 2B Optional modules of a computing device according to some implementations are shown.

[0171] Figure 3A An example process for generating a schematic model of a simulated network corresponding to a trained neural network according to some implementations is shown. Figure 3B An example manual prototyping process for generating a target chip model according to some implementations is shown.

[0172] Figure 4A , Figure 4B and Figure 4C Shown are examples of neural networks transformed into mathematically equivalent analog networks according to some implementations.

[0173] Figure 5 Examples of mathematical models of neurons according to some implementations are shown.

[0174] Figure 6A-6C An example process for simulating a hardware implementation of a neural network for computing an exclusive OR (XOR) of input values ​​according to some implementations is shown.

[0175] Figure 7 An example perceptron according to some implementations is shown.

[0176] Figure 8 An example pyramid neural network is shown according to some implementations.

[0177] Fig. 9 An example pyramid single neural network is shown according to some implementations.

[0178] Fig.10 Examples of transformed neural networks according to some implementations are shown.

[0179] Figures 11A-11C An application of the T-transform algorithm to a single-layer neural network according to some implementations is shown.

[0180] Fig.12 An example recurrent neural network (RNN) is shown according to some implementations.

[0181] Fig.13A is a block diagram of an LSTM neuron according to some implementations.

[0182] Fig. 13B Delay blocks according to some implementations are shown.

[0183] Fig. 13C It is a neuron schema based on some implemented LSTM neurons.

[0184] Fig.14A is a block diagram of a GRU neuron according to some implementations.

[0185] Fig. 14B is the neuron schema of a GRU neuron according to some implementations.

[0186] Fig.15A and Fig. 15B is a neuron diagram of a variation of a single Conv1D filter according to some implementations.

[0187] Fig.16 An example architecture of a transformed neural network according to some implementations is shown.

[0188] Figure 17A-Figure 17C Example graphs illustrating correlations between output error and classification error or weight error according to some implementations are provided.

[0189] Fig.18 Example schemes of neuron models for resistor quantization according to some implementations are provided.

[0190] Fig.19A A schematic diagram of an operational amplifier fabricated on CMOS according to some implementations is shown. Fig.19B Shown according to some implementations Fig.19A A table describing the example circuits is shown.

[0191] Figure 20A-Figure 20E A schematic diagram of an LSTM block according to some implementations is shown. Fig.20F Shown according to some implementations Figures 20A-20D A table describing the example circuits is shown.

[0192] Figure 21A-Figure 21I A schematic diagram of a multiplier block according to some implementations is shown. Fig.21J Shown according to some implementations Figure 21A-Figure 21I Table of descriptions of the schematic diagrams shown.

[0193] Fig.22A A schematic diagram of a sigmoid neuron according to some implementations is shown. Fig. 22B Shown according to some implementations Fig.22A Table of descriptions of the schematic diagrams shown.

[0194] Fig.23A A schematic diagram of a hyperbolic tangent function block is shown according to some implementations. Fig. 23B Shown according to some implementations Fig.23A Table of descriptions of the schematic diagrams shown.

[0195] Figure 24A-24C A schematic diagram of a single-neuron CMOS operational amplifier is shown according to some implementations. Fig.24D Shown according to some implementations Figure 24A-24C Table of descriptions of the schematic diagrams shown.

[0196] Figure 25A-Figure 25D Schematic diagram showing a variation of a single-neuron CMOS operational amplifier according to some implementations. Fig.25E Shown according to some implementations Figure 25A-Figure 25D Table of descriptions of the schematic diagrams shown.

[0197] Figure 26A-26K Example weight distribution histograms are shown according to some implementations.

[0198] Figure 27A-27J A flow chart of a method for hardware implementation of a neural network according to some implementations is shown.

[0199] Figure 28A-28S A flow chart of a method for hardware implementation of a neural network according to hardware design constraints is shown according to some implementations.

[0200] Figure 29A-29F A flow chart of a method for hardware implementation of a neural network according to hardware design constraints is shown according to some implementations.

[0201] Figure 30A-30M A flow chart of a method for hardware implementation of a neural network according to hardware design constraints is shown according to some implementations.

[0202] Figure 31A-Figure 31Q A flow chart of a method for fabricating an integrated circuit including an analog network of analog components is shown, according to some implementations.

[0203] Figure 32A-Figure 32E A flow diagram of a method for generating a library of hardware implementations of a neural network according to some implementations is shown.

[0204] Figure 33A-Figure 33K A flow chart of a method for optimizing energy efficiency of an analog neuromorphic circuit that models a trained neural network is shown in accordance with some implementations.

[0205] Fig.34 A table describing the MobileNet v1 architecture according to some implementations is shown.

[0206] Fig.35 An example 1D convolutional neural network for voice clarification is shown in accordance with some implementations.

[0207] Fig.36 Shown according to some implementations Fig.35 An example T transformation of a fully connected layer of the neural network shown.

[0208] Fig.37 Shown according to some implementations Fig.35 An example of a 1D convolution for the neural network shown in T-Transform.

[0209] Fig.38 Shown according to some implementations Fig.35 An example of the max pooling operator for the neural network shown in T transform.

[0210] Fig.39 An example of partitioning a dataset into a training dataset and a validation dataset according to some implementations is shown.

[0211] Fig.40 Two graphs of activity tags are shown according to some implementations.

[0212] Fig.41A and Fig.41B An example graph of test data for partitioning an experiment into training or testing is shown according to some implementations.

[0213] Fig.42A , Fig.42B and Fig.42C Schematic diagrams of an example encoder, an example decoder, and an example classifier are respectively shown according to some implementations.

[0214] Figure 43A-43F Graphs showing results according to some implementations.

[0215] Figure 44A-Figure 44D Graphs showing results according to some implementations.

[0216] Fig.45A and Fig.45B A table presenting predictions for a base class activity classifier model according to some implementations is shown.

[0217] Figure 46A-46F Graphs showing results according to some implementations.

[0218] Figure 47A-Figure 47F Graphs showing results according to some implementations.

[0219] Figure 49A-49D Graphs showing results according to some implementations.

[0220] Fig.48 According to some implementations, a method for Fig.47E and Fig.47F Visualization of the t-Distributed Stochastic Neighbor Embedding (t-SNE) of the exercise set for different ways of embedding the Crossfit sit-up exercise in the experiment shown.

[0221] Fig.50 An example human activity recognition device according to some implementations is shown.

[0222] Fig.51 A flow chart of an example method for identifying human activities according to some implementations is shown.

[0223] Fig.52 A flow chart of another example method for identifying human activities according to some implementations is shown.

[0224] Fig.53 An example sparse autoencoder for support fault classification according to some implementations is shown.

[0225] Fig.54 An example deep ResNet framework with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings according to some implementations is shown.

[0226] Fig.55 A flow chart illustrating an example method for deep autoencoder training for support fault classification according to some implementations.

[0227] Fig.56 An example system for predictive maintenance of a machine is shown according to some implementations.

[0228] Reference will now be made to implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention can be practiced without these specific details.

[0229] Description of the implementation

[0230] Figure 1Ais a block diagram of a system 100 for hardware implementation of a trained neural network using analog components according to some implementations. The system includes transforming (126) a trained neural network 102 into a simulated neural network 104. In some implementations, analog integrated circuit constraints 184 constraints (146) are transformed (126) to generate the simulated neural network 104. Subsequently, the system derives (calculates or generates) weights 106 of the simulated neural network 104 through a process sometimes referred to as weight quantization (128). In some implementations, the simulated neural network includes a plurality of simulated neurons, each simulated neuron represented by an analog component (e.g., an operational amplifier), and each simulated neuron is connected to another simulated neuron via a connection. In some implementations, a connection is represented using a resistor that reduces current between two simulated neurons. In some implementations, the system transforms (148) the weights 106 into resistance values ​​112 for the connection. The system then generates (130) one or more schematic models 108 for implementing the simulated neural network 104 based on the weights 106. In some implementations, the system optimizes the resistance values ​​112 (or weights 106) to form an optimized simulated neural network 114, which is further used to generate (150) the schematic model 108. In some implementations, the system generates (132) a lithography mask 110 for the connections and / or generates (136) a lithography mask 120 for the simulated neurons. In some implementations, the system fabricates (134 and / or 138) an analog integrated circuit 118 that implements the simulated neural network 104. In some implementations, the system generates (152) a lithography mask library 116 based on the lithography mask 110 for the connections and / or the lithography mask 120 for the simulated neurons. In some implementations, the system fabricates (154) the lithography mask library 116 to fabricate the analog integrated circuit 118. In some implementations, when the trained neural network 142 is retrained (142), the system regenerates (or recalculates) (144) the resistance values ​​112 (and / or weights 106), the schematic model 108, and / or the lithographic mask 110 for the connections. In some implementations, the system reuses the lithographic mask 120 for the simulated neurons 120. In other words, in some implementations, only the weights 106 (or the resistance values ​​112 corresponding to the changed weights) and / or the lithographic mask 110 for the connections are regenerated. Because only the connections, weights, the schematic model, and / or the corresponding lithographic mask for the connections are regenerated, as indicated by the dashed line 156, the process for manufacturing (or the path to) the analog integrated circuit for the retrained neural network is significantly simplified when compared to conventional techniques for hardware implementation of neural networks, and the time to market for hardware for re-spinning the neural network is reduced.

[0231] Figure 1Bis a block diagram of an alternative representation of a system 100 for hardware implementation of a trained neural network using analog components according to some implementations. The system includes training (156) the neural network in software, determining weights of connections, generating (158) an electronic circuit equivalent to the neural network, calculating (160) resistor values ​​corresponding to the weights of each connection, and then generating (162) a photolithography mask using the resistor values.

[0232] Figure 1C is a block diagram of another representation of a system 100 for hardware implementation of a trained neural network using analog components according to some implementations. According to some implementations, the system is distributed as a software development kit (SDK) 180. A user develops and trains (164) a neural network and inputs the trained neural network 166 to the SDK 180. The SDK estimates (168) the complexity of the trained neural network 166. If the complexity of the trained neural network can be reduced (e.g., some connections and / or neurons can be removed, some layers can be reduced, or the density of neurons can be changed), the SDK 180 prunes (178) the trained neural network and retrains (182) the neural network to obtain an updated trained neural network 166. Once the complexity of the trained neural network is reduced, the SDK 180 transforms (170) the trained neural network 166 into a sparse network of analog components (e.g., a pyramidal or ladder-shaped network). The SDK 180 also generates a circuit model 172 of the analog network. In some implementations, the SDK uses software simulation to estimate (176) the deviation of the output generated by the circuit model 172 relative to the trained neural network for the same inputs. If the estimated error exceeds a threshold error (e.g., a value set by the user), the SDK 180 prompts the user to reconfigure, redevelop, and / or retrain the neural network. In some implementations, although not shown, the SDK automatically reconfigures the trained neural network 166 to reduce the estimated error. This process is repeated multiple times until the error is reduced to less than the threshold error. Figure 1C , the dashed line from block 176 ("Estimation of errors occurring in the circuit system") to block 164 ("Development and training of the neural network") indicates a feedback loop. For example, if the pruned network does not show the desired accuracy, some implementations prune the network in different ways until the accuracy exceeds a predetermined threshold (e.g., 98% accuracy) for a given application. In some implementations, this process includes recalculating weights because pruning includes retraining of the entire network.

[0233] In some implementations, the components of the system 100 described above are implemented as computing modules in one or more computing devices or server systems. Figure 2Ais a block diagram of a computing device 200 according to some implementations. As used herein, the term "computing device" includes both personal devices 102 and servers. The computing device 200 typically includes one or more processing units / cores (CPUs) 202 for executing modules, programs and / or instructions stored in a memory 214 and thereby performing processing operations; one or more network or other communication interfaces 204; memory 214; and one or more communication buses 212 for interconnecting these components. The communication bus 212 may include circuitry for interconnecting and controlling communications between system components. The computing device 200 may include a user interface 206, which includes a display device 208 and one or more input devices or mechanisms 210. In some implementations, the input device / mechanism 210 includes a keyboard; in some implementations, the input device / mechanism includes a "soft" keyboard that is displayed on the display device 208 as needed, enabling a user to "press" a "key" that appears on the display 208. In some implementations, the display 208 and the input device / mechanism 210 include a touch screen display (also referred to as a touch-sensitive display). In some implementations, the memory 214 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, the memory 214 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, the memory 214 includes one or more storage devices located away from the CPU 202. The memory 214 or alternatively a non-volatile memory device within the memory 214 includes a computer-readable storage medium. In some implementations, the memory 214 or a computer-readable storage medium of the memory 214 stores the following programs, modules, and data structures or a subset thereof:

[0234] operating system 216, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0235] A communication module 218 for connecting the computing device 200 to other computers and devices via one or more communication network interfaces 204 (wired or wireless) and one or more communication networks (e.g., the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0236] A trained neural network 220 including weights 222 and a neural network topology 224. The following describes references according to some implementations. Figure 4A-4C , Fig.12 , Fig.13A and Fig.14A An example of input to a neural network;

[0237] Neural network transformation module 226 includes a transformed simulated neural network 228, mathematical formulas 230, basic function blocks 232, simulation models 234 (sometimes referred to as neuron models), and / or simulated integrated circuit (IC) design constraints 236. Figure 5 , Figure 6A-6C , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Figures 11A-11C and in Figure 27A-27J and Figure 28A-28S , and / or

[0238] A weight matrix calculation (sometimes referred to as weight quantization) module 238 includes the weights 272 of the transformed network and optionally includes a resistance calculation module 240, resistance values ​​242. According to some implementations, at least reference Figure 17A-17C , Fig.18 and Figure 29A-29F hereinafter is used to describe example operations of the weight matrix calculation module 238 and / or weight quantization.

[0239] Some implementations include Figure 2B One or more optional modules 244 are shown. Some implementations include a simulated neural network optimization module 246. According to some implementations, the following reference Figure 30A-30M Describe an example of simulated neural network optimization.

[0240] Some implementations include a lithography mask generation module 248, which further includes a lithography mask 250 for resistors (corresponding to connections) and / or a lithography mask for analog components (e.g., operational amplifiers, multipliers, delay blocks, etc.) instead of resistors (or connections). In some implementations, a Cadence, Synopsys, or Mentor Graphics software package is used to generate a lithography mask based on a chip design layout after chip design. Some implementations use a design kit from a silicon wafer manufacturing plant (sometimes referred to as a fab). The lithography mask is intended to be used in that particular fab that provides a design kit (e.g., a TSMC 65nm design kit). The generated lithography mask file is used to manufacture chips at the fab. In some implementations, a chip design based on a Cadence, Mentor Graphics, or Synopsys software package is semi-automatically generated from a SPICE or Fast SPICE (Mentor Graphics) software package. In some implementations, a user with chip design skills drives the conversion from SPICE or Fast SPICE circuits to Cadence, Mentor Graphics, or Synopsis chip designs. Some implementations combine Cadence design blocks of individual neuron units, making appropriate interconnections between the blocks.

[0241] Some implementations include a library generation module 254, which further includes a lithography mask library 256. According to some implementations, the following reference Figure 32A-Figure 32E Describes examples of library generation.

[0242] Some implementations include an integrated circuit (IC) manufacturing module 258, which further includes an analog-to-digital converter (ADC), a digital-to-analog converter (DAC), or other similar interface 260 and / or a manufactured IC or model 262. According to some implementations, the following reference Figure 31A-Figure 31Q Example integrated circuits and / or related modules are described.

[0243] Some implementations include an energy efficiency optimization module 264, which further includes an inference module 266, a signal monitoring module 268, and / or a power optimization module 270. According to some implementations, the following reference Figure 33A-Figure 33K Describe examples of energy efficiency optimization.

[0244] Each of the executable modules, applications or process concentrations identified above can be stored in one or more of the aforementioned memory devices and corresponds to an instruction set for performing the above functions. The modules or programs (i.e., instruction sets) identified above do not need to be implemented as separate software programs, processes or modules, so the various subsets of these modules can be combined or otherwise rearranged in various implementations. In some implementations, memory 214 stores a subset of the modules and data structures identified above. In addition, in some implementations, memory 214 stores additional modules or data structures not described above.

[0245] although Figure 2A Computing device 200 is shown, but Figure 2A It is intended more as a functional description of various features that may be present rather than as a schematic diagram of the architecture of the implementations described herein. In practice and as recognized by one of ordinary skill in the art, items shown separately may be combined and some items may be separated.

[0246] Example procedure for generating a schematic model of a simulation network

[0247] Figure 3A An example process 300 for generating a schematic model of a simulated network corresponding to a trained neural network is shown according to some implementations. Figure 3A As shown, a trained neural network 302 (e.g., MobileNet) is transformed (322) into a target or equivalent analog network 304 (using a process sometimes referred to as a T-transform). The target neural network (sometimes referred to as a T-network) 304 is exported (324) to SPICE using a single neuron model (SNM) (as a SPICE model 306), which is exported (326) from SPICE to a CADENCE and full-on-chip design using a CADENCE model 308. The CADENCE model 308 is cross-validated (328) against the initial neural network for one or more validation inputs.

[0248] In the above and following descriptions, a mathematical neuron is a mathematical function that receives one or more weighted inputs and produces a scalar output. In some implementations, a mathematical neuron can have a memory (e.g., long short-term memory (LSTM), recurrent neuron). A trivial neuron is a mathematical neuron that performs a function, representing an "ideal" mathematical neuron. where f(x) is the activation function. SNM is a mathematical representation of a particular type of neuron (e.g., a trivial neuron) using analog components (e.g., op amps, resistors R 1 , …, R nThe SNM output voltage is represented by the corresponding formula, which depends on the K input voltages and the SNM component values. According to some implementations, with appropriately selected component values, the SNM formula is equivalent to a mathematical neuron formula with a desired set of weights. In some implementations, the weight set is completely determined by the resistors used in the SNM. The target (simulated) neural network 304 (sometimes referred to as a T-network) is a set of mathematical neurons that define the SNM representation and weight the connections between them to form a neural network. The T-network follows several restrictions, such as inbound restrictions (maximum restrictions on inbound connections of any neuron within the T-network), outbound restrictions (maximum restrictions on outbound connections of any neuron within the T-network), and signal ranges (e.g., all signals should be within a predefined signal range). The T-transformation (322) is the process of converting a desired neural network (e.g., MobileNet) into a corresponding T-network. The SPICE model 306 is a SPICE neural network model of the T-network 304, in which each mathematical neuron is replaced by a corresponding one or more SNMs. The Cadence NN model 310 is a Cadence model of the T-network 304, in which each mathematical neuron is replaced by a corresponding one or more SNMs. In addition, as described herein, if for all neuron outputs of these networks Then the two networks L and M have mathematical equivalence, where eps is relatively small (e.g., between 0.1%-1% of the operating voltage range). Similarly, if for a given verification input data set {I 1 ,…,I n}, the classification results are mostly the same, that is, P(L(I k )=M(I k ))=1-eps, then the two networks L and M have functional equivalence, where eps is relatively small.

[0249] Figure 3BAn example manual prototyping process for generating a target chip model 320 based on an SNM model 314 on Cadence according to some implementations is shown. Note that although the following description uses Cadence, according to some implementations, alternative tools from Mentor Graphic Design or Synopsys (e.g., Synopsys Design Suite) can be used instead of the Cadence tool. The process includes selecting SNM constraints, including inbound and outbound constraints and signal constraints, selecting analog components (e.g., resistors, including specific resistor array technologies) for connections between neurons, and developing a Cadence SNM model 314. A prototype SNM model 316 (e.g., a PCB prototype) is developed (330) based on the SNM model 314 on Cadence. The prototype SNM model 316 is compared to the SPICE model for equivalence. In some implementations, the neural network is selected for the on-chip prototype when the neural network meets the equivalence requirement. Because the neural network size is small, the T transform can be manually verified for equivalence. Subsequently, an on-chip SNM model 318 is generated (332) based on the SNM model prototype 316. According to some implementations, the on-chip SNM model is optimized as much as possible. In some implementations, after finalizing the SNM model, the on-chip density of the SNM model is calculated before generating (334) the target chip model 320 based on the on-chip SNM model 318. During the prototyping process, the practitioner can iteratively select a neural network task or application and a specific neural network (e.g., a neural network with approximately 100,000 to 1.1 million neurons), perform a T-transformation, build a Cadence neural network model, and design an interface and / or a target chip model.

[0250] Example input to a neural network

[0251] Figure 4A , Figure 4B and Figure 4C An example of a trained neural network (eg, neural network 220) is shown, according to some implementations, which is input to system 100 and transformed into a mathematically equivalent analog network. Figure 4A An example neural network (sometimes referred to as an artificial neural network) is shown, which is composed of artificial neurons that receive inputs, combine the inputs using an activation function, and produce one or more outputs. The inputs include data, such as images, sensor data, and documents. In general, each neural network performs a specific task, such as object recognition. The network includes connections between neurons, and each connection provides the output of a neuron as the input of another neuron. After training, each connection is assigned a corresponding weight. As shown in FIG. Figure 4AAs shown, neurons are usually organized into multiple layers, and each layer of neurons is only connected to the neurons in the immediately previous and next layers. The input layer 402 of the neurons receives external input (e.g., input X 1 , X 2 , …, X n ). Input layer 402 is followed by one or more hidden layers of neurons (e.g., layers 404 and 406), followed by output layer 408 that produces output 410. Various types of connection patterns connect neurons in successive layers, such as a fully connected pattern that connects each neuron in one layer to all neurons in the next layer or a pooled pattern that connects the output of a group of neurons in one layer to a single neuron in the next layer. Figure 4A In contrast to the neural network shown, which is sometimes called a feedforward network, Figure 4B The neural network shown includes one or more connections from neurons in a layer to other neurons in the same layer or to neurons in a previous layer. Figure 4B The example shown in is an example of a recurrent neural network and includes two input neurons 412 (which accepts input X1) and 414 (which accepts input X2) in the input layer, followed by two hidden layers. The first hidden layer includes neurons 416 and 418, which are fully connected to neurons in the input layer and neurons 420, 422, and 424 in the second hidden layer. The output of neuron 420 in the second hidden layer is connected to neuron 416 in the first hidden layer, providing a feedback loop. The hidden layer including neurons 420, 422, and 424 is input to a neuron 426 in the output layer that produces an output y.

[0252] Figure 4C An example of a convolutional neural network (CNN) according to some implementations is shown. Figure 4A and Figure 4B Compared to the neural network shown, Figure 4CThe example shown includes different types of neural network layers, including a first stage of layers for feature learning and a second stage of layers for classification tasks (e.g., object recognition). The feature learning stage includes a convolutional and rectified linear unit (ReLU) layer 430, followed by a pooling layer 432, which is followed by another convolutional and ReLU layer 434, which is followed by another pooling layer 436. The first layer 430 extracts features from an input 428 (e.g., an input image or portion thereof) and performs a convolution operation and one or more nonlinear operations (e.g., ReLU, tanh, or sigmoid) on its input. Pooling layers (e.g., layer 432) reduce the number of parameters when the input is large. The output of pooling layer 436 is flattened by layer 438 and input to a fully connected neural network having one or more layers (e.g., layers 440 and 442). The output of the fully connected neural network is input to a softmax layer 444 to classify the output of layer 442 of the fully connected network to produce one of many different outputs 446 (e.g., the object class or type of the input image 428).

[0253] Some implementations store the layout or organization of the input neural network in memory 214 as a neural network topology 224, including the number of neurons in each layer, the total number of neurons, the operation or activation function of each neuron, and / or the connections between neurons.

[0254] Figure 5 An example of a mathematical model 500 of a neuron according to some implementations is shown. The mathematical model includes an input of an incoming signal 502, which is multiplied by a synaptic weight 504 and summed by a unit summation 506. According to some implementations, the result of the unit summation 506 is input to a nonlinear transformation unit 508 to produce an output signal 510.

[0255] Figure 6A-6C An example process for a simulated hardware implementation of a neural network for computing an XOR of input values ​​(and a classification of XOR results) according to some implementations is shown. Fig. 6A shows the possible input values ​​X along the x-axis and y-axis respectively. 1 and X 2 Table 600. The expected result values ​​are indicated by open circles (representing a value of 1) and solid or black circles (representing a value of 0), which is a typical XOR problem with 2 input signals and 2 classes. 1 and X 2 The expected result is 1 only when any one but not both of are 1, otherwise it is 0. The training set consists of 4 possible input signal combinations (X 1 and X 2 The binary value of the input). Figure 6B Shows a solution according to some implementations Fig. 6A A ReLU-based neural network 602 for XOR classification. The neurons do not use any bias values, but use ReLU activation. Inputs 604 and 606 (which correspond to X 1 and X 2 ) is input to the first ReLU neuron 608-2. Inputs 604 and 606 are also input to the second ReLU neuron 608-4. The results of the two ReLU neurons 608-2 and 608-4 are input to the third neuron 608-6 which performs a linear summation of the input values ​​to produce an output value 510 (Out value). The neural network 602 has weights -1 and 1 for the ReLU neuron 608-2 (for the input value X, respectively). 1 and X 2 ), for the weights 1 and -1 of ReLU neuron 608-4 (for the input value X 1 and X 2 ) and weights 1 and 1 (for the outputs of ReLU neurons 608-2 and 608-4, respectively). In some implementations, the weights of the trained neural network are stored in memory 214 as weights 222.

[0256] Figure 6C An example equivalent analog network of network 602 according to some implementations is shown. 1 Enter 604 and X 2 The simulated equivalents of input 606, 614 and 616, are input to the simulated neurons N1 618 and N2 620 of the first layer. Neurons N1 and N2 are densely connected to neurons N3 and N4 of the second layer. The neurons of the second layer (i.e., neuron N3 622 and neuron N4 624) are connected to output neuron N5 626 which produces output Out (equivalent to output 610 of network 602). Neurons N1, N2, N3, N4, and N5 have a ReLU (max=1) activation function.

[0257] Some implementations use Keras learning, which converges at approximately 1000 iterations and produces weights for the connections. In some implementations, the weights are stored in memory 214 as part of weights 222. In the example below, the data format is "neuron [1st link weight, 2nd link weight, bias]".

[0258] N1[-0.9824321,0.976517,-0.00204677];

[0259] N2[1.0066702,-1.0101418,-0.00045485];

[0260] N3[1.0357606,1.0072469,-0.00483723];

[0261] N4[-0.07376373,-0.7682612,0.0]; and

[0262] N5[1.0029935,-1.1994369,-0.00147767].

[0263] Next, to calculate the resistor values ​​for the connections between neurons, some implementations calculate the resistor range. Some implementations set a resistor nominal value (R+, R-) of 1MΩ, a possible resistor range of 100KΩ to 1MΩ, and a nominal series E24. Some implementations calculate w1, w2, wbias resistor values ​​for each connection as follows. For each weight value wi (e.g., weight 222), some implementations evaluate all possible (Ri-, Ri+) resistor pair options within the selected nominal series and select the one that produces the minimum error value According to some implementations, the following table provides example values ​​for the weights w1, w2, and bias for each connection.

[0264]

[0265]

[0266] Example Benefits of Transformed Neural Networks

[0267] Before describing the example of transformation, it is worth noting that the transformed neural network has some advantages over the traditional architecture. As described herein, the input trained neural network is transformed into a pyramid or ladder analog network. Some advantages of the pyramid or ladder over the crossbar include lower latency, simultaneous analog signal propagation, the possibility of manufacturing using standard integrated circuit (IC) design components (including resistors and operational amplifiers), high parallelism of calculations, high accuracy (e.g., relative to traditional methods, the accuracy increases with the number of layers), tolerance to errors in each weight and / or at each connection (e.g., the pyramid balances the errors), low RC (low resistance-capacitance delay associated with the propagation of the signal through the network) and / or the ability to manipulate the bias and function of each neuron in each layer of the transformed network. In addition, the pyramid itself is an excellent computational block because it is a multi-level perceptron that can model any neural network with one output. According to some implementations, different pyramid or ladder geometries are used to implement networks with several outputs. The pyramid can be considered as a multi-layer perceptron with one output and several layers (e.g., N layers), where each neuron has n inputs and 1 output. Similarly, the trapezoid is a multilayer perceptron where each neuron has n inputs and m outputs. According to some implementations, each trapezoid is a pyramidal network where each neuron has n inputs and m outputs, where n and m are limited by IC analog chip design constraints.

[0268] Some implementations perform lossless transformation of any trained neural network to a subsystem of a pyramid or a ladder. Therefore, pyramids and ladders can be used as general building blocks for transforming any neural network. One advantage of a neural network based on a pyramid or a ladder is the possibility of implementing any neural network using standard IC analog components (e.g., operational amplifiers, resistors, signal delay lines in the case of recurrent neurons) using standard photolithography techniques. It is also possible to restrict the weights of the transformed network to a certain interval. In other words, according to some implementations, lossless transformation is performed when the weights are restricted to a certain predefined range. Another advantage of using a pyramid or a ladder is the high degree of parallelism in signal processing or the simultaneous propagation of analog signals, which increases the speed of calculation and provides lower latency. In addition, many modern neural networks are sparsely connected networks, and when transformed into a pyramid, they are much better (e.g., more compact, with low RC values, no leakage current) than when transformed into a crossbar switch, and pyramid and ladder networks are relatively more compact than crossbar-based memristor networks.

[0269] In addition, the analog neuromorphic ladder chip has many characteristics that are not typical for analog devices. For example, the signal-to-noise ratio does not increase with the number of cascades in the analog chip, external noise is suppressed, and the influence of temperature is greatly reduced. Such characteristics make the ladder-shaped analog neuromorphic chip similar to a digital circuit. For example, according to some implementations, individual neurons level the signal based on an operational amplifier and operate at a frequency of 20,000Hz-100,000Hz, and are not affected by noise or signals with a frequency higher than the operating range. Due to the uniqueness of how the operational amplifier operates, the ladder-shaped analog neuromorphic chip also performs filtering of the output signal. Such a ladder-shaped analog neuromorphic chip suppresses synphase noise. Due to the low-ohmic output of the operational amplifier, the noise is also significantly reduced. Due to the leveling of the signal at each operational amplifier output and the synchronous operation of the amplifier, the drift of the parameters caused by temperature does not affect the signal at the final output. The ladder-shaped analog neuromorphic circuit tolerates errors and noise in the input signal and tolerates deviations in the resistor values ​​corresponding to the weight values ​​in the neural network. The ladder-shaped analog neuromorphic network is also tolerant to any type of systematic errors, such as errors in resistor value settings, if such errors are the same for all resistors based on the operational amplifier due to the nature of the analog neuromorphic ladder-shaped circuit.

[0270] Example lossless transform (T-transform) of a trained neural network

[0271] In some implementations, the example transformations described herein are performed by a neural network transformation module 226 that transforms the trained neural network 220 based on the mathematical formula 230, the basic function blocks 232, the analog component models 234, and / or the analog design constraints 236 to obtain a transformed neural network 228.

[0272] Figure 7 An example perceptron 700 is shown according to some implementations. The perceptron includes K=8 inputs and 8 neurons 702-2, ..., 702-16 in an input layer that receives 8 inputs. There is an output layer in which there are 4 neurons 704-2, ..., 704-8 corresponding to L=4 outputs. The neurons in the input layer are fully connected to the neurons in the output layer, establishing 8×4=32 connections. Assume that the weights of the connections are represented by the weight matrix WP (elements WP i,j corresponds to the weight of the connection between the ith neuron in the input layer and the jth neuron in the output layer). Further assume that each neuron implements an activation function F.

[0273] Figure 8 An example pyramid neural network (P-NN) 800 is shown, according to some implementations, which is a target neural network (T-NN or TNN) equivalent to Figure 7 The perceptron shown. To perform the perceptron ( Figure 7 ) to PN-NN architecture ( Figure 8 ), assuming that for the T-NN, the number of inputs is limited to Ni=4 and the number of outputs is limited to No=2. The T-NN includes an input layer LTI with neurons 802-2, ..., 802-34, which is a concatenation of two copies of an input layer with neurons 802-2, ..., 802-16, for a total of 2×8=16 input neurons. A set of neurons 804 (including neurons 802-20, ..., 802-34) is a copy of neurons 802-2, ..., 802-18, and the inputs are replicated. For example, the input of neuron 802-2 is also input to neuron 802-20, the input of neuron 802-4 is also input to neuron 802-22, and so on. Figure 8 Also included is a hidden layer LTH1 with neurons 806-02, ..., 806-16 (2 times 16 divided by 4 = 8 neurons), which are linear neurons. Each group of Ni neurons from the input layer LTI is fully connected to two neurons from the LTH1 layer. Figure 8 Also included is an output layer LTO having 2 times 8 divided by 4 = 4 neurons 808-02, ..., 808-08, each neuron implementing an activation function F. Each neuron in the LTO layer is connected to a different neuron from a different group in the layer LTH1. Figure 8 The network shown includes 40 connections. As shown below, some implementations for Figure 8 The P-NN in performs weight matrix calculation. The weight (WTH1) of the hidden layer LTH1 is calculated according to the weight matrix WP, and the weight (WTO) corresponding to the output layer LTO forms a sparse matrix with elements equal to 1.

[0274] Fig. 9 shows a corresponding Figure 8 The PSNN includes a pyramid single neural network (PSNN) 900 having an output neuron of . The PSNN includes input neurons 902-02, ..., 902-16 (corresponding to Figure 78 input neurons in the network 700) (LPSI). The hidden layer LPSH1 includes 8 divided by 4 = 2 linear neurons 904-02 and 904-04, and each group of Ni neurons from LTI is connected to one neuron of the LPSH1 layer. The output layer LPSO consists of 1 neuron 906 with activation function F, which is connected to neurons 904-02 and 904-04 of the hidden layer at the same time. To calculate the weight matrix of the PSNN 900, some implementations calculate a vector WPSH1 equal to the first row of WP for the LPSH1 layer. For the LPSO layer, some implementations calculate a weight vector WPSO with 2 elements, each element equal to 1. This process is repeated for the first, second, third and fourth output neurons. P-NN (e.g. Figure 8 The network shown) is a union of PSNNs (for 4 output neurons). The input layer of each PSNN is a separate copy of the input layer of P. For this example, P-NN 800 includes an input layer with 8 times 4 = 32 inputs, a hidden layer with 2 times 4 = 8 neurons, and an output layer with 4 neurons.

[0275] Example transformation for a target neuron with N inputs and 1 output

[0276] In some implementations, the example transformations described herein are performed by a neural network transformation module 226 that transforms the trained neural network 220 based on the mathematical formula 230, the basic function blocks 232, the analog component models 234, and / or the analog design constraints 236 to obtain a transformed neural network 228.

[0277] Single-layer perceptron with one output

[0278] Assume that a single-layer perceptron SLP(K,1) consists of K inputs and one output neuron with activation function F. Further assume that U∈R K is the vector of weights for SLP(K,1). The following algorithm Neuron2TNN1 constructs a T neural network from T neurons with N inputs and 1 output (referred to as TN(N,1)).

[0279] Algorithm Neuron2TNN1

[0280] 1. Construct the input layer of T-NN by including all inputs from SLP(K,1).

[0281] 2. If K>N, then:

[0282] a. Divide K input neurons into groups, such that each group consists of no more than N inputs.

[0283] b. From m1 The neurons construct the first hidden layer LTH of T-NN 1 , each neuron performs the same activation function.

[0284] c. Connect the input neurons from each group to the corresponding neurons from the next layer. 1 Each neuron of has no more than N input connections.

[0285] d. Set the weight of the new connection according to the following equation:

[0286] w ij =u j , j = (i-1)*N+1, ..., i*N

[0287]

[0288] 3. Otherwise (i.e., if K<=N), then):

[0289] a. Construct the output layer with 1 neuron that calculates the activation function F

[0290] b. Connect the input neurons to a single output neuron. It has K ≤ N connections.

[0291] c. Set the weight of the new connection with the help of the following equation:

[0292]

[0293] d. Termination algorithm

[0294] 4. Set l = 1

[0295] 5. If m l >N:

[0296] a. Divide ml neurons into groups, each group consists of no more than N neurons.

[0297] b. From m l+1 The hidden layer LTH of T-NN is constructed by neurons l+1 , each neuron has an identical activation function.

[0298] c. Connect the input neurons from each group to the corresponding neurons from the next layer.

[0299] d. Set the weight of the new connection according to the following equation:

[0300]

[0301] e. Set l = l + 1

[0302] 6. Otherwise (if m>=N):

[0303] a. Construct the output layer with 1 neuron that calculates the activation function F

[0304] b. All LTH l The neurons are connected to a single output neuron.

[0305] c. Set the weight of the new connection according to the following equation:

[0306]

[0307] d. Termination algorithm

[0308] 7. Repeat steps 5 and 6.

[0309] Here, is the smallest integer not less than x. The number of layers in the T-NN constructed with the aid of the algorithm Neuron2TNN1 is The total number of weights in T-NN is:

[0310]

[0311] Fig.10 An example of a constructed T-NN according to some implementations is shown. All layers except the first layer perform an identity transformation of their inputs. According to some implementations, the weight matrix of the constructed T-NN has the following form.

[0312] Layer 1 (e.g. layer 1002):

[0313]

[0314] Layer i=2, 3, ..., h (e.g., layers 1004, 1006, 1008, and 1010):

[0315]

[0316] The output value of T-NN is calculated according to the following formula:

[0317] y=F(W m W m-1 ...W 2 W 1 x).

[0318] The output of the first layer is calculated as the output vector according to the following formula:

[0319]

[0320] Multiply the obtained vector by the weight matrix of the second layer:

[0321]

[0322] Each subsequent layer outputs a vector with components equal to the linear combination of some subvector of x.

[0323] Finally, the output of T-NN is equal to:

[0324]

[0325] This is the same value calculated in SLP(K,1) for the same input vector x. So the output values ​​of SLP(K,1) and the constructed TNN are equal.

[0326] Single-layer perceptron with several outputs

[0327] Assume that there is a single-layer perceptron SLP(K,L) with K input and L output neurons, each of which implements an activation function F. Further assume that U∈R L×K is the weight matrix of SLP(K,L). The following algorithm Layer2TNN1 constructs a T neural network from neurons TN(N,1).

[0328] Algorithm Layer2TNN1

[0329] 1. For each output neuron i=1,…,L

[0330] a. Applying the algorithm Neuron2TNN1 to SLP i (K,1), which consists of K inputs, 1 output neuron and a weight vector U ij ,j=1,2,…,K. TNN i is constructed as a result.

[0331] 2. By placing all TNN i Combined into a neural network to build PTNN:

[0332] a. All TNN i The input vectors of are concatenated together, so the input of PTNN has L groups of K inputs, each group is a copy of the input layer of SLP(K,L).

[0333] For the same input vector, the output of PTNN is equal to the output of SLP(K,L) because each pair of SLP i (K,1) and TNN i The outputs are equal.

[0334] Multilayer Perceptron

[0335] Assume that a multilayer perceptron (MLP) consists of K inputs, S layers, and L in the i-th layer i computational neurons, which are represented as MLP(K,S,L 1 ,…L S ). Assume is the weight matrix of the i-th layer.

[0336] Below is an example algorithm for constructing a T-neural network from neurons TN(N,1) according to some implementations.

[0337] Algorithm MLP2TNN1

[0338] 1. For each layer i=1,…,S

[0339] a. Apply the algorithm Layer2TNN1 to the L i-1 Input, L i output neurons and weight matrix U i SLP i (L i-1 ,L i ), build PTNN i As a result.

[0340] 2. By placing all PTNN i Stacked into a neural network to build MTNN; TNN i-1 The output of TNN is set i Input.

[0341] For the same input vector, the output of MTNN is equal to MLP(K,S,L 1 ,…L S ) output, because each pair of SLP i (L i-1 ,L i ) and PTNN i The outputs are equal.

[0342] For N I input and N O Example of T transformation of the target neuron with 1 output

[0343] In some implementations, the example transformations described herein are performed by a neural network transformation module 226 that transforms the trained neural network 220 based on the mathematical formula 230, the basic function blocks 232, the analog component models 234, and / or the analog design constraints 236 to obtain a transformed neural network 228.

[0344] Example transformation of a single layer perceptron with several outputs

[0345] Assume that a single-layer perceptron SLP(K,L) consists of K input and L output neurons, each of which implements an activation function F. Further assume that U∈R L×K is the weight matrix of SLP(K,L). According to some implementations, the following algorithm is derived from the neuron TN(N I ,N O )Construct T neural network.

[0346] AlgorithmLayer2TNNX

[0347] 1. Construct PTNN from SLP(K,L) by using algorithm Layer2TNN1 (see description above). PTNN has an input layer consisting of L groups of K inputs.

[0348] 2. Composed from L groups subsets. Each subset contains no more than N O groups of copies of the input vector.

[0349] 3. Replace the groups in each subset with a copy of the input vector.

[0350] 4. By building N from each input neuron O The output connections are used to construct PTNNX from the reconstructed connections in each subset.

[0351] According to some implementations, the output of PTNNX is calculated with the same formula as for PTNN (as described above), and therefore the outputs are equal.

[0352] Figures 11A-11C The above algorithm according to some implementations is shown for a neuron with 2 output neurons and TN (N I ,2) Application of single-layer neural network (NN) 1100. Fig.11A An example source or input NN according to some implementations is shown. K inputs are input to two neurons 1 and 2 belonging to layer 1104. Fig. 11B The PTNN constructed after the first step of the algorithm according to some implementations is shown. The PTNN consists of the implementation and Fig.11A The output of the NN shown is composed of two parts of the subnetwork corresponding to neuron 1 and neuron 2. Fig. 11B In FIG. 1 , input 1102 is replicated and input to two sets of input neurons 1106-2 and 1106-4. Each set of input neurons is connected to a subsequent layer of neurons having two sets of neurons 1108-2 and 1108-4, each set of neurons including m 1The input layer is followed by identity transform blocks 1110-2 and 1110-4, each of which contains one or more layers with identity weight matrices. The output of identity transform block 1110-2 is connected to output neuron 1112 (corresponding to Fig.11A The output neuron 1 in the same block 1110-4 is connected to the output neuron 1114 (corresponding to Fig.11A The output neuron in 1). Fig. 11C The application of the final steps of the algorithm is shown, including replacing the two copies of the input vector (1106-2 and 1106-4) with one vector 1116 (step 3), and reestablishing the connections in the first layer 1118 by establishing two output links from each input neuron: one link to the subnetwork associated with output 1 and the other link to the subnetwork for output 2.

[0353] Example transformation of a multilayer perceptron

[0354] Assume that a multilayer perceptron (MLP) consists of K inputs, S layers, and L in the i-th layer i computational neurons, represented as MLP(K,S,L 1 ,…L S ). Assume is the weight matrix of the i-th layer. According to some implementations, the following example algorithm starts with the neuron TN (N I ,N O )Construct T neural network.

[0355] Algorithm MLP2TNNX

[0356] 1. For each layer i=1,…,S

[0357] a. Apply the Layer2TNNX algorithm to the L i-1 Input, L i output neurons and weight matrix U i SLP i (L i-1 ,L i ).PTNNX i is constructed as a result.

[0358] 2. By transferring all PTNNX i Stacked into a neural network to build MTNNX;

[0359] a.TNNX i-1 The output is set to TNNX i Input.

[0360] According to some implementations, for the same input vector, the output of MTNNX is equal to MLP(K,S,L1 ,…L S ) output, because each pair of SLP i (L i-1 ,L i ) and PTNNX i The outputs are equal.

[0361] Example transformations of recurrent neural networks

[0362] Recurrent Neural Networks (RNNs) contain backward connections that allow information to be preserved. Fig.12 An example RNN 1200 is shown according to some implementations. The example shows a block 1204 executing an activation function A, which accepts an input X t 1206 and execute activation function A, and output value h t 1202. According to some implementations, the reverse arrow from block 1204 to itself indicates a reverse connection. The equivalent network is shown on the right until the activation block receives the input X t 1206 hours. At time 0, the network accepts input X t 1208 and execute activation function A1204, and output value h o 1210; At time 1, the network accepts input X 1 1212 and the output of the network at time 0, and execute the activation function A1204, and output the value h 1 1214; At time 2, the network accepts input X 2 1216 and the output of the network at time 1, and executes activation function A 1204, and outputs the value h 1 1218. According to some implementations, this process continues until time t, when the network accepts input X t 1206 and the output of the network at time t-1, and execute activation function A 1204, and output value h t 1202.

[0363] Data processing in RNN is performed with the help of the following formula:

[0364] h t =f(W (hh) h t-1 +W (hx) x t ).

[0365] In the above equation, x t is the current input vector, and h t-1 is the RNN's information about the previous input vector x t-1 The output of . This expression consists of several operations: For two fully connected layers W (hh) ht-1 and W (hx) x t The first and third operations can be implemented by ladder-based networks (a fully connected layer is implemented by a pyramid-based network, which is a special case of a ladder-based network). The second operation is a common operation that can be implemented in networks of any structure.

[0366] In some implementations, the layers of the RNN without recurrent connections are transformed with the help of the Layer2TNNX algorithm described above. After the transformation is completed, recurrent links are added between the relevant neurons. Some implementations use the following reference Fig. 13B Describes the delay block.

[0367] Example Transformation of LSTM Network

[0368] Long Short-Term Memory (LSTM) neural network is a special case of RNN. The operation of LSTM network is represented by the following equation:

[0369] f t =σ(W f [h t-1 ,x t ]+b f );

[0370] i t =σ(W i [h t-1 ,x t ]+b i );

[0371] D t =tanh(W D [h t-1 ,x t ]+b D );

[0372] C t =(f t ×C t-1 +i t ×D t );

[0373] o t =σ(W o [h t-1 ,x t ]+b o );as well as

[0374] h t =σ t ×tanh(C t ).

[0375] In the above equation, W f , W i , W D and W O is a trainable weight matrix, b f 、b i 、b D and b O is the trainable bias, x t is the current input vector, h t-1 is for the previous input vector x t-1 The internal state of the LSTM calculated, and o t is the output for the current input vector. In the equation, subscript t denotes time instance t, and subscript t-1 denotes time instance t-1.

[0376] Fig.13A is a block diagram of an LSTM neuron 1300 according to some implementations. The sigmoid (σ) block 1318 processes the input h t-1 1330 and x t 1332, and produces output f t 1336. The second sigmoid (σ) block 1320 processes the input h t-1 1330 and x t 1332, and produces output i t 1338. The hyperbolic tangent (tanh) block 1322 processes the input h t-1 1330 and x t 1332, and produces output D t 1340. The third sigmoid (σ) block 1328 processes the input h t-1 1330 and x t 1332, and produces output O t 1342. Multiplier block 1304 processes f t 1336 and the output C of summation block 1306 (from the previous time instance) t-1 1302 to generate an output, which in turn, together with the output i t 1338 and D t The outputs of the second multiplier block 1314 are summed together by the summing block 1306 to produce the output C t 1310. Output C t 1310 is input to another tanh block 1312 which produces an output which is multiplied by a third multiplier block 1316 with the output O t 1342 is multiplied to produce the output h t 1334.

[0377] There are several types of operations utilized in these expressions: (i) computation of linear combinations of several fully connected layers, (ii) element-wise addition, (iii) Hadamard product, and (iv) computation of nonlinear functions (e.g., sigmoid (σ) and hyperbolic tangent (tanh)). Some implementations implement operations (i) and (iv) via ladder-based networks (a fully connected layer is implemented via a pyramid-based network, which is a special case of a ladder-based network). Some implementations use networks of various structures for operations (ii) and (iii), which are common operations.

[0378] According to some implementations, the layers in the LSTM layer without recurrent connections are transformed by using the Layer2TNNX algorithm described above. According to some implementations, after the transformation is completed, recurrent links are added between the relevant neurons.

[0379] Fig. 13B 1304 processes the output C of the summation block 1306 (from the previous time instance). t-1 1302. Fig. 13B Two examples of delay blocks according to some implementations are shown. Example 1350 includes a delay block 1354 on the left that accepts an input x at time t. t 1352, and outputs the input after a delay of dt, which is represented by the output x t-dt 1356 indicates. According to some implementations, the example 1360 on the right shows that the cascaded (or multiple) delay blocks 1364 and 1366 output the input x after a time delay of 2 units. t 1362, which is output by x t- 2dt 1368 instructions.

[0380] Fig. 13C is a neuron diagram of an LSTM neuron according to some implementations. The diagram includes weighted summer nodes (sometimes referred to as adder blocks) 1372, 1374, 1376, 1378, and 1396, multiplier blocks 1384, 1392, and 1394, and delay blocks 1380 and 1382. Input x t 1332 is connected to adder blocks 1372, 1374, 1376 and 1378. For the previous input x t-1 The output h t-1 1330 is also input to adder blocks 1372, 1374, 1376, and 1378. Adder block 1372 produces an output which is input to the output ft 1336 of the sigmoid block 1394-2. Similarly, the adder block 1374 produces an output which is input to the sigmoid block 1394-2 that produces the output i t 1338 of the sigmoid block 1386. Similarly, the adder block 1376 produces an output which is input to produce the output D t 1340 of the hyperbolic tangent block 1388. Similarly, the adder block 1378 produces an output which is input to produce the output O t 1342 of the sigmoid block 1390. The multiplier block 1392 uses the output i t 1338, f t 1336 and the output C from the previous time instance of adder block 1396 t-1 1302 to generate a first output. Multiplier block 1394 uses output i t 1338 and D t 1340 to generate the second output. Adder block 1396 adds the first output and the second output to generate output C t 1310. Output C t 1310 is input to a hyperbolic tangent block 1398 which produces an output which, together with the output of the sigmoid block 1390, is t 1342 are input to multiplier block 1384 to produce an output h t 1334. Delay block 1382 is used to call (e.g., save and restore) the output of adder block 1396 from the previous time instance. Similarly, delay block 1380 is used to call or save and restore the output of multiplier block 1384 for the previous input x t-1 (e.g., from a previous time instance). According to some implementations, the above reference Fig. 13B An example of a delay block is described.

[0381] Example transformations of a GRU network

[0382] The Gated Recurrent Unit (GRU) neural network is a special case of RNN. The operation of RNN is represented by the following expression:

[0383] z t =σ(W z x t +U z h t-1 ):

[0384] r t =σ(W r x t +U r h t-1 );

[0385] jt =tanh(Wx t +r t × t-1 ):

[0386] h t =z t ×h t-1 +(1-z t )×j t ).

[0387] In the above equation, x t is the current input vector, and h t-1 is for the previous input vector x t-1 The output of the calculation.

[0388] Fig.14A is a block diagram of a GRN neuron according to some implementations. The sigmoid (σ) block 1418 processes the input h t-1 1402 and x t 1422, and produces output r t 1426. The second sigmoid (σ) block 1420 processes the input h t-1 1402 and x t 1422, and produces output z t 1428. The multiplier block 1412 makes the output r t 1426 and input h t-1 1402 to produce an output which (together with the input x t 1422) are input to a hyperbolic tangent (tanh) block 1424 to generate an output j t 1430. The second multiplier block 1414 makes the output j t 1430 and output z t 1428 multiplies to produce the first output. Block 1410 calculates 1-output z t 1428 to generate an output which is input to the third multiplier block 1404, which multiplies the output and the input h t-1 1402 to produce a product which, along with the first output (from multiplier block 1414), is input to adder block 1406 to produce an output h t 1408. Input h t-1 1402 is the output of the GRU neuron from the previous time interval output t-1.

[0389] Fig. 14Bis a neuron diagram of a GRU neuron 1440 according to some implementations. The diagram includes weighted summer nodes (sometimes referred to as adder blocks) 1404, 1406, 1410, 1406, and 1434, multiplier blocks 1404, 1412, and 1414, and delay block 1432. Input x t 1422 is connected to adder blocks 1404, 1410 and 1406. The previous input x t-1 The output h t-1 1402 is also input to adder blocks 1404 and 1406 and multiplier blocks 1404 and 1412. Adder block 1404 produces an output which is input to produce output Z t 1428 of the sigmoid block 1418. Similarly, the adder block 1406 produces an output that is input to the sigmoid block 1420, which produces an output r that is input to the multiplier block 1412. t 1426. The output of multiplier block 1412 is input to adder block 1410, which is input to hyperbolic tangent block 1424 which produces output 1430. Output 1430 and the output of sigmoid block 1418 are input to multiplier block 1414. The output of sigmoid block 1418 is input to multiplier block 1404 which multiplies the output with the input from delay block 1432 to produce a first output. The multiplier block produces a second output. Adder block 1434 adds the first output and the second output to produce an output h t 1408. Delay block 1432 is used to recall (e.g., save and restore) the output of adder block 1434 from a previous time instance. Fig. 13B An example of a delay block is described.

[0390] According to some implementations, the types of operations used in the GRU are the same as those for the LSTM network (described above), so the GRU is transformed into a ladder-based network according to the principles described above for the LSTM (e.g., using the Layer2TNNX algorithm).

[0391] Example transformations of a convolutional neural network

[0392] In general, a convolutional neural network (CNN) includes several basic operations, such as convolution (a collection of linear combinations of image (or internal graph) segments with kernels), activation functions, and pooling (such as maximum, average, etc.). Each computational neuron in a CNN follows the general processing scheme of a neuron in an MLP: a linear combination of some inputs and subsequent computation of an activation function. Therefore, according to some implementations, the CNN is transformed using the MLP2TNNX algorithm described above for a multilayer perceptron.

[0393] Conv1D is a convolution performed on the time coordinate. Fig.15A and Fig. 15B is a neuron diagram of a variant of a single Conv1D filter according to some implementations. Fig.15A , the weighted summer node 1502 (sometimes referred to as an adder block, marked with a “+”) has 5 inputs, so it corresponds to a 1D convolution with a kernel of 5. The input is x at time t t 1504. x at time t-1 t-1 1514 (obtained by inputting the input to the delay block 1506), x at time t-2 t-2 1516 (obtained by inputting the output of delay block 1506 into another delay block 1508), x at time t-3 t-3 1518 (obtained by inputting the output of delay block 1508 into another delay block 1510) and x at time t-4 t-4 1520 (obtained by inputting the output of delay block 1510 into another delay block 1512). For large cores, it is sometimes beneficial to use different frequency delay blocks, so that some blocks produce larger delays. Fig. 15B As shown, some implementations replace one large delay block with several small delay blocks. Fig.15A In addition to the delay block in , this example also uses x to generate time t-3 t-3 The delay_3 block 1524 of 1518 and the x generated at time t-5 t-5 Another delay block 1526 of 1522. According to some implementations, the delay_3 1524 block is an example of multiple delay blocks. According to some implementations, this operation does not reduce the total number of blocks, but it can reduce the total number of subsequent operations performed on the input signal and reduce the accumulation of errors.

[0394] In some implementations, the convolutional layers are represented by ladder-shaped neurons, and the fully connected layers are represented by a crossbar of resistors. Some implementations use a crossbar and compute the resistance matrix of the crossbar.

[0395] Example approximation algorithm for a single layer perceptron with multiple outputs

[0396] In some implementations, the example transformations described herein are performed by a neural network transformation module 226 that transforms a trained neural network 220 and / or a simulated neural network optimization module 246 based on mathematical formulas 230, basic function blocks 232, simulated component models 234, and / or simulated design constraints 236 to obtain a transformed neural network 228.

[0397] Assume that a single-layer perceptron SLP(K,L) consists of K input and L output neurons, each of which implements an activation function F. Further assume that U∈R L×K is the weight matrix of SLP(K,L). The following is an implementation of the approximation algorithm Layer2TNNX_Approx from the neuron TN(N I ,N O ) Example of building a T neural network. The algorithm applies the (described above) Layer2TNN1 algorithm in a first stage in order to reduce the number of neurons and connections, and then applies Layer2TNNX to process the input of reduced size. The output of the resulting neural network is calculated using the shared weights of the layers built by the Layer2TNN1 algorithm. The number of these layers is determined by the value p (a parameter of the algorithm). If p is equal to 0, only the Layer2TNNX algorithm is applied and the transformations are equivalent. If p>0, the p layers have shared weights and the transformations are approximate.

[0398] AlgorithmLayer2TNNX_Approx

[0399] 1. Use from collection Set the value of parameter p.

[0400] 2. If p>0, for neuron TN (N I ,1) is applied to the network SLP(K,L) and the first p layers of the resulting subnetwork (PNN) are constructed. The network PNN has A neuron.

[0401] 3. Application to TN neurons (N I ,N O ) and construct an algorithm Layer2TNNX with N p A neural subnetwork TNN with 1 input and L outputs.

[0402] 4. Set the weight of the PNN network. According to the rules Set the weight of each neuron i in the first layer of PNN. Here, C is any constant not equal to zero, k i =(i-1)N I +1, and for all except k i All the weights j of this neuron outside All other weights of the PNN network are set to 1. For neurons i and k in the first layer i The weights of the first layer of connections between (as indicated by the superscript (1)).

[0403] 5. Set the weight of the TNN subnet. According to the equation The weight of each neuron i in the first layer of the TNN (considering the entire network, this is the (p+1)th layer) is set. All other weights of the TNN are set to 1.

[0404] 6. Set the activation function of all neurons in the last layer of the TNN subnet to F. The activation functions of all other neurons are identical.

[0405] Fig.16 An example architecture 1600 of the resulting neural network according to some implementations is shown. The example includes a PNN 1602 connected to a TNN 1606. The PNN 1602 includes a layer of K inputs and produces N p 1610, which are connected as inputs 1612 to TNN 1606. According to some implementations, TNN 1606 generates L outputs 1610.

[0406] Approximation Algorithm for Multilayer Perceptron with Several Outputs

[0407] Assume that a multilayer perceptron (MLP) consists of K inputs, S layers, and L in the i-th layer i computational neurons, represented as MLP(K,S,L 1 ,…L S ). Further assume is the weight matrix of the i-th layer. According to some implementations, the following example algorithm starts with the neuron TN (N I ,N O )Construct T neural network.

[0408] Algorithm MLP2TNNX_Approx

[0409] 1. For each layer i=1,…,S:

[0410] a. Apply the (above) algorithm Layer2TNNX_Approx to the L i-1 Input, L i output neurons and weight matrix U i SLP i (L i-1 ,L i ). If i = 1, then L 0 = K. Assume that this step constructs PTNNX i As a result.

[0411] 2. By transferring all PTNNX i Stacked into a neural network to build MTNNX (Multilayer Perceptron), where TNNX i-1 The output is set to TNNXi Input.

[0412] Example method for compression of transformed neural networks

[0413] In some implementations, the example transformations described herein are performed by a neural network transformation module 226 that transforms a trained neural network 220 and / or a simulated neural network optimization module 246 based on mathematical formulas 230, basic function blocks 232, simulated component models 234, and / or simulated design constraints 236 to obtain a transformed neural network 228.

[0414] This section describes example methods for compression of neural networks according to some implementations of transformations. Some implementations compress analog pyramid neural networks in order to minimize the number of operational amplifiers and resistors necessary to implement the analog network on a chip. In some implementations, the method of compression of analog neural networks is pruning, similar to pruning in software neural networks. However, there are some particularities in the compression of pyramid analog networks, which can be implemented as IC analog chips in hardware. Because the number of components (such as operational amplifiers and resistors) is limited to the weights in analog-based neural networks, it is crucial to minimize the number of operational amplifiers and resistors placed on the chip. This will also help minimize the power consumption of the chip. Modern neural networks (such as convolutional neural networks) can be compressed 5-200 times without significant loss of accuracy of the network. Entire blocks in modern neural networks can often be pruned without significant loss of accuracy. The transformation of dense neural networks to sparsely connected pyramid or ladder or crossbar-shaped neural networks provides the opportunity to prune sparsely connected pyramid or ladder-shaped analog networks, which are then represented by operational amplifiers and resistors in analog IC chips. In some implementations, such techniques are applied in addition to traditional neural network compression techniques. In some implementations, compression techniques are applied based on the particular architecture of the input neural network and / or the transformed neural network (e.g., pyramid versus ladder versus crossbar).

[0415] For example, because the network is implemented with the aid of analog components (e.g., operational amplifiers), some implementations determine the current flowing through the operational amplifier when a standard training data set is presented, and thereby determine whether the node (operational amplifier) ​​is needed for the entire chip. Some implementations analyze the SPICE model of the chip and determine the nodes and connections where no current flows and no power is consumed. Some implementations determine the current flow through the analog IC network and thereby determine the nodes and connections, which will then be pruned. In addition, some implementations also remove connections if the weight of the connection is too high, and / or replace resistors with direct connectors if the weight of the connection is too low. If all connections leading to the node have a weight below a predetermined threshold (e.g., close to 0), some implementations prune the node, delete connections where the operational amplifier always provides zero at the output, and / or change the operational amplifier to a linear junction if the amplifier gives a linear function without amplification.

[0416] Some implementations apply compression techniques specific to pyramid, ladder or crossbar type neural networks. Some implementations generate pyramids or ladders with a larger number of inputs (compared to the case without compression), thus minimizing the number of layers in the pyramid or ladder. Some implementations generate more compact ladder networks by maximizing the number of outputs per neuron.

[0417] Example generation of an optimal resistor set

[0418] In some implementations, the example calculations described herein are performed by a weight matrix calculation or weight quantization module 238 (e.g., using a resistance calculation module 240), which calculates weights 272 for connections of the transformed neural network and / or corresponding resistance values ​​242 for the weights 272.

[0419] According to some implementations, this section describes an example of generating an optimal set of resistors for a trained neural network. An example method is provided for converting connection weights into resistor nominals for implementing a neural network (sometimes referred to as a NN model) on a microchip with potentially fewer resistor nominals and potentially higher allowed resistor variations.

[0420] Assume that the test set "Test" includes about 10000 values ​​of the input vector (x and y coordinates), where both coordinates are changed in the range [0; 1] with a step size of 0.01. Assume that the output of the network NN for a given input X is given by Out = NN(X). Further assume that the input value class is found as follows: Class_nn(X) = NN(X)>0.61?1:0.

[0421] The mathematical network model M is compared below with the schematic network model S. The schematic network model includes possible resistor variations of rv and processes the "Test" set, each time producing a different vector of output values ​​S(Test) = Out_s. The output error is defined by the following equation:

[0422]

[0423] The classification error is defined by the following equation:

[0424]

[0425] Some implementations set the desired classification error to be no greater than 1%.

[0426] Example Error Analysis

[0427] Fig.17A An example graph 1700 is shown showing the correlation between output error and classification error on an M network according to some implementations. Fig.17A , the x-axis corresponds to the classification interval 1704, and the y-axis corresponds to the total error 1702 (see above). The graph shows the total error (the difference between the output of model M and the true data) for different classification intervals of the output signal. For this example, according to the graph, the best classification interval 1706 is 0.610.

[0428] Assuming that another network O produces output values ​​with a constant offset relative to the associated M output values, there will be a classification error between O and M. To keep the classification error below 1%, this offset should be in the range [-0.045, 0.040]. Therefore, the possible output error of S is 45mV.

[0429] The possible weight error is determined by analyzing the correlation between the relative error of weights / biases over the entire network and the output error. According to some implementations, the error is obtained by averaging 20 randomly modified networks on the "Test" set. Fig. 17B and Fig. 17C 1710 and 1720 shown in FIG. 1710 and 1720. In these graphs, the x-axis represents the absolute weight error 1712, and the y-axis represents the absolute output error 1714. As can be seen from the graphs, the output error limit of 45mV (y=0.045) allows a 0.01 relative error value or a 0.01 absolute error value (value of x) for each weight. The maximum weight modulus (maximum value of the absolute value of the weight among all weights) of the neural network is 1.94.

[0430] Example Procedure for Selecting a Resistor Set

[0431] The resistor set together with the {R+, R-} pair selected from the set has a value function within the desired weight range [-wlim; wlim] with a certain degree of resistor error r_err. In some implementations, the value function of the resistor set is calculated as follows:

[0432] An array of possible weight options is calculated along with the average error of the weights that depends on the resistor error;

[0433] The weight options in the array are restricted to the desired weight range [-wlim;wlim];

[0434] Values ​​that are worse than their neighbors in terms of weight error are removed;

[0435] An array of distances between neighboring values ​​is calculated; and

[0436] The value function is the composition of the squared mean or maximum of the distance array.

[0437] Some implementations iteratively search for the best resistor set by continuously adjusting each resistor value in the resistor set based on a learning rate value. In some implementations, the learning rate varies over time. In some implementations, the initial resistor set is selected to be uniform (e.g., [1; 1; ...; 1]), where the minimum and maximum resistor values ​​are selected to be within two orders of magnitude (e.g., [1; 100] or [0.1; 10]). Some implementations select R+=R-. In some implementations, the iterative process converges to a local minimum. In one case, the process results in the following set: [0.17, 1.036, 0.238, 0.21, 0.362, 1.473, 0.858, 0.69, 5.138, 1.215, 2.083, 0.275]. For the weight range [-2; 2] with rmin=0.1 (minimum resistance), rmax=10 (maximum resistance), and r_err=0.001 (estimated error in resistance), this is a local optimal resistor set of 12 resistors. Some implementations do not use the entire available range [rmin; rmax] for finding a good local optimum. Only a portion of the available range (e.g., [0.17; 5.13] in this case) is used. The values ​​of the resistor set are relative, not absolute. In this case, a relative value range of 30 is sufficient for the resistor set.

[0438] In one example, for the parameters mentioned above, the following set of resistors of length 20 is obtained: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02]. In this example, the value 1.763 is also the R-=R+ value. This set is then used to generate the weights of the NN, resulting in a corresponding model S. Assuming the relative resistor error is close to zero, the mean square output error of model S is 11 mV, so the set of 20 resistors exceeds the requirement. The maximum error on the set of input data is calculated to be 33 mV. In one example, S, DAC, and ADC converters with 256 levels were analyzed as separate models, and the results showed a 14mV mean square output error and a 49mV maximum output error. The 45mV output error on the NN corresponds to a relative identification error of 1%. The 45mV output error value also corresponds to a 0.01 relative weight error or a 0.01 absolute weight error, which is acceptable. The maximum weight coefficient in the NN is 1.94. In this way, an iterative process is used to determine the optimal (or near-optimal) resistor set based on the desired weight range [-wlim; wlim], the resistor error (relative), and the possible resistor range.

[0439] In general, very wide resistor sets are not very beneficial (e.g., between 1-1 / 5 orders of magnitude are sufficient) unless different precision is required in different layers or parts of the weight spectrum. For example, suppose the weights are in the range [0,1], but most weights are in the range [0,0.001], then better precision is required in that range. In the example described above, assuming the relative resistor error is close to zero, the set of 20 resistors is more than enough to quantize the NN network with a given precision. In one example, an average S output error of 11 mV is obtained over the set of resistors [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] (note that the values ​​are relative).

[0440] Example process for quantization of resistor values

[0441] In some implementations, the example calculations described herein are performed by a weight matrix calculation or weight quantization module 238 (e.g., using a resistance calculation module 240), which calculates weights 272 for connections of the transformed neural network and / or corresponding resistance values ​​242 for the weights 272.

[0442] This section describes example processes for quantizing resistor values ​​corresponding to weights of a trained neural network according to some implementations. The example processes substantially simplify the process of using analog hardware components to manufacture chips for implementing neural networks. As described above, some implementations use resistors to represent neural network weights and / or represent biases of operational amplifiers that simulate neurons. The example processes described herein particularly reduce the complexity in manufacturing sets of resistors for chip lithography. Using the process of quantizing resistor values, only selected values ​​of resistors are required for chip manufacturing. In this way, the example process simplifies the overall process of chip manufacturing and enables automatic resistor lithography mask manufacturing when necessary.

[0443] Fig.18 An example scheme of a neuron model 1800 for resistor quantization according to some implementations is provided. In some implementations, the circuit is based on an operational amplifier 1824 (e.g., an AD824 series precision amplifier) ​​that receives input signals from negative weight fixed resistors (R1-1804, R2-1806, Rb-bias 1816, Rn-1818, and R-1812) and positive weight fixed resistors (R1+1808, R2+1810, Rb+bias 1820, Rn+1822, and R+1814). The positive weight voltage is fed to the direct input of the operational amplifier 1824, and the negative weight voltage is fed to the inverting input of the operational amplifier 1824. The operational amplifier 1824 is used to allow a weighted summation operation of the weighted outputs from each resistor, where the negative weights are subtracted from the positive weights. The operational amplifier 1824 also amplifies the signal to a degree necessary for the operation of the circuit. In some implementations, operational amplifier 1824 also performs a RELU transformation of the output signal at its output cascade.

[0444] The following equation determines the weight based on the resistor value:

[0445] The voltage at the output of the neuron is determined by the variance:

[0446]

[0447] The weight of each connection is determined by the following equation:

[0448]

[0449] According to some implementations, the following example optimization process quantizes the value of each resistor and minimizes the error of the neural network output:

[0450] 1. Get the set of connection weights and biases {w1,…,wn,b}.

[0451] 2. Obtain the minimum and maximum possible resistor values ​​{Rmin, Rmax}. These parameters are determined by the technology used for manufacturing. Some implementations use TaN or tellurium high resistivity materials. In some implementations, the minimum value of the resistor is determined by the minimum square that can be formed photolithographically. The maximum value is determined by the allowable length of the resistor (e.g., a resistor made of TaN or tellurium) to adjust to the desired area, which in turn is determined by the area of ​​the op amp square on the photolithography mask. In some implementations, the area of ​​the array of resistors is less than the area of ​​one op amp because the array of resistors is stacked (e.g., one in the BEOL and the other in the FEOL).

[0452] 3. Assume that each resistor has a relative tolerance value of r_err.

[0453] 4. The goal is to select a set of resistor values ​​{R1, ..., Rn} of a given length N within the defined [Rmin; Rmax] based on the {w1, ..., wn, b} values. An example search algorithm is provided below to find a suboptimal set of {R1, ..., Rn} based on a specific optimization criterion.

[0454] 5. Another algorithm selects {Rn, Rp, Rni, Rpi} for the network, assuming that {R1…Rn} is determined.

[0455] Example {R1, ..., Rn} search algorithm

[0456] Some implementations use an iterative approach for resistor set search. Some implementations select an initial (random or uniform) set {R1, ..., Rn} within a defined range. Some implementations select one of the elements of the resistor set as the R-=R+ value. Some implementations change each resistor in the set by the current learning rate value until such changes produce a "better" set (according to the value function). This process is repeated for all resistors in the set and using several different learning rate values ​​until no further improvement is possible.

[0457] Some implementations define the value function of a resistor set as follows:

[0458] Possible weight options are calculated according to the formula (above):

[0459]

[0460] Estimate the expected error value for each weighting option based on the potential resistor relative error r_err determined by the IC manufacturing technology

[0461] The weight option list is limited or restricted to the range [-wlim; wlim]

[0462] Remove values ​​whose expected error exceeds a high threshold (e.g., 10 times r_err)

[0463] The value function is calculated as the square mean of the distance between two adjacent weighted options.

[0464] Therefore, the value function is minimized when the weight options are uniformly distributed within the range [-wlim; wlim].

[0465] Assume that the required weight range [-wlim; wlim] of the model is set to [-5; 5], and other parameters include N = 20, r_err = 0.1%, rmin = 100KΩ, rmax = 5MΩ. Here, rmin and rmax are the minimum and maximum values ​​of the resistance, respectively.

[0466] In one example, for the above mentioned parameters, the following set of resistors of length 20 is obtained: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] MΩ. R−=R+=1.763 MΩ.

[0467] Example {Rn,Rp,Rni,Rpi} search algorithm

[0468] Some implementations use an iterative algorithm (such as the one above) to determine Rn and Rp. Some implementations set Rp = Rn (the task of determining Rn and Rp is symmetric, and the two quantities usually converge to a similar value). Then, for each weight w i , some implementations choose a pair of resistors {Rni, Rpi} that minimizes the estimated weight error value:

[0469]

[0470] Some implementations then use the set of {Rni; Rpi; Rn; Rp} values ​​to implement the neural network circuit diagram. In one instance, according to some implementations, the circuit diagram produces a mean square output error of 11mV (sometimes referred to as the S mean square output error described above) and a maximum error of 33mV on a set of 10,000 evenly distributed input data samples. In one instance, the S model is analyzed together with a digital-to-analog converter (DAC) and an analog-to-digital converter (ADC), with 256 levels as a separate model. According to some implementations, the model produces a 14mV mean square output error and a 49mV maximum output error on the same data set. DACs and ADCs have levels because they convert analog values ​​into bit values ​​and vice versa. 8 bits of a digital value equals 256 levels. For an 8-bit ADC, the accuracy cannot be better than 1 / 256.

[0471] Some implementations are based on Kirchhoff's circuit laws and the fundamentals of operational amplifiers when the weights of the connections are known (see below). Fig.19A Described), use Mathcad or any other similar software to calculate the resistance value of the analog IC chip. In some implementations, the operational amplifier is used for both amplification of the signal and transformation according to an activation function (such as ReLU, sigmoid, tangent hyperbolic or linear mathematical equation).

[0472] Some implementations fabricate resistors in a photolithographic layer where the resistors are formed as cylindrical holes in a SiO2 matrix and the resistance value is determined by the diameter of the hole. Some implementations use amorphous TaN, TiN, CrN or tellurium as high resistance materials to fabricate high density resistor arrays. Some ratios of Ta to N, Ti to N and Cr to N provide high resistance for fabricating ultra-high density, high resistivity element arrays. For example, for TaN, Ta5N6, Ta3N5, the higher the ratio of N to Ta, the higher the resistivity. Some implementations use Ti2N, TiN, CrN or Cr5N and determine the ratio accordingly. TaN deposition is a standard procedure used in chip manufacturing and is available in all major foundries.

[0473] Example Op Amp

[0474] Fig.19A Schematic diagram of an operational amplifier (CMOS OpAmp) 1900 fabricated on CMOS according to some implementations is shown. Fig.19AIn 1404, In+ (positive input or pos) 1404, In- (negative input or neg) 1406, and Vdd- (positive supply voltage relative to GND) 1402 are contact inputs. Contact Vss- (negative supply voltage or GND) is indicated by label 1408. The circuit output is Out 1410 (contact output). The parameters of a CMOS transistor are determined by the ratio of the following geometric dimensions: L (gate channel length) to W (gate channel width), examples of which are described below. Fig.19B 1944, M12 1946 and resistor R1 1921 (with an example resistance value of 12 kΩ) are shown. A current mirror is fabricated on NMOS transistors M11 1944, M12 1946 and resistor R1 1921 (with an example resistance value of 12 kΩ) and provides a compensation current for the differential pair (M1 1926 and M3 1930). A differential amplifier stage (differential pair) is fabricated on NMOS transistors M1 1926 and M3 1930. Transistors M1, M3 perform amplification, while PMOS transistors M2 1928 and M4 1932 act as active current loads. A signal is input from the M3 transistor to the gate of the output PMOS transistor M7 1936. A signal is input from transistor M1 to the PMOS transistor M5 (inverter) 1934 and an active load on the NMOS transistor M6 1934. The current flowing through transistor M5 1934 is set for NMOS transistor M8 1938. Transistor M7 1936 is included in the scheme with a common source for the positive half wave signal. M8 transistor 1938 is enabled by the common source circuit for the negative half wave signal. To increase the overall load capability of the op amp, the outputs of M7 1936 and M8 1938 include inverters on the M9 1940 and M10 1942 transistors. Capacitors C1 1912 and C2 1914 are blocking.

[0475] Fig.19B Shown according to some implementations Fig.19A Table 1948 of the description of the example circuit shown. The values ​​of the parameters are provided as examples, and various other configurations are possible. Transistors M1, M3, M6, M8, M10, M11 and M12 are N-channel MOSFET transistors with explicit substrate connections. Other transistors M2, M4, M5, M7 and M9 are P-channel MOSFET transistors with explicit substrate connections. The table shows the shutter ratio of length (L, column 1) and shutter ratio of width (W, column 2) for the examples provided for each transistor (column 3).

[0476] In some implementations, an operational amplifier (such as the examples described above) is used as a basic element of an integrated circuit for a hardware implementation of a neural network. In some implementations, the operational amplifier has a size of 40 square microns and is manufactured according to the 45nm node standard.

[0477] In some implementations, activation functions (e.g., ReLU, hyperbolic tangent, and sigmoid functions) are represented by an operational amplifier with a modified output cascade. For example, according to some implementations, a ReLU, sigmoid, or tangent function is implemented as an output cascade of an operational amplifier (sometimes referred to as an OpAmp) using a corresponding well-known analog circuit diagram.

[0478] In the examples described above and below, in some implementations, the operational amplifiers are replaced by inverters, current mirrors, two- or four-quadrant multipliers, and / or other analog function blocks that allow for weighted summation operations.

[0479] Example scheme of LSTM block

[0480] Figure 20A-Figure 20E A schematic diagram of an LSTM neuron 20000 according to some implementations is shown. The inputs to the neuron are Vin1 20002 and Vin2 20004, which are values ​​in the range [-0.1, 0.1]. The LSTM neuron also inputs the value H(t-1) (previous value; see the description of the LST neuron above) 20006 that computes the result of the neuron at a certain time and the state vector C(t-1) (previous value) 20008 of the neuron at a certain time. Neuron LSTM (at Fig. 20B The output of the neuron (shown in FIG. 1 ) includes the result H(t) 20118 of calculating the neuron at the current time and the state vector C(t) 20120 of the neuron at the current time. The scheme includes:

[0481] exist Fig. 20A The operational amplifiers U1 20094 and U2 20100 are assembled on a "Neuron O".

[0482] Resistors R_Wo1 20018, R_Wo2 20016, R_Wo3 20012, R_Wo4 20010, R_Uop1 20014, R_Uom1 20020, Rr 20068, and Rf2 20066 set the weights of the connections of a single "neuron O". "Neuron O" uses sigmoid (module X1 20078, Fig. 20B ) as a nonlinear function;

[0483] In the operational amplifier U3 20098 (in Fig. 20C ) and U4 20100 (shown in Fig. 20A20028, R_Wc3 20024, R_Wc4 20022, R_Ucp1 20026, R_Ucm1 20032, Rr 20122, and Rf2 20120 set the weights of the connections of "Neuron C". "Neuron C" uses the hyperbolic tangent (module X2 22080, Figure 2B ) as a nonlinear function;

[0484] exist Fig. 20C "Neuron 1" is mounted on operational amplifiers U5 20102 and U6 20104 shown in FIG. Resistors R_Wi1 20042, R_Wi2 20040, R_Wi3 20036 and R_Wi4 20034, R_Uip1 20038, R_Uim1 20044, Rr20124 and Rf2 20126 set the weights of the connections of "Neuron 1". "Neuron 1" uses sigmoid (module X3 20082) as a nonlinear function; and

[0485] As in Fig.20D "Neuron f" is mounted on operational amplifiers U7 20106 and U8 20108 shown in FIG. Resistors R_Wf1 20054, R_Wf2 20052, R_Wf3 20048, R_Wf4 20046, R_Ufp1 20050, R_Ufm1 20056, Rr20128, and Rf2 20130 set the weights of the connections of "Neuron f." "Neuron f" uses sigmoid (module X4 20084) as a nonlinear function.

[0486] Module X2 20080( Fig. 20B ) and X3 20082( Fig. 20C ) is input to the X5 multiplier module 20086 ( Fig. 20B ). Module X4 20084( Fig.20D ) and the output of the buffer U9 20010 are input to the multiplier module X6 20088. The outputs of the modules X5 20086 and X6 20088 are input to the adder (U10 20112). The voltage divider 10 is assembled on the resistors R1 20070, R2 20072 and R3 20074. With the release of the divisor signal, the hyperbolic tangent (module X7 20090, Fig. 20B). The output C(t) 20120 (the current state vector of the LSTM neuron) is obtained using a buffer-inverter applied to the output signal of U11 20114. The outputs of modules X1 20078 and X7 20090 are input to a multiplier (module X8 20092), the output of which is input to a buffer divider divided by 10 on U12 20116. The result of calculating the LSTM neuron at the current time, H(t) 20118, is obtained from the output signal of U12 20116.

[0487] Fig.20E Shown according to some implementations Figures 20A-20D Example values ​​of different configurable parameters (e.g., voltages) of the circuit shown. According to some implementations, Vdd 20058 is set to +1.5V, Vss 20064 is set to -1.5V, Vdd1 20060 is set to +1.8V, Vss1 20062 is set to -1.0V, and GND 20118 is set to GND.

[0488] Fig.20F Shown according to some implementations Figures 20A-20D Table 20132 shows a description of the example circuit. The values ​​of the parameters are provided as examples, and various other configurations are possible. Transistors U1-U12 are CMOS OpAmp (referenced above). Fig.19A and Fig.19B As described). X1, X3 and X4 are modules that perform the Sigmoid function. X2 and X7 are modules that perform the hyperbolic tangent function. X5 and X8 are modules that perform the multiplication function. Example resistor ratings include: Rw = 10kΩ and Rr = 1.25kΩ. Other resistor values ​​are expressed relative to Rw. For example, Rf2 = 12 times Rw, R_Wo4 = 5 times Rw, R_Wo3 = 8 times Rw, R_Uop1 = 2.6 times Rw, R_Wo2 = 12 times Rw, R_W1 = w times Rw, and R_Uom1 = 2.3 times Rw, R_wc4 = 4 times Rw, R_Wc3 = 5.45 times Rw, R_Ucp1 = 3 times Rw, R_Wc2 = 12 times Rw, R_Wc1 = 2.72 times Rw, R_Ucm 1=3.7 times Rw, R_Wi4=4.8 times Rw, W_Wi3=6 times Rw, W_Uip1=2 times Rw, R_Wi2=12 times Rw, R_Wi1=3 times Rw, R_Uim1=2.3 times Rw, R_Wf4=2.2 times Rw, R_Wf3=5 times Rw, R_Wfp=4 times Rw, R_Wf2=2 times Rw, R_Wf1=5.7 times Rw, and Rfm1=4.2 times Rw.

[0489] Example scheme for a multiplier block

[0490] Figure 21A-Figure 21I 21040 and U2 21042 (in Fig. 21B ), U3 21044 (in Fig.21H shown), U4 21046 and U5 21048 (in Fig.21I The input of the multiplier includes V_one 21020 21006 and V_two 21008 (in Fig. 21B ) and the connection point Vdd (positive power supply voltage, for example, +1.5V relative to GND) 21004 and the connection point Vss (negative power supply voltage, for example, -1.5V relative to GND) 21002. In this scheme, additional power supply voltages are used: the connection point input Vdd1 (positive power supply voltage, for example, +1.8V relative to GND), the connection point Vss1 (negative power supply voltage, for example, -1.0V relative to GND). The result of the circuit calculation is mult_out (output pin) 21170 (at Fig.21I is outputted at (shown in).

[0491] refer to Fig. 21B , the input signal (V_one) from V_one 21006 is connected to an inverter and a single gain is developed on U1 21040, the output of which forms signal negA 21006, which is equal in amplitude but opposite in sign to signal V_one. Similarly, the signal (V_two) from input V_two 21008 is connected to an inverter and a single gain is developed on U2 21042, the output of which forms signal negB 21012, which is equal in amplitude but opposite in sign to signal V_two. Paired combinations of signals from the possible combinations (V_one, V_two, negA, negB) are output to corresponding mixers on CMOS transistors.

[0492] Come back for reference Fig.21A , V_two 21008 and negA 21010 are input to a multiplexer assembled on NMOS transistors M19 21086, M20 21088, M21 21090, M22 21092 and PMOS transistors M23 21094 and M24 21096. The output of the multiplexer is input to NMOS transistor M6 21060 ( Fig.21D ).

[0493] Similar transformations that occur on signals include:

[0494] negB 21012 and V_one 21020 are input to a multiplexer fabricated on NMOS transistors M11 21070, M12 2072, M13 2074, M14 21076 and PMOS transistors M15 2078 and M16 21080. The output of the multiplexer is input to M5 21058 NMOS transistor (on Fig.21D )

[0495] V_one 21020 and negB 21012 are input to a multiplexer fabricated on PMOS transistors M18 21084, M48 21144, M49 21146, and M50 21148 and NMOS transistors M17 21082, M47 21142. The output of the multiplexer is input to M9 PMOS transistor 21066 (at Fig.21D )

[0496] negA 21010 and V_two 21008 are input to a multiplexer fabricated on PMOS transistors M52 21152, M54 21156, M552 1158, and M56 21160, and NMOS transistors M51 21150 and M53 21154. The output of the multiplexer is input to M2 NMOS transistor 21054 (at Fig. 21C )

[0497] negB 21012 and V_one 21020 are input to a multiplexer fabricated on NMOS transistors M11 21070, M12 21072, M13 21074, and M14 21076 and PMOS transistors M15 21078 and M16 21080. The output of the multiplexer is input to M10 NMOS transistor 21068 (on Fig.21D )

[0498] negB 21012 and negA 21010 are input to a multiplexer fabricated on NMOS transistors M35 21118, M36 21120, M37 21122, and M38 21124, and PMOS transistors M39 21126 and M40 21128. The output of the multiplexer is input to the M27 PMOS transistor 21102 (at Fig.21H )

[0499] V_two 21008 and V_one 21020 are input to a multiplexer fabricated on NMOS transistors M41 21130, M42 21132, M43 21134, and M44 21136 and PMOS transistors M45 21138 and M46 21140. The output of the multiplexer is input to M30 NMOS transistor 21108 (at Fig.21H )

[0500] V_one 21020 and V_two 21008 are input to a multiplexer fabricated on PMOS transistors M58 21162, M60 21166, M612 1168, and M62 21170 and NMOS transistors M57 21160 and M59 21164. The output of the multiplexer is input to M34 PMOS transistor 21116 (on Fig.21H ); and

[0501] negA 21010 and negB 21012 are input to PMOS transistors M64 21174, M66 21178,

[0502] A multiplexer is fabricated on M67 21180 and M68 21182 and NMOS transistors M63 21172 and M65 21176. The output of the multiplexer is input to PMOS transistor M33 21114 (at Fig.21H ).

[0503] The current mirror (transistors M1 21052, M2 21053, M3 21054, and M4 21056) powers the portion of the four-quadrant multiplier circuit shown on the left made of transistors M5 21058, M6 21060, M7 21062, M8 21064, M9 21066, and M10 21068. The current mirror (on transistors M25 21098, M26 21100, M27 21102, and M28 21104) powers the power supply for the right portion of the four-quadrant multiplier made of transistors M29 21106, M30 21108, M31 21110, M32 21112, M33 21114, and M34 21116. The multiplication result is taken from the enabled resistor Ro21022 in parallel with transistor M3 21054 and the enabled resistor Ro 21188 in parallel with transistor M28 21104 and is provided to the adder on U3 21044. The output of U3 21044 is provided to the adder with a gain of 7,1, which is assembled on U5 21048, the second input of which is compensated by the reference voltage set by resistors R1 21024 and R2 21026 and buffer U4 21046, as shown Fig.21I The multiplication result is output from the output of U5 21048 via the Mult_Out output 21170.

[0504] Fig.21J Shown according to some implementations Figure 21A-Figure 21ITable 21198 of the description of the schematic diagram shown. U1-U5 are CMOS OpAmp. N-channel MOSFET transistors with explicit substrate connection include transistors M1, M2, M25 and M26 (with shutter length ratio (L) = 2.4u and shutter width ratio (W) = 1.26u), transistors M5, M6, M29 and M30 (with L = 0.36u and W = 7.2u), transistors M7, M8, M31 and M32 (with L = 0.36u and W = 199.98u), transistors M11-M14, M19-M22, M35-M38 and M41-M44 (with L = 0.36u and W = 0.4u) and transistors M17, M47, M51, M53, M57, M59, M43 and M64 (with L = 0.36u and W = 0.72u). P-channel MOSFET transistors with explicit substrate connections include transistors M3, M4, M27 and M28 (with shutter length ratio (L) = 2.4u and shutter width ratio (W) = 1.26u), transistors M9, M10, M33 and M34 (with L = 0.36u and W = 7.2u), transistors M18, M48, M49, M50, M52, M54, M55, M56, M58, M60, M61, M62, M64, M66, M67 and M68 (with L = 0.36u and W = 0.8u) and transistors M15, M16, M23, M24, M39, M40, M45 and M46 (with L = 0.36u and W = 0.72u). According to some implementations, example resistor ratings include Ro=1 kΩ, Rin=1 kΩ, Rf=1 kΩ, Rc4=2 kΩ, and Rc5=2 kΩ.

[0505] Example scheme of Sigmoid block

[0506] Fig.22A 2264 and NMOS transistors M1 2266, M2 2268, and M3 2270 to implement the sigmoid function (e.g., as described above with reference to FIG. 2264). Figures 20A-20F2232 and R13 2236. U6 2260 has a reference voltage source of 0.32687V, which is set by voltage dividers R14 2238 and R15 2240. U7 2262 has a reference voltage source of -0.5V which is set by voltage dividers R16 2242 and R17 2244. U8 2264 has a reference voltage source of -0.33V which is set by voltage dividers R18 2246 and R19 2248.

[0507] The sigmoid function is formed by adding the corresponding reference voltages on the differential modules assembled on transistors M1 2266 and M2 2268. The current mirror of the differential stage is assembled using an active regulation operational amplifier U3 2254 and an NMOS transistor M3 2270. The signal from the differential stage is removed with NMOS transistor M2 and resistor R5 2220 is input to adder U2 2252. The output signal sigm_out 2210 is removed from the output of the U2 adder 2252.

[0508] Fig. 22B Shown according to some implementations Fig.22A Table 2278 of the description of the schematic shown. U1-U8 are CMOS OpAmp. According to some implementations, M1, M2, and M3 are N-channel MOSFET transistors with shutter length ratio (L) = 0.18u and shutter width ratio (W) = 0.9u.

[0509] Example scheme for the hyperbolic tangent block

[0510] Fig.23A Schematic diagram of a hyperbolic tangent function block 2300 according to some implementations is shown. The hyperbolic tangent function (e.g., as described above with reference to FIG. 1 ) is implemented using operational amplifiers (U1 2312, U2 2314, U3 2316, U4 2318, U5 2320, U6 2322, U7 2328, and U8 2330) and NMOS transistors (M1 2332, M2 2334, and M3 2336). Figures 20A-20F2360 and R13 2362. U6 2322 has a reference voltage source of 0.32687V, which is set by voltage dividers R14 2364 and R15 2366. U7 2328 has a reference voltage source of -0.5V, which is set by voltage divider R16 2368 and R17 2370. U8 2330 has a reference voltage source of -0.33V, which is set by voltage divider R18 2372 and R19 2374. The hyperbolic tangent function is formed by adding the corresponding reference voltages on the differential modules made on transistors M1 2332 and M2 2334. The current mirror of the differential stage is obtained with active regulation operational amplifier U3 2316 and NMOS transistor M3 2336. Using NMOS transistor M2 2334 and resistor R5 2346, the signal is removed from the differential stage and input to adder U2 2314. The output signal tanh_out 2310 is removed from the output of the U2 adder 2314.

[0511] Fig. 23B Shown according to some implementations Fig.23A Table 2382 of the description of the schematic shown. U1-U8 are CMOS OpAmp, and M1, M2 and M3 are N-channel MOSFET transistors with shutter length ratio (L) = 0.18u and shutter width ratio (W) = 0.9u.

[0512] Example scheme of single neuron OP1CMOS OpAmp

[0513] Figure 24A-24CA schematic diagram of a single neuron OP1 CMOS OpAmp 2400 according to some implementations is shown. This example is a variation of a single neuron on an operational amplifier made on CMOS according to the OP1 scheme described herein. In this scheme, contacts V1 2410 and V2 2408 are the inputs of the single neuron, contact bias 2406 is a voltage of +0.4V relative to GND, contact input Vdd 2402 is a positive supply voltage of +5.0V relative to GND, contact Vss 2404 is GND, and contact Out 2474 is the output of the single neuron. The parameters of the CMOS transistors are determined by the ratio of the geometric dimensions: L (length of the gate channel) and W (width of the gate channel). The Op Amp has two current mirrors. The current mirrors on NMOS transistors M3 2420, M6 2426 and M13 2440 provide compensation currents for the differential pair on NMOS transistors M2 2418 and M5 2424. The current mirror in the PMOS transistors M7 2428, M8 2430 and M15 2444 provides a compensation current for the differential pair on the PMOS transistors M9 2432 and M10 2434. In the first differential amplifier stage, the NMOS transistors M2 2418 and M5 2424 perform amplification, while the PMOS transistors M1 2416 and M4 2422 act as active current loads. The signal is output from the M5 2424 transistor to the PMOS gate of the transistor M13 2440. The signal is output from the M2 2418 transistor to the right input of the second differential amplifier stage on the PMOS transistors M9 2432 and M10 2434. The NMOS transistors M11 2436 and M12 2438 act as active current loads for the M9 2432 and M10 2434 transistors. The M17 2448 transistor is turned on according to the scheme with a common source for the positive half wave of the signal. The M18 2450 transistor is turned on according to the scheme with a common source for the negative half wave of the signal. In order to increase the total load capacity of the Op Amp, the inverters on the M17 2448 and M18 2450 transistors are enabled at the output of the M13 2440 and M14 2442 transistors.

[0514] Fig.24D Shown according to some implementations Figure 24A-24CTable 2476 of the description of the schematic diagram shown. The weights of the connections of a single neuron (with two inputs and one output) are set by the resistor ratios: w1 = (Rp / R1+) - (Rn / R1-); w2 = (Rp / R2+) - (Rn / R2-); wbias = (Rp / Rbias+) - (Rn / Rbias-). The normalizing resistors (Rnorm- and Rnorm+) are necessary to obtain exact equality: (Rn / R1-) + (Rn / R2-) + (Rn / Rbias-) + (Rn / Rnorm-) = (Rp / R1+) + (Rp / R2+) + (Rp / Rbias+) + (Rp / Rnorm+). N-channel MOSFET transistors with explicit substrate connection include transistors M2 and M5 with L=0.36u and W=3.6u, transistors M3, M6, M11, M12, M14 and M16 with L=0.36u and W=1.8u, and transistor M18 with L=0.36u and W=18u. P-channel MOSFET transistors with explicit substrate connection include transistors M1, M4, M7, M8, M13 and M15 with L=0.36u and W=3.96u, transistors M9 and M10 with L=0.36u and W=11.88u, and transistor M17 with L=0.36u and W=39.6u.

[0515] Example scheme for a single neuron OP3CMOSOpAmp

[0516] Figure 25A-Figure 25DA schematic diagram of a variation of a single neuron 25000 on an operational amplifier fabricated on CMOS according to the OP3 scheme according to some implementations is shown. According to some implementations, the single neuron consists of three simple operational amplifiers (OpAmps). The unit neuron adder is implemented on two Opamps with bipolar power supplies, and the RELU activation function is implemented on an OpAmp with a unipolar power supply and a gain of =10. Transistors M1 25028-M16 25058 are used for summing the negative connections of the neuron. Transistors M17 25060-M32 25090 are used to add the positive connections of the neuron. The RELU activation function is implemented on transistors M33 25092-M46 25118. In this scheme, the contacts V1 25008 and V2 25010 are the inputs of the single neuron, the contact Bias 25002 is a voltage of +0.4V relative to GND, the contact Input Vdd 25004 is a positive supply voltage of +2.5V relative to GND, the contact Vss 25006 is a negative supply voltage of -2.5V, and the contact Out 25134 is the output of the single neuron. The parameters of the CMOS transistor used in the single neuron are determined by the ratio of the geometric dimensions: L (the length of the gate channel) and W (the width of the gate channel). Consider the operation of the simplest OpAmp included in the single neuron. Each op amp has two current mirrors. Current mirrors on NMOS transistors M3 25032 (M19 25064, M35 25096), M6 25038 (M22 25070, M38 25102), and M16 25058 (M32 25090, M48 25122) provide compensation current for the differential pair on NMOS transistors M2 25030 (M18 25062, M34 25094) and M5 25036 (M21 25068, M35 25096). Current mirrors in PMOS transistors M7 25040 (M23 25072, M39 25104), M8 25042 (M24 25074, M40 25106) and M15 25056 (M31 2588) provide compensation current for the differential pair on PMOS transistors M9 25044 (M25 25076, M41 25108) and M10 25046 (M26 25078, M42 25110). In the first differential amplifier stage, NMOS transistors M2 25030 (M18 25062, M34 25094) and M5 25036 (M21 25068, M37 25100) perform amplification, and PMOS transistors M1 25028 (M17 25060, M33 25092) and M4 25034 (M20 25066, M36 25098) act as active current loads.The signal is input from transistor M5 25036 (M21 25068, M37 25100) to the PMOS gate of transistor M13 25052 (M29 25084, M45 25116). The signal is input from transistor M2 25030 (M18 25062, M34 25094) to the right input of the second differential amplifier stage on PMOS transistors M9 25044 (M25 25076, M41 25108) and M10 25046 (M26 25078, M42 25110). NMOS transistors M11 25048 (M27 25080, M43 25112) and M12 25048 (M28 25080, M44 25114) act as active current loads for transistors M9 25044 (M25 25076, M41 25108) and M10 25046 (M26 25078, M42 25110). Transistor M13 25052 (M29 25082, M45 25116) is included in the scheme with a common source for the positive half-wave signal. Transistor M14 25054 (M30 25084, M46 25118) is turned on according to the scheme with a common source for the negative half-wave of the signal.

[0517] The weights of the connections of a single neuron (with two inputs and one output) are set by the resistor ratios: w1 = (Rfeedback / R1+) - (Rfeedback / R1-); w2 = (Rfeedback / R2+) - (Rfeedback / R2-); wbias = (Rfeedback / Rbias+) - (Rfeedback / Rbias-); w1 = (Rp*Kamp / R1+) - (Rn*Kamp / R1-); w2 = (Rp*Kamp / R2+) - (Rn*Kamp / R2-); wbias = (Rp*Kamp / Rbias+) - (Rn*Kamp / Rbias-), where Kamp = R1ReLU / R2ReLU. Rfeedback = 100k, used only for calculating w1, w2, wbias. According to some implementations, example values ​​include: Rfeedback=100k, Rn=Rp=Rcom=10k, KampReLU=1+90k / 10k=10, w1=(10k*10 / 22.1k)-(10k*10 / 21.5k)=-0.126276, w2=(10k*10 / 75k)-(10k*10 / 71.5k)=-0.065268, wbias=(10k*10 / 71.5k)-(10k*10 / 78.7k)=0.127953.

[0518] The input of the negative chain adder (M1-M17) of the neuron is received from the positive chain adder (M17-M32) of the neuron through the Rcom resistor.

[0519] Fig.25E Shown according to some implementations Figure 25A-Figure 25D Table 25136 of the description of the schematic shown. N-channel MOSFET transistors with explicit substrate connection include transistors M2, M5, M18, M21, M34 and M37 with L=0.36u and W=3.6u, transistors M3, M6, M11, M12, M14, M16, M19, M22, M27, M28, M32, M38, M35, M38, M43, M44, M46 and M48 with L=0.36u and W=1.8u. P-channel MOSFET transistors with explicit substrate connection include transistors M1, M4, M7, M8, M13, M15, M17, M20, M23, M24, M29, M31, M33, M36, M39, M40, M45 and M47 with L=0.36u and W=3.96u, and transistors M9, M10, M25, M26, M41 and M42 with L=0.36u and W=11.88u. Example method for simulating hardware implementation of trained neural network

[0520] Figure 27A-27J A flow chart of a method 2700 for hardware implementation (2702) of a neural network according to some implementations is shown. The method is performed (2704) at a computing device 200 (e.g., using a neural network transformation module 226) having one or more processors 202 and a memory 214 storing one or more programs configured for execution by the one or more processors 202. The method includes obtaining (2706) a neural network topology (e.g., topology 224) and weights (e.g., weights 222) of a trained neural network (e.g., network 220). In some implementations, the trained neural network is trained (2708) using software simulation to generate weights.

[0521] The method also includes transforming (2710) the neural network topology into an equivalent analog network of analog components. Fig.27CIn some implementations, the neural network topology includes (2724) one or more layers of neurons. Each layer of neurons calculates a corresponding output based on a corresponding mathematical function. In such a case, transforming the neural network topology into an equivalent analog network of analog components includes performing (2726) a series of steps for each layer of one or more layers of neurons. The series of steps includes identifying (2728) one or more function blocks for the corresponding layer based on the corresponding mathematical function. Each function block has a corresponding schematic implementation, which has a block output that conforms to the output of the corresponding mathematical function. In some implementations, identifying one or more function blocks includes selecting (2730) one or more function blocks based on the type of the corresponding layer. For example, a layer may be composed of neurons, and the output of the layer is a linear superposition of its inputs. If the output of the layer is a linear superposition, then selecting one or more function blocks is based on this identification of the layer type or similar pattern recognition. Some implementations determine whether the number of outputs is greater than 1 and then use a trapezoidal or pyramidal transformation.

[0522] Next reference Fig.27D In some implementations, the one or more function blocks include one or more basic function blocks (e.g., basic function block 232) selected (2734) from the group consisting of: (i) having a block output V out =ReLU(∑w i .V i in + bias) is a weighted summation block (2736). ReLU is a rectified linear unit (ReLU) activation function or a similar activation function (e.g., ReLU with threshold), V i represents the i-th input, w i represents the weight corresponding to the i-th input, and bias represents the bias value, and ∑ is a summation operator; (ii) with block output V out =coeff.V i .V j The signal multiplier block (2738) of V i represents the ith input, and V j represents the jth input, and coeff is a predetermined coefficient; (iii) has a block output sigmoid activation block (2740). V represents the input, and A and B are predetermined coefficient values ​​of the sigmoid activation block (e.g., A = -0.1; B = 11.3); (iv) having a block output V out =A*tanh(B*V in )'s hyperbolic tangent activation block (2742), V inrepresents input, and A and B are predetermined coefficient values ​​(e.g., A=0.1, B=-10.1); and a signal delay block (2744) having a block output U(t)=V(t-dt). t represents the current time period, V(t-1) represents the output of the signal delay block of the previous time period t-1, and dt is the delay value.

[0523] Now come back to reference Fig.27C The series of steps also includes generating (2732) a corresponding multilayer network of simulated neurons based on arranging the one or more function blocks. Each simulated neuron implements a corresponding function of the one or more function blocks, and each simulated neuron of a first layer of the multilayer network is connected to one or more simulated neurons of a second layer of the multilayer network.

[0524] Now come back to reference Fig.27A According to some implementations, for some networks, such as GRU and LSTM, transforming (2710) the neural network topology into an equivalent simulated network of simulated components requires more complex processing. Fig.27E , assuming that the neural network topology includes (2746) one or more layers of neurons. Further assuming that each layer of neurons calculates a corresponding output based on a corresponding mathematical function. In such a case, transforming the neural network topology into an equivalent analog network of analog components includes: (i) decomposing (2748) a first layer of the neural network topology into a plurality of sublayers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sublayer implements an intermediate mathematical function. In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting (2750) the one or more weights so that combining the one or more intermediate functions produces the mathematical function; and (ii) performing (2752) a series of steps for each sublayer of the first layer of the neural network topology. The series of steps includes: selecting (2754) one or more subfunction blocks for the corresponding sublayer based on the corresponding intermediate mathematical function; and generating (2756) a corresponding multi-layer analog subnetwork of analog neurons based on arranging the one or more subfunction blocks. Each simulated neuron implements a corresponding function of one or more sub-function blocks, and each simulated neuron of a first layer of the multi-layer simulated sub-network is connected to one or more simulated neurons of a second layer of the multi-layer simulated sub-network.

[0525] Next reference Fig.27H, assuming that the neural network topology includes (2768) one or more GRU or LSTM neurons. In that case, transforming the neural network topology includes generating (2770) one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons. In some implementations, an external loop timer activates one or more signal delay blocks at a constant time period (e.g., 1, 5, or 10 time steps). Some implementations use multiple delay blocks on a signal for generating additional time shifts. In some implementations, the activation frequency of one or more signal delay blocks is synchronized with the network input signal frequency. In some implementations, one or more signal delay blocks are activated (2772) at a frequency that matches a predetermined input signal frequency of the neural network topology. In some implementations, the predetermined input signal frequency can depend on the application, such as human activity recognition (HAR) or PPG. For example, the predetermined input signal frequency for video processing is 30Hz-60Hz, for HAR and PPG is around 100Hz, for sound processing is 16KHz, and for battery management is around 1Hz-3Hz. Some implementations activate different signal delay blocks that activate at different frequencies.

[0526] Next reference Fig.27I , assuming that the neural network topology includes (2774) one or more layers of neurons that implement infinite activation functions. In some implementations, in such cases, transforming the neural network topology includes applying (2776) one or more transformations selected from the group consisting of: replacing (2778) the infinite activation function with a finite activation (e.g., replacing ReLU with a threshold ReLU); and adjusting (2780) the connections or weights of the equivalent simulated network so that for a predetermined one or more inputs, the difference in output between the trained neural network and the equivalent simulated network is minimized.

[0527] Now come back to reference Fig.27A The method further includes calculating (2712) a weight matrix of the equivalent simulation network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection between simulation components of the equivalent simulation network.

[0528] The method also includes generating (2714) a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components. Fig.27BIn some implementations, generating the schematic model includes generating (2716) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix and represents a resistance value. In some implementations, the method includes regenerating the resistance matrix of only the resistors for the retrained network. In some implementations, the method also includes obtaining (2718) new weights for the trained neural network, calculating (2720) a new weight matrix for the equivalent analog network based on the new weights, and generating (2722) a new resistance matrix for the new weight matrix.

[0529] Next reference Fig.27J In some implementations, the method further includes generating (2782) one or more photolithography masks for fabricating a circuit that implements an equivalent analog network of analog components based on the resistance matrix (e.g., using mask generation module 248 to generate masks 250 and / or 252). In some implementations, the method includes regenerating a mask (e.g., mask 250) for only resistors for the retrained network. In some implementations, the method further includes: (i) obtaining (2784) new weights for the trained neural network; (ii) calculating (2786) a new weight matrix for the equivalent analog network based on the new weights; (iii) generating (2788) a new resistance matrix for the new weight matrix; and (iv) generating (2790) a new photolithography mask for fabricating a circuit that implements an equivalent analog network of analog components based on the new resistance matrix.

[0530] Now come back to reference Figure 27G , the analog components include (2762) multiple operational amplifiers and multiple resistors. Each operational amplifier represents a simulated neuron of the equivalent analog network, and each resistor represents a connection between two simulated neurons. Some implementations include other analog components, such as a four-quadrant multiplier, a sigmoid function circuit and a hyperbolic tangent function circuit, a delay line, a summer, and / or a voltage divider. In some implementations, selecting (2764) component values ​​of the analog components includes executing (2766) a gradient descent method and / or other weight quantization method to identify possible resistance values ​​of the multiple resistors.

[0531] Now come back to reference Fig.27F In some implementations, the method further includes digitally implementing certain activation functions (e.g., Softmax) in the output layer. In some implementations, the method further includes generating (2758) an equivalent digital network of digital components for one or more output layers of the neural network topology, and connecting (2760) the outputs of one or more layers of the equivalent analog network to the equivalent digital network of digital components.

[0532] An example approach for constrained analog hardware implementation of neural networks

[0533] Figure 28A-28SA flow chart of a method 28000 for hardware implementation (28002) of a neural network according to hardware design constraints is shown according to some implementations. The method is performed (28004) at a computing device 200 (e.g., using a neural network transformation module 226) having one or more processors 202 and a memory 214 storing one or more programs configured for execution by the one or more processors 202. The method includes obtaining (28006) a neural network topology (e.g., topology 224) and weights (e.g., weights 222) of a trained neural network (e.g., network 220).

[0534] The method also includes calculating (28008) one or more connection constraints based on analog integrated circuit (IC) design constraints (e.g., constraints 236). For example, the IC design constraints may set a current limit (e.g., 1A), and the neuron circuit diagram and operational amplifier (OpAmp) design may set the OpAmp output current to be within the range [0-10mA], so this limits the output neuron connections to 100. This means that the neuron has 100 outputs, which allows current to flow through 100 connections to the next layer, but the current at the output of the operational amplifier is limited to 10mA, so some implementations use a maximum of 100 outputs (0.1mA times 100=10mA). In the absence of this constraint, some implementations use current repeaters to increase the number of outputs to, for example, more than 100.

[0535] The method also includes transforming (28010) the neural network topology (e.g., using the neural network transformation module 226) into an equivalent sparsely connected network of simulated components that satisfies the one or more connectivity constraints.

[0536] In some implementations, transforming the neural network topology includes deriving (28012) possible input connectivity N according to one or more connection constraints. i and output connectivity N o .

[0537] Next reference Fig.28B In some implementations, the neural network topology includes (28018) at least one densely connected layer having K inputs (neurons in a previous layer) and L outputs (neurons in a current layer) and a weight matrix U, and transforming (28020) the at least one densely connected layer includes constructing (28022) a neural network topology having K inputs, L outputs, and The equivalent sparsely connected network of layers makes the input connectivity no more than N i , and the output connectivity does not exceed N o .

[0538] Next reference Fig.28C In some implementations, the neural network topology includes (28024) at least one densely connected layer having K inputs (neurons in a previous layer) and L outputs (neurons in a current layer) and a weight matrix U, and transforming (28026) the at least one densely connected layer includes: constructing (28028) a neural network topology having K inputs, L outputs, and Each layer m is represented by the corresponding weight matrix U m Represented by , where missing connections are represented by zeros, so that the input connectivity does not exceed N i , and the output connectivity does not exceed N o The equation U = ∏ m＝1..M U m The predetermined accuracy is a reasonable accuracy value that statistically guarantees that the output of the modified network does not differ from the output of the referent network by more than an allowed error value that is task-dependent (usually between 0.1% and 1%).

[0539] Next reference Fig.28D In some implementations, the neural network topology includes (28030) having K inputs and L outputs, a maximum input connectivity P i , maximum output connectivity P o and a single sparsely connected layer with a weight matrix U, where missing connections are represented by zeros. In such a case, transforming (28032) the single sparsely connected layer includes constructing (28034) a network with K inputs, L outputs, Each layer m is represented by the corresponding weight matrix U m Represented by , where missing connections are represented by zeros, so that the input connectivity does not exceed N i , and the output connectivity does not exceed N o , and the equation U = ∏ m＝1..M U m is met with a predetermined accuracy.

[0540] Next reference Fig.28E In some implementations, the neural network topology includes (28036) a convolutional layer (e.g., a depthwise convolutional layer or a separable convolutional layer) having K inputs (neurons in the previous layer) and L outputs (neurons in the current layer). In such a case, transforming (28038) the neural network topology into an equivalent sparsely connected network of analog components includes decomposing (28040) the convolutional layer into a network having K inputs, L outputs, a maximum input connectivity P, and a maximum input connectivity P. i and the maximum output connectivity P o A single sparsely connected layer, where P i ≤N iand P o ≤N o .

[0541] Come back for reference Fig.28A The method also includes calculating (28014) a weight matrix of an equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection between simulated components of the equivalent sparsely connected network.

[0542] Reference now Fig.28F In some implementations, the neural network topology includes (28042) a recurrent neural layer, and transforming (28044) the neural network topology into an equivalent sparsely connected network of the analog component includes transforming (28046) the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections.

[0543] Next reference Figure 28G In some implementations, the neural network topology includes a recurrent neural layer (e.g., a long short-term memory (LSTM) layer or a gated recurrent unit (GRU) layer), and transforming the neural network topology into an equivalent sparsely connected network of the analog component includes decomposing the recurrent neural layer into several layers, wherein at least one of the layers is equivalent to a densely or sparsely connected layer having K inputs (neurons in a previous layer) and L outputs (neurons in a current layer) and a weight matrix U, wherein missing connections are represented by zeros.

[0544] Next reference Fig.28H In some implementations, the method includes performing a transformation of a single-layer perceptron having one computational neuron. In some implementations, the neural network topology includes (28054)K inputs, a weight vector U∈R K and a single layer perceptron having computational neurons having an activation function F. In such a case, transforming (28056) the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving (28058) a connectivity N of the equivalent sparsely connected network based on one or more connectivity constraints; (ii) using equation to calculate (28060) the number of layers m of the equivalent sparsely connected network; and (iii) construct (28062) an equivalent sparsely connected network with K inputs, m layers and connectivity N. The equivalent sparsely connected network includes one or more corresponding simulated neurons in each of the m layers. Each simulated neuron of the first m-1 layers implements the identity transformation, and the simulated neuron of the last layer implements the activation function F of the computational neuron of the single-layer perceptron. In addition, in such a case, calculating (28064) the weight matrix of the equivalent sparsely connected network includes calculating (28066) the weight vector W of the connection of the equivalent sparsely connected network by solving the system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S uses equations is calculated.

[0545] Next reference Fig.28I , in some implementations, the method includes performing a transformation of a single layer perceptron having L computational neurons. In some implementations, the neural network topology includes (28068) K inputs, a single layer perceptron having L computational neurons, and a weight matrix V including a row of weights for each of the L computational neurons. In such a case, transforming (28070) the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving (28072) a connectivity N of the equivalent sparsely connected network based on one or more connectivity constraints; (ii) using equation to calculate (28074) the number of layers m of the equivalent sparsely connected network; (iii) decomposing (28076) the single-layer perceptron into L single-layer perceptron networks. Each single-layer perceptron network includes a corresponding computational neuron among the L computational neurons; (iv) for each single-layer perceptron network (28078) in the L single-layer perceptron networks, constructing (28080) a corresponding equivalent pyramidal subnetwork with K inputs, m layers and connectivity N for the corresponding single-layer perceptron network. The equivalent pyramidal subnetwork includes one or more corresponding simulated neurons in each of the m layers, each simulated neuron in the first m-1 layers implements the identity transformation, and the simulated neuron in the last layer implements the activation function corresponding to the corresponding computational neuron of the corresponding single-layer perceptron; and (v) constructing (28082) an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork together, including connecting the inputs of each equivalent pyramidal subnetwork of the L single-layer perceptron networks together to form an input vector with L*K inputs. Furthermore, in such a case, computing (28084) the weight matrix of the equivalent sparsely connected network includes, for each of the L single-layer perceptron networks (28086): (i) setting (28088) the weight vector U = V i, the i-th row of the weight matrix V corresponds to the corresponding computational neuron, the corresponding computational neuron corresponds to the corresponding single-layer perceptron network; and (ii) calculating (28090) the weight vector W of the connection of the corresponding equivalent pyramidal subnetwork by solving the system of equations based on the weight vector U i The system of equations consists of K equations with S variables, and S uses equations is calculated.

[0546] Next reference Fig.28J In some implementations, the method includes performing a transformation algorithm on a multilayer perceptron. In some implementations, the neural network topology includes (28092)K inputs, a multilayer perceptron having S layers, each layer i of the S layers including a set of corresponding computational neurons L i And the corresponding weight matrix V i , weight matrix V i Including L i A row of weights for each computational neuron in the computational neurons. In this case, transforming (28094) the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving (28096) the connectivity N for the equivalent sparsely connected network based on one or more connection constraints; (ii) decomposing (28098) the multilayer perceptron into Q = ∑ i＝1,S (L i ) single-layer perceptron networks. Each single-layer perceptron network includes a corresponding computational neuron among the Q computational neurons. Decomposing the multilayer perceptron includes duplicating one or more of the K inputs shared by the Q computational neurons; (iii) for each single-layer perceptron network (28100) among the Q single-layer perceptron networks: (a) using equation Calculate (28102) the number of layers m of the corresponding equivalent pyramid subnetwork. K i,j is the number of inputs to the corresponding computing neuron in the multilayer perceptron; and (b) for the corresponding single-layer perceptron network construction (28104) with K i,j The equivalent pyramidal subnetwork has a corresponding equivalent pyramidal subnetwork with inputs, m layers and a connectivity of N. The equivalent pyramidal subnetwork includes one or more corresponding simulated neurons in each of the m layers, each simulated neuron in the first m-1 layers implements the identity transformation, and the simulated neuron in the last layer implements the activation function corresponding to the corresponding computational neuron of the corresponding single-layer perceptron network; and (iv) constructing (28106) an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork together, including connecting the inputs of each equivalent pyramidal subnetwork of the Q single-layer perceptron networks together to form a sparsely connected network with Q*K i,jIn this case, computing (28108) the weight matrix of the equivalent sparsely connected network includes: for each of the Q single-layer perceptron networks (28110): (i) setting (28112) the weight vector U = V i j , the i-th row of the weight matrix V corresponds to the corresponding computational neuron, the corresponding computational neuron corresponds to the corresponding single-layer perceptron network, where j is the corresponding layer of the corresponding computational neuron in the multilayer perceptron; and (ii) calculating (28114) the weight vector W of the connection of the corresponding equivalent pyramidal subnetwork by solving the system of equations based on the weight vector U i The system of equations consists of K with S variables i,j equations, and is calculated.

[0547] Next reference Figure 28K In some implementations, the neural network topology includes (28116) a convolutional neural network (CNN) having K inputs and S layers, each layer i of the S layers including a set of corresponding computational neurons L i And the corresponding weight matrix V i , weight matrix V i Including for L i A row of weights for each computational neuron in the computational neurons. In this case, transforming (28118) the neural network topology into an equivalent sparsely connected network of the analog component includes: (i) deriving (28120) the connectivity N for the equivalent sparsely connected network based on one or more connection constraints; (ii) decomposing (28122) the CNN into Q = ∑ i＝1,S (L i ) single-layer perceptron networks. Each single-layer perceptron network includes a corresponding computational neuron among the Q computational neurons. Decomposing the CNN includes replicating one or more of the K inputs shared by the Q computational neurons; (iii) for each of the Q single-layer perceptron networks: (a) using equation Calculate the number of layers m of the corresponding equivalent pyramidal subnetwork. j is the corresponding layer of the corresponding computational neuron in the CNN, and K i,j is the number of inputs to the corresponding computational neurons in the CNN; and (b) constructing a K i,jThe equivalent pyramidal subnetwork has a corresponding equivalent pyramidal subnetwork with inputs, m layers and a connectivity of N. The equivalent pyramidal subnetwork includes one or more corresponding simulated neurons in each of the m layers, each simulated neuron in the first m-1 layers implements the identity transformation, and the simulated neuron in the last layer implements the activation function corresponding to the corresponding computational neuron of the corresponding single-layer perceptron network; and (iv) constructing (28130) an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork together, including connecting the inputs of each equivalent pyramidal subnetwork of the Q single-layer perceptron networks together to form a sparsely connected network with Q*K i,j In this case, computing (28132) the weight matrix of the equivalent sparsely connected network includes for each of the Q single-layer perceptron networks (28134): (i) setting the weight vector U = V i j , the i-th row of the weight matrix V corresponds to the corresponding computational neuron, the corresponding computational neuron corresponds to the corresponding single-layer perceptron network, where j is the corresponding layer of the corresponding computational neuron in the CNN; and (ii) calculating the weight vector W of the connection of the corresponding equivalent pyramidal subnetwork by solving the system of equations based on the weight vector U i The system of equations consists of K with S variables i,j equations, and is calculated.

[0548] Next reference Figure 28L In some implementations, the method includes transforming the two layers into a ladder-based network. In some implementations, the neural network topology includes (28140) K inputs, a layer L with K neurons p , layer L with L neurons n and the weight matrix W∈R L×K , where R is the set of real numbers, layer L p Each neuron is connected to layer L n Each neuron of layer L n Each neuron in layer L performs the activation function F, so that n The output for input x using equation Y o =F(Wx) is calculated. In such a case, transforming (28142) the neural network topology into an equivalent sparsely connected network of analog components includes performing a trapezoidal transformation, which includes: (i) deriving (28144) possible input connectivity N according to one or more connection constraints I >1 and possible output connectivity N O >1; and (ii) according to K·L <L·N I +K·N ODetermine, construct (28146) a layer LA including K simulated neurons that implement the identity activation function p , with the same activation function Layer LA of simulated neurons h and a layer LA with L simulated neurons implementing the activation function F o The three-layer simulation network makes the LA p Each simulated neuron in has N O output, in layer LA h Each simulated neuron in has no more than N I input and N O outputs, and at layer LA o Each simulated neuron in has N I In some such cases, computing (28148) the weight matrix of the equivalent sparsely connected network includes applying the matrix equation W o .W h =W is solved to generate (2850) the sparse weight matrix W o and W h , the matrix equation W o .W h =W includes the use of K·N O +L·N I K·L equations of variables, so that the layer LA o The total output of Y is calculated using the equation o =F(W o .W h .x) is calculated. The sparse weight matrix W o ∈R K×M Indicates that in layer LA p and LA h The connection between them, and the sparse weight matrix W h ∈R M×L Indicates that in layer LA h and LA o The connection between.

[0549] Next reference Figure 28M In some implementations, performing the trapezoidal transformation further comprises: according to K·L≥L·M I +K·N o Determination of: (i) Make layer L p Split (28154) to obtain a sublayer L with K' neurons p1 and a sublayer L with (K-K') neurons p2 , so that K′·L ≥ L·N I +K′·N o ; (ii) For a sublayer L with K' neuronsp1 , performing (28156) construction and generation steps; and (iii) for a sublayer L having K-K' neurons p2 , cyclically executing (28158) split, build, and generate steps.

[0550] Next reference Fig.28N , the method includes transforming the multilayer perceptron into a trapezoidal-based network. In some implementations, the neural network topology includes (28160) a multilayer perceptron network. The method also includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing (28162) a trapezoidal transformation and calculating a weight matrix for an equivalent sparsely connected network.

[0551] Next reference Fig.28O , the method includes transforming a recurrent neural network into a trapezoid-based network. In some implementations, the neural network topology includes (28164) a recurrent neural network (RNN) that includes (i) computation of linear combinations of two fully connected layers, (ii) element-wise addition, and (iii) nonlinear function computation. In such a case, the method also includes performing (28166) a trapezoidal transformation and computing a weight matrix of an equivalent sparsely connected network for (i) two fully connected layers, and (ii) nonlinear function computation. Element-wise addition is a common operation that can be implemented in networks of any structure, examples of which are provided above. Nonlinear function computation is an operation on neurons that is independent of the No and Ni restrictions, and is typically computed on each neuron individually using a "sigmoid" or "tanh" block.

[0552] Next reference Figure 28P , the neural network topology includes (28168) a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network, which includes (i) calculation of linear combinations of multiple fully connected layers, (ii) element-by-element addition, (iii) Hadamard products, and (iv) multiple nonlinear function calculations (sigmoid and hyperbolic tangent operations). In such a case, the method also includes performing (28170) a trapezoidal transformation and calculating a weight matrix of an equivalent sparsely connected network for (i) multiple fully connected layers, and (ii) multiple nonlinear function calculations. Element-by-element addition and Hadamard products are common operations that can be implemented in any of the structures described above.

[0553] Next reference Figure 28Q, the neural network topology includes (28172) a convolutional neural network (CNN) comprising (i) a plurality of partially connected layers (e.g., a sequence of convolutional and pooling layers; each pooling layer is assumed to be a convolutional layer with a stride greater than 1) and (ii) one or more fully connected layers (the sequence ends in a fully connected layer). In such a case, the method further includes: (i) transforming (28174) the plurality of partially connected layers into equivalent fully connected layers by inserting missing connections with zero weights; and for each pair of consecutive layers of the equivalent fully connected layer and the one or more fully connected layers, iteratively performing (28176) a trapezoidal transformation and calculating a weight matrix of the equivalent sparsely connected network.

[0554] Next reference Figure 28R In some implementations, the neural network topology includes (28178)K input, L output neurons and a weight matrix U∈R L×K , where R is a set of real numbers and each output neuron performs an activation function F. In such a case, transforming (28180) the neural network topology into an equivalent sparsely connected network of analog components includes performing an approximate transformation that includes: (i) deriving (28182) possible input connectivity N based on one or more connection constraints I >1 and possible output connectivity N o >1; (ii) from the set (iii) based on the determination of p>0, construct (28186) a pyramid neural network of the first p layers of the equivalent sparsely connected network, so that the pyramid neural network has in its output layer Each neuron in the pyramid neural network performs the same function; and (iv) construct (28188) with N p inputs and L outputs. Each neuron in the last layer of the ladder neural network performs an activation function F, and all other neurons perform an identity function. In addition, in such a case, computing (28190) the weight matrix of the equivalent sparsely connected network includes: (i) generating (28192) the weights of the pyramid neural network, including (i) setting the weight of each neuron i in the first layer of the pyramid neural network according to the following rule: (a) C is a non-zero constant, and k i =(i-1)N I +1; and (b) for all k i All weights j of neurons other than and (ii) setting all other weights of the pyramid neural network to 1; and (ii) generating the weights of the (28194) ladder neural network, including (i) according to equation Setting the weight of each neuron i in the first layer of the ladder neural network (considering the entire network, this is the (p+1)th layer); and (ii) setting the other weights of the ladder neural network to 1.

[0555] Next reference Figure 28S In some implementations, the neural network topology includes (28196) having K inputs, S layers, and L in the i-th layer i＝1,S The computational neurons and the weight matrix of the i-th layer A multilayer perceptron, where L 0 = K. In this case, transforming (28198) the neural network topology into an equivalent sparsely connected network of analog components includes: for each layer j (28200) of the S layers of the multilayer perceptron: (i) by transforming the L j-1 Input, L j output neurons and weight matrix U j The corresponding single-layer perceptron composed of the corresponding pyramid-ladder network PTNNX performs approximate transformation to construct (28202) j ; and (ii) constructing (28204) an equivalent sparsely connected network by stacking each pyramid ladder network (e.g., the output of pyramid ladder network PTNNXj-1 is set as the input of PTNNXj).

[0556] Come back for reference Fig.28A In some implementations, the method further includes generating (28016) a schematic model for implementing an equivalent sparsely connected network using a weight matrix.

[0557] Example method for calculating resistor values ​​for analog hardware implementation of trained neural network

[0558] Figure 29A-29F A flowchart of a method 2900 for hardware implementation (2902) of a neural network according to hardware design constraints is shown according to some implementations. The method is performed (2904) at a computing device 200 (e.g., using a weight quantization module 238) having one or more processors 202 and a memory 214 storing one or more programs configured for execution by the one or more processors 202.

[0559] The method includes obtaining (2906) a neural network topology (e.g., topology 224) and weights (e.g., weights 222) of a trained neural network (e.g., network 220). In some implementations, weight quantization is performed during training. In some implementations, the trained neural network is trained (2908) such that each layer of the neural network topology has quantized weights (e.g., specific values ​​from a list of discrete values; e.g., each layer has only three weight values ​​of +1, 0, and -1).

[0560] The method also includes transforming (2910) the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (e.g., using neural network transformation module 226). Each operational amplifier represents a simulated neuron of the equivalent analog network, and each resistor represents a connection between two simulated neurons.

[0561] The method also includes calculating (2912) a weight matrix of an equivalent simulated network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection.

[0562] The method also includes generating (2914) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix and represents a resistance value.

[0563] Next reference Fig.29B In some implementations, generating a resistance matrix for the weight matrix includes an iterative method based on simplified gradient descent to find a set of resistors. In some implementations, generating a resistance matrix for the weight matrix includes: (i) obtaining (2916) a predetermined range of possible resistance values ​​{R min ,R max} and select an initial basic resistance value R within a predetermined range 基本 . For example, the range and base resistors are selected based on the values ​​of the elements of the weight matrix; the values ​​are determined by the manufacturing process; range - resistors that can actually be manufactured; large resistors are not preferred; quantification of what can actually be manufactured. In some implementations, the predetermined range of possible resistance values ​​includes (2918) resistors in the range 100KΩ to 1MΩ according to the nominal series E24; (ii) selecting (2920) a finite length set of resistance values ​​within the predetermined range, the resistance value for {R i ,R j All combinations of} are provided in the range [-R 基本 ,R 基本 ] possible weights within In some implementations, the weight values ​​are outside this range, but the squared average distance between weights within this range is the minimum; (iii) the maximum weight and bias w based on the incoming connections of each neuron or equivalently each layer of the simulated network max , for each simulated neuron or for each layer of the equivalent simulated network, select (2922) resistor values ​​R from a finite length set of resistor values + =R - , so that R + =R - is closest to R 基本 *wmax In some implementations, R is independently selected (2924) for each layer of the equivalent analog network. + and R - In some implementations, R is independently selected (2926) for each simulated neuron of the equivalent simulated network. + and R - ; and (iv) for each element of the weight matrix, R within a predetermined range of possible resistance values 1 and R 2 Among all possible values ​​of , choose (2928) according to the equation The corresponding first resistance value R that minimizes the error 1 and the corresponding second resistance value R 2 . w is the corresponding element of the weight matrix, and r err is a predetermined relative tolerance value of possible resistance values.

[0564] Next reference Fig.29C , some implementations perform weight reduction. In some implementations, the first one or more weights and the first one or more inputs of the weight matrix represent (2930) one or more connections to a first operational amplifier of the equivalent analog network. The method also includes: before generating (2932) the resistor matrix, (i) modifying (2934) the first one or more weights by a first value (e.g., dividing the first one or more weights by the first value to reduce the weight range, or multiplying the first one or more weights by the first value to increase the weight range); and (ii) configuring (2936) the first operational amplifier before performing the activation function so that a linear combination of the first one or more weights and the first one or more inputs is multiplied by the first value. Some implementations perform weight reduction so as to change the multiplication factor of one or more operational amplifiers. In some implementations, the set of resistor values ​​produces a range of weights, and in some parts of the range, the error will be higher than in other parts. Assuming there are only 2 nominal quantities (e.g., 1Ω and 4Ω), these resistors can produce weights [-3; -0.75; 0; 0.75; 3]. Assuming that the first layer of the neural network has weights of {0,9}, and the second layer has weights of {0,1}, some implementations divide the weights of the first layer by 3 and multiply the weights of the second layer by 3 to reduce the overall error. Some implementations consider limiting the weight values ​​during training by adjusting the loss function (e.g., using an l1 or l2 regularizer) so that the resulting network does not have weights that are too large for the resistor set.

[0565] Next reference Fig.29D, the method further includes restricting the weights to an interval. For example, the method further includes obtaining (2938) a predetermined range of weights; and updating (2940) the weight matrix according to the predetermined range of weights so that for the same input, the equivalent simulated network produces an output similar to the trained neural network.

[0566] Next reference Fig.29E , the method also includes reducing the weight sensitivity of the network. For example, the method also includes retraining (2942) the trained neural network to reduce sensitivity to errors in weights or resistor values ​​that cause an equivalent analog network to produce different outputs than the trained neural network. In other words, some implementations include additional training of an already trained neural network to make it less sensitive to small randomly distributed weight errors. Quantization and resistor manufacturing produce small weight errors. Some implementations transform the network so that the resulting network is less sensitive to each specific weight value. In some implementations, this is performed by adding small relatively random values ​​to each signal in at least some layers during training (e.g., similar to a dropout layer).

[0567] Next reference Fig.29F , some implementations include reducing the weight distribution range. Some implementations include retraining (2944) the trained neural network to minimize the weight in any layer that is greater than the average absolute weight of the layer by a predetermined threshold. Some implementations perform this step via retraining. Example penalty functions include the sum over all layers (e.g., A*max(abs(w)) / mean(abs(w)), where max and mean are calculated on one layer. Another example includes higher orders of magnitude. In some implementations, the function affects weight quantization and network weight sensitivity. For example, small relative changes in weights due to quantization may result in high output errors. Example techniques include introducing some penalty functions during training that penalize the network when it has such weight outcasts.

[0568] Example method for optimizing simulated hardware implementation of trained neural network

[0569] Figure 30A-30M A flowchart of a method 3000 of hardware implementation (3002) of a neural network according to hardware design constraints according to some implementations is shown. The method is performed (3004) at a computing device 200 (e.g., using a simulated neural network optimization module 246) having one or more processors 202 and a memory 214 storing one or more programs configured for execution by the one or more processors 202.

[0570] The method includes obtaining (3006) a neural network topology (eg, topology 224) and weights (eg, weights 222) of a trained neural network (eg, network 220).

[0571] The method also includes transforming (3008) the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (e.g., using neural network transformation module 226). Each operational amplifier represents a simulated neuron of the equivalent analog network, and each resistor represents a connection between two simulated neurons.

[0572] Next reference Figure 30L In some implementations, the method further comprises pruning the trained neural network. In some implementations, the method further comprises pruning (3052) the trained neural network using a neural network pruning technique to update the neural network topology and weights of the trained neural network before transforming the neural network topology so that the equivalent simulated network includes less than a predetermined number of simulated components. In some implementations, the pruning is performed (3054) iteratively taking into account the accuracy or level of matching of outputs between the trained neural network and the equivalent simulated network.

[0573] Next reference Figure 30M In some implementations, the method further includes performing (3056) network knowledge extraction before transforming the neural network topology into an equivalent analog network. Knowledge extraction is different from random / learning like pruning, but is more deterministic than pruning. In some implementations, knowledge extraction is performed independently of the pruning step. In some implementations, before transforming the neural network topology into an equivalent analog network, the connection weights are adjusted according to a predetermined optimality criterion (e.g., zero weights or weights within a specific range are preferred over other weights) by deriving causal relationships between the inputs and outputs of hidden neurons through a method of knowledge extraction. Conceptually, in a single neuron or a collection of neurons, on a particular data set, there may be causal relationships between inputs and outputs that allow the weights to be readjusted in such a way that: (1) the new set of weights produces the same network output, and (2) the new set of weights is easier to implement with resistors (e.g., more evenly distributed values, more zero values, or no connections). For example, if a neuron output is always 1 on a certain data set, some implementations remove the output connection of that neuron (and the neuron as a whole) and instead adjust the bias weights of the neurons that follow that neuron. In this way, the knowledge extraction step differs from pruning because pruning requires relearning after removing neurons, and the learning is stochastic, while knowledge extraction is deterministic.

[0574] Come back for reference Fig. 30AThe method further comprises calculating (3010) a weight matrix of an equivalent simulation network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection.

[0575] Next reference Fig. 30J In some implementations, the method further includes removing or transforming neurons based on the bias value. In some implementations, the method further includes: for each simulated neuron of the equivalent simulated network: (i) calculating (3044) a corresponding bias value of the corresponding simulated neuron based on the weights of the trained neural network when calculating the weight matrix; (ii) removing (3046) the corresponding simulated neuron from the equivalent simulated network based on a determination that the corresponding bias value is above a predetermined maximum bias threshold; and (iii) replacing (3048) the corresponding simulated neuron with a linear junction in the equivalent simulated network based on a determination that the corresponding bias value is below a predetermined minimum bias threshold.

[0576] Next reference Figure 30K In some implementations, the method further includes minimizing the number of neurons or making the network compact. In some implementations, the method further includes reducing (3050) the number of neurons of the equivalent simulated network by increasing the number of connections (inputs and outputs) from one or more simulated neurons of the equivalent simulated network before generating the weight matrix.

[0577] Come back for reference Fig. 30A The method further includes generating (3012) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix.

[0578] The method also includes trimming (3014) the equivalent analog network based on the resistor matrix to reduce the number of multiple operational amplifiers or multiple resistors to obtain an optimized analog network of the analog components.

[0579] Next reference Fig. 30B In some implementations, the method includes replacing worthless resistors with conductors. In some implementations, pruning the equivalent analog network includes replacing (3016) the following resistors with conductors, the resistors corresponding to one or more elements of the resistor matrix having a resistance value below a predetermined minimum threshold resistance value.

[0580] Next reference Fig. 30C In some implementations, the method further comprises removing connections having very high resistance. In some implementations, pruning the equivalent analog network comprises removing (3018) one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix having resistance values ​​above a predetermined maximum threshold.

[0581] Next reference Fig.30DIn some implementations, pruning the equivalent simulated network includes removing (3020) one or more connections of the equivalent simulated network corresponding to one or more elements of the weight matrix that are approximately zero. In some implementations, pruning the equivalent simulated network also includes removing (3022) one or more simulated neurons of the equivalent simulated network that do not have any input connections.

[0582] Next reference Fig.30E , in some implementations, the method includes removing unimportant neurons. In some implementations, pruning the equivalent simulated network includes: (i) ranking (3024) simulated neurons of the equivalent simulated network based on detecting the use of the simulated neurons when calculating one or more data sets. For example, the training data set is used to train the trained neural network, the typical data set, and the data set developed for the pruning program. When subjected to the training data set, some implementations perform ranking of neurons for pruning based on the frequency of use of a given neuron or neuron block. For example, (a) when using a test data set, if there is never a signal at a given neuron, it means that the neuron or neuron block has never been in use and is pruned; (b) if the frequency of use of the neuron is very low, the neuron is pruned without a significant loss of accuracy; and (c) the neuron is always in use, then the neuron cannot be pruned; (ii) selecting (3026) one or more simulated neurons of the equivalent simulated network based on the ranking; and (iii) removing (3028) one or more simulated neurons from the equivalent simulated network.

[0583] Next reference Fig.30F In some implementations, detecting the use of simulated neurons includes: (i) using modeling software (e.g., SPICe or similar software) to build (3030) a model of an equivalent simulated network; and (ii) measuring (3032) the propagation of the simulated signal (current) by using the model (removing blocks where the signal does not propagate when a particular training set is used) to generate calculations for one or more data sets.

[0584] Next reference Figure 30G In some implementations, detecting the use of simulated neurons includes: (i) using modeling software (e.g., SPICe or similar software) to build (3034) a model of an equivalent simulated network; and (ii) measuring (3036) an output signal (current or voltage) of the model by using the model to generate calculations for one or more data sets (e.g., signals at the output of some blocks or amplifiers in the SPICe model or in a real circuit, and removing regions where the output signal of the training set is always zero volts).

[0585] Next reference Fig. 30HIn some implementations, detecting the use of simulated neurons includes: (i) using modeling software (e.g., SPICe or similar software) to build (3038) a model of an equivalent simulated network; and (ii) measuring (3040) the power consumed by the simulated neurons (e.g., the power consumed by certain neurons or blocks of neurons represented by operational amplifiers in a SPICE model or in a real circuit, and removing neurons or blocks of neurons that do not consume any power) by using the model to generate calculations for one or more data sets.

[0586] Next reference Fig. 30I In some implementations, the method further includes recalculating (3042) a weight matrix of the equivalent analog network after pruning the equivalent analog network and before generating one or more photolithography masks for manufacturing a circuit implementing the equivalent analog network, and updating the resistance matrix based on the recalculated weight matrix.

[0587] Example analog neuromorphic integrated circuits and methods of fabrication

[0588] Example Method for Fabricating Analog Integrated Circuits for Neural Networks

[0589] Figure 31A-Figure 31Q A flow chart of a method 3100 for fabricating an integrated circuit 3102 including an analog network of analog components according to some implementations is shown. The method is performed at a computing device 200 (e.g., using an integrated circuit fabrication module 258) having one or more processors 202 and a memory 214 storing one or more programs configured for execution by the one or more processors 202. The method includes obtaining (3104) a neural network topology and weights of a trained neural network.

[0590] The method also includes transforming (3106) the neural network topology into an equivalent analog network of analog components (for a recurrent neural network, signal delay lines, multipliers, Tanh analog blocks, Sigmoid analog blocks are also used) (e.g., using the neural network transformation module 226) including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents a corresponding analog neuron, and each resistor represents a corresponding connection between a corresponding first analog neuron and a corresponding second analog neuron.

[0591] The method also includes calculating (3108) a weight matrix of an equivalent simulated network based on the weights of the trained neural network. Each element of the weight matrix represents a corresponding connection.

[0592] The method also includes generating (3110) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a corresponding weight of the weight matrix.

[0593] The method also includes generating (3112) one or more photolithography masks for manufacturing a circuit that implements an equivalent analog network of analog components based on the resistance matrix (e.g., using mask generation module 248 to generate masks 250 and / or 252), and manufacturing (3114) a circuit (e.g., IC 262) based on the one or more photolithography masks using a photolithography process.

[0594] Next reference Fig.31B In some implementations, the integrated circuit also includes one or more digital-to-analog converters (3116) (e.g., DAC converter 260) configured to generate analog inputs for an equivalent analog network of analog components based on one or more digital signals (e.g., signals from one or more CCD / CMOS image sensors).

[0595] Next reference Fig.31C In some implementations, the integrated circuit also includes an analog signal sampling module (3118) configured to process one-dimensional or two-dimensional analog inputs at a sampling frequency based on the number of inferences of the integrated circuit (the number of inferences of the IC is determined by the product specifications, and we know the sampling rate from the neural network operations and the exact tasks that the chip is expected to solve).

[0596] Next reference Fig.31D In some implementations, the integrated circuit also includes a voltage converter module (3120) that scales down or scales up the analog signal to match the operating range of the plurality of operational amplifiers.

[0597] Next reference Fig.31E In some implementations, the integrated circuit further includes a tactile signal processing module (3122) configured to process one or more frames obtained from the CCD camera.

[0598] Next reference Fig.31F In some implementations, the trained neural network is a long short-term memory (LSTM) network, and the integrated circuit also includes one or more clock modules to synchronize the signal haptics and enable time series processing.

[0599] Next reference Figure 31G In some implementations, the integrated circuit also includes one or more analog-to-digital converters (3126) (eg, ADC converter 260) configured to generate digital signals based on outputs of an equivalent analog network of analog components.

[0600] Next reference Fig.31H In some implementations, the integrated circuit includes one or more signal processing modules (3128) configured to process one-dimensional or two-dimensional analog signals obtained from an edge application.

[0601] Next reference Fig.31I , using a training data set containing signals from an array of gas sensors (e.g., 2 to 25 sensors) for different gas mixtures to train (3130) a trained neural network for selectively sensing different gases in a gas mixture containing a predetermined amount of gas to be detected (in other words, the trained chip operates to individually determine each of the known neural network gases in the gas mixture despite the presence of other gases in the mixture). In some implementations, the neural network topology is a one-dimensional deep convolutional neural network (1D-DCNN) designed to detect 3 binary gas components based on measurements from 16 gas sensors, and includes (3132) 16 sensor-based 1D convolutional blocks, 3 shared or common 1D convolutional blocks, and 3 dense layers. In some implementations, the equivalent simulated network includes (3134): (i) up to 100 input and output connections per simulated neuron, (ii) delay blocks that produce delays of any number of time steps, (iii) signals limited to 5, (iv) 15 layers, (v) approximately 100,000 simulated neurons, and (vi) approximately 4,900,000 connections.

[0602] Next reference Fig.31J , using a training dataset containing thermal aging time series data for different MOSFETs (e.g., a NASAMOSFET dataset containing thermal aging time series for 42 different MOSFETs; the data is sampled every 400 ms and generally includes several hours of data for each device) to train (3136) a trained neural network for predicting the remaining useful life (RUL) of the MOSFET devices. In some implementations, the neural network topology includes (3138) 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively. In some implementations, the equivalent simulated network includes (3140): (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3000 and 3200 simulated neurons (e.g., 3137 simulated neurons), and (v) between 123000 and 124000 connections (e.g., 123,200 connections).

[0603] Next reference Figure 31K, using a training dataset containing time series data to train (3142) a trained neural network, the time series data including discharge and temperature data during continuous use of different commercially available Li-ion batteries (e.g., a NASA battery usage dataset; the dataset provides data on continuous use of 6 commercially available Li-ion batteries; the network operation is based on analysis of the battery's discharge curve), the training is used to monitor the state of health (SOH) and state of charge (SOC) of lithium-ion batteries for use in a battery management system (BMS). In some implementations, the neural network topology includes (3144) an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values. The equivalent simulated network includes (3146): (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 simulated neurons (e.g., 1271 simulated neurons), and (v) between 51,000 and 52,000 connections (e.g., 51,776 connections).

[0604] Next reference Figure 31L , a trained neural network is trained (3148) using a training dataset containing time series data, the time series data including discharge and temperature data during continuous use of different commercially available Li-ion batteries (e.g., a NASA battery use dataset; the dataset provides data on continuous use of 6 commercially available Li-ion batteries; the network operation is based on analysis of the discharge curves of the batteries), the training is used to monitor the health state (SOH) of lithium-ion batteries for use in a battery management system (BMS). In some implementations, the neural network topology includes (3150) an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron. In some implementations, the equivalent simulated network includes (3152): (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 simulated neurons (e.g., 201 simulated neurons), and (v) between 2,200 and 2,400 connections (e.g., 2,300 connections).

[0605] Next reference Figure 31M, using a training dataset containing voice commands (e.g., a Google voice command dataset) to train (3154) a trained neural network for recognizing voice commands (e.g., 10 short spoken keywords including "yes", "no", "up", "down", "left", "right", "on", "off", "stop", "run"). In some implementations, the neural network topology is (3156) a deep separable convolutional neural network (DS-CNN) layer with 1 neuron. In some implementations, the equivalent simulated network includes (3158): (i) a maximum of 100 input and output connections per simulated neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 simulated neurons, and (v) approximately 2.6 million connections.

[0606] Next reference Fig.31N A trained neural network is trained (3160) using a training dataset containing photoplethysmography (PPG) data, accelerometer data, temperature data, and galvanic skin response signal data for different individuals performing various physical activities over a predetermined time period, and reference heart rate data obtained from an ECG sensor (e.g., PPG data from the PPG-Dalia dataset (CHECK LICENSE)). Data is collected for 15 individuals performing various physical activities during each 1-4 hour period. The wrist-based sensor data contains PPG, 3-axis accelerometer, temperature, and galvanic skin response signals sampled from 4 Hz to 64 Hz, and reference heart rate data obtained from an ECG sensor at a sampling frequency of approximately 2 Hz. The raw data is divided into sequences of 1000 time steps (approximately 15 seconds) with a shift of 500 time steps, resulting in a total of 16,541 samples. The dataset is divided into 13233 training samples and 3308 test samples for determining pulse rate during physical exercise (e.g., jogging, gym exercise, stair climbing) based on PPG sensor data and 3-axis accelerometer data. The neural network topology includes (3162): two Conv1D layers, each with 16 filters and a kernel of 20 that perform time series convolution; two LSTM layers, each with 16 neurons; and two dense layers with 16 neurons and 1 neuron, respectively. In some implementations, the equivalent simulated network includes (3164): (i) a delay block that produces any number of time steps, (ii) a maximum of 100 input and output connections per simulated neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 simulated neurons (e.g., 713 simulated neurons), and (vi) between 12,000 and 12,500 connections (e.g., 12,072 connections).

[0607] Next reference Fig.31O, the trained neural network is trained (3166) to classify different objects (e.g., people, cars, bicycles, scooters) based on pulse Doppler radar signals (removing clutter and providing noise to the Doppler radar signal), and the neural network topology includes (3168) a multi-scale LSTM neural network.

[0608] Next reference Fig.31P The trained neural network is trained (3170) to perform human activity type recognition (e.g., walking, running, sitting, climbing stairs, exercising, activity tracking) based on inertial sensor data (e.g., 3-axis accelerometer, magnetometer, or gyroscope data from a fitness tracking device, smart watch, or mobile phone; 3-axis accelerometer data sampled at up to 96 Hz as input). The network is trained on 3 different publicly available datasets, presenting activities such as "opening and closing the dishwasher", "drinking water while standing", "closing the left hand door", "jogging", "walking", "climbing stairs", etc. In some implementations, the neural network topology includes (3172) three channelized convolutional networks, each channelized convolutional network having a convolutional layer with 12 filters and a kernel size of 64, and each convolutional layer is followed by a max pooling layer, and two common dense layers with 1024 neurons and N neurons, respectively, where N is the number of categories. In some implementations, the equivalent simulated network includes (3174): (i) delay blocks for generating any number of time steps, (ii) up to 100 input and output connections per simulated neuron, (iii) an output layer of 10 simulated neurons, (iv) a signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 simulated neurons (e.g., 1296 simulated neurons), and (vii) between 20,000 and 21,000 connections (e.g., 20,022 connections).

[0609] Next reference Figure 31Q The trained neural network is further trained (3176) to detect abnormal patterns of human activity based on accelerometer data merged with heart rate data using a convolution operation (so as to detect pre-stroke or pre-heart attack states or signals in the case of sudden abnormal patterns caused by injuries or dysfunctions due to medical reasons such as epilepsy, etc.).

[0610] Some implementations include components that are not integrated into the chip (i.e., these are external elements connected to the chip), which are selected from the group consisting of: voice recognition, video signal processing, image sensing, temperature sensing, pressure sensing, radar processing, LIDAR processing, battery management, MOSFET circuit current and voltage, accelerometer, gyroscope, magnetic sensor, heart rate sensor, gas sensor, volume sensor, liquid level sensor, GPS satellite signal, body conductivity sensor, gas flow sensor, concentration sensor, pH meter and IR vision sensor.

[0611] According to some implementations, examples of analog neuromorphic integrated circuits fabricated according to the above process are provided in the following sections.

[0612] Example Analog Neuromorphic IC for Selective Gas Detection

[0613] In some implementations, a neuromorphic IC is manufactured according to the above process. The neuromorphic IC is based on a deep convolutional neural network that is trained to selectively sense different gases in a gas mixture containing some amount of the gas to be detected. The deep convolutional neural network is trained using a training data set containing signals from an array of gas sensors (e.g., 2 to 25 sensors) that respond to different gas mixtures. The integrated circuit (or chip manufactured according to the techniques described herein) can be used to determine one or more known gases in a gas mixture despite the presence of other gases in the mixture.

[0614] In some implementations, the trained neural network is a multi-label 1D-DCNN network for mixed gas classification. In some implementations, the network is designed to detect 3 binary gas components based on measurements from 16 gas sensors. In some implementations, the 1D-DCNN includes sensor-like 1D convolution blocks (16 such blocks), 3 common 1D convolution blocks, and 3 dense layers. In some implementations, the performance of the 1D-DCNN network for this task is 96.3%.

[0615] In some implementations, the original network is T-transformed with the following parameters: maximum input and output connections per neuron = 100; delay blocks can generate delays of any number of time steps; and signal limit is 5.

[0616] In some implementations, the resulting T-network has the following properties: 15 layers, approximately 100,000 simulated neurons, approximately 4,900,000 connections.

[0617] Example simulation neuromorphic IC for MOSFET failure prediction

[0618] MOSFET on-resistance degradation due to thermal stress is a well-known and serious problem in power electronics. In real-world applications, MOSFET device temperature often changes in a short period of time. This temperature sweep produces thermal degradation of the device, since the device may exhibit an exponential distribution. This effect is generally studied through power cycling (which produces a temperature gradient), which causes MOSFET degradation.

[0619] In some implementations, a neuromorphic IC is manufactured according to the above process. The neuromorphic IC is based on a network discussed in an article titled “Real-time Deep Learning at the Edge for Scalable Reliability Modeling of SI-MOSFET Power Electronics Converters” for predicting the remaining useful life (RUL) of a MOSFET device. The neural network can be used to determine the remaining useful life (RUL) of a device with an accuracy of over 80%.

[0620] In some implementations, the network is trained on the NASAMOSFET dataset containing thermal aging time series for 42 different MOSFETs. The data is sampled every 400ms and typically includes several hours of data for each device. The network contains 4 LSTM layers, each with 64 neurons, followed by 2 dense layers with 64 and 1 neurons.

[0621] In some implementations, the network is T-transformed with the following parameters: the maximum input and output connections per neuron is 100; the signal limit is 5, and the resulting T-network has the following characteristics: 18 layers, approximately 3,000 neurons (e.g., 137 neurons), and approximately 120,000 connections (e.g., 123200 connections).

[0622] Example Analog Neuromorphic IC for Li-ion Battery Health and SoC Monitoring

[0623] In some implementations, a neuromorphic IC is manufactured according to the above process. Neuromorphic ICs can be used for predictive analysis of lithium-ion batteries for use in a battery management system (BMS). BMS devices generally provide functions such as overcharge and overdischarge protection, monitoring of the state of health (SOH) and state of charge (SOC), and load balancing of several batteries. SOH and SOC monitoring typically require a digital data processor, which increases the cost of the device and consumes power. In some implementations, an integrated circuit is used to obtain accurate SOC and SOH data without implementing a digital data processor on the device. In some implementations, the integrated circuit determines the SOC with an accuracy of more than 99% and determines the SOH with an accuracy of more than 98%.

[0624] In some implementations, the network operation is based on analysis of the temperature and discharge curve of the battery, and / or the data is presented as a time series. Some implementations use data from the NASA battery usage dataset. The dataset provides data on the continuous use of 6 commercially available Li-ion batteries. In some implementations, the network includes an input layer, 2 LSTM layers (64 neurons each), and an output dense layer of 2 neurons (SOC and SOH values).

[0625] In some implementations, the network is T-transformed with the following parameters: maximum number of input and output connections per neuron = 100 and signal limit of 5. In some implementations, the resulting T network includes the following characteristics: 9 layers, approximately 1,200 neurons (e.g., 1,271 neurons) and approximately 50,000 connections (e.g., 51,776 connections). In some implementations, the network operation is based on analysis of the temperature and discharge curve of the battery. The network is trained using the network IndRnn disclosed in a paper titled "State-of-Health Estimation of Li-ion Batteries in Electric Vehicle Using IndRNN under Variable Load Condition", which is designed to process data from a NASA battery usage dataset. The dataset provides data on the continuous use of 6 commercially available Li-ion batteries. The IndRnn network includes an input layer with 18 neurons, a simple recurrent layer of 100 neurons, and a dense layer of 1 neuron.

[0626] In some implementations, the IndRnn network is T-transformed with the following parameters: maximum input and output connections per neuron = 100 and signal limit of 5. In some implementations, the resulting T network has the following characteristics: 4 layers, about 200 neurons (e.g., 201 neurons), and about 2,000 connections (e.g., 2,300 connections). Some implementations output SOH with only an estimation error of 1.3%. In some implementations, SOC is obtained similar to how SOH is obtained.

[0627] Example Simulated Neuromorphic IC for Keyword Detection

[0628] In some implementations, a neuromorphic IC is manufactured according to the above process. The neuromorphic IC can be used for keyword detection.

[0629] The input network is a neural network with 2-D convolutional layers and 2-D deep convolutional layers with an input audio mel-spectrogram of size 49 by 10. In some implementations, the network includes 5 convolutional layers, 4 deep convolutional layers, an average pooling layer, and a final dense layer.

[0630] In some implementations, the network is pre-trained to recognize 10 short spoken keywords ("yes", "no", "up", "down", "left", "right", "on", "off", "stop", "run") from the Google voice command dataset with 94.4% recognition accuracy.

[0631] In some implementations, an integrated circuit is fabricated based on a deep separable convolutional neural network (DS-CNN) for voice command recognition. In some implementations, the original DS-CNN network is T-transformed with the following parameters: maximum input and output connections per neuron = 100, signal limit 5. In some implementations, the resulting T network has the following characteristics: 13 layers, about 72,000 neurons, and about 2.6 million connections.

[0632] Example DS-CNN keyword detection network

[0633] In one instance, according to some implementations, the keyword detection network is transformed into a T network. The network is a neural network of 2-D convolutional layers and 2-D deep convolutional layers with an input audio spectrogram of size 49x10. The network consists of 5 convolutional layers, 4 deep convolutional layers, an average pooling layer, and a final dense layer. The network is pre-trained to recognize 10 short spoken keywords ("yes", "no", "up", "down", "left", "right", "on", "off", "stop", "operation") from the Google voice command dataset https: / / ai.googleblog.com / 2017 / 08 / launching-speech-commands-dataset.html. There are two additional categories corresponding to "silence" and "unknown". The network output is a softmax of length 12.

[0634] According to some implementations, the trained neural network (on transformed input) has a recognition accuracy of 94.4%. In the neural network topology, each convolutional layer is followed by a BatchNorm layer and a ReLU layer, and the ReLU activation is unbounded and includes approximately 2.5 million multiplication-addition operations.

[0635] After the transformation, the transformed simulated network was tested with a test set of 1000 samples (100 for each spoken command). All test samples were also used as test samples in the original dataset. The original DS-CNN network gave a recognition error close to 5.7% for this test set. The network was converted into a T-network composed of trivial neurons. The Batch normalization (sometimes called BatchNorm) layer in the "test" mode produces a simple linear signal transformation and can therefore be interpreted as a weight multiplier + some additional bias. The T transformation is very straightforward for the convolutional layers, average pooling layers, and dense layers. The Softmax activation function is not implemented in the T network and is applied solely to the T network output.

[0636] The resulting T network has 12 layers, including an input layer, approximately 72,000 neurons, and approximately 2.5 million connections.

[0637] Figure 26A-26K An example histogram 2600 of the absolute weights of layers 1 to 11 according to some implementations is shown, respectively. A weight distribution histogram (for absolute weights) is calculated for each layer. The dashed lines in the graph correspond to the average absolute weight values ​​of the corresponding layers. After the transformation (i.e., T-transformation), the average output absolute error (calculated on the test set) of the converted network compared to the original network is calculated to be 4.1e-9.

[0638] According to some implementations, various examples for setting network limits for the transformed network are described herein. For signal limits, because the ReLU activations used in the network are unbounded, and some implementations use signal limits on each layer. This may potentially affect mathematical equivalence. To this end, some implementations use a signal limit of 5 on all layers, which corresponds to a supply voltage of 5 related to the input signal range.

[0639] To quantize the weights, some implementations use a nominal set of 30 resistors [0.001, 0.003, 0.01, 0.03, 0.1, 0.324, 0.353, 0.436, 0.508, 0.542, 0.544, 0.596, 0.73, 0.767, 0.914, 0.985, 0.989, 1.043, 1.101, 1.149, 1.157, 1.253, 1.329, 1.432, 1.501, 1.597, 1.896, 2.233, 2.582, 2.844].

[0640] Some implementations select R- and R+ values ​​for each layer individually (see description above). For each layer, some implementations select values ​​that provide the maximum weight accuracy. In some implementations, all weights (including biases) in the T network are then quantized (e.g., set to the closest value that can be achieved using the input or selected resistors).

[0641] Some implementations transform the output layer as follows. The output layer is a dense layer without ReLU activations. According to some implementations, this layer has a softmax activation which is not implemented in the T transformation and is left to the numeric part. Some implementations do not perform additional transformations.

[0642] Example analog neuromorphic IC for obtaining heart rate

[0643] PPG is an optically obtained plethysmogram that can be used to detect changes in blood volume in the microvascular bed of tissue. PPG is often obtained by using a pulse oximeter, which illuminates the skin and measures changes in light absorption. PPG is often processed in devices (such as fitness trackers) to determine heart rate. Deriving heart rate (HR) from PPG signals is an important task in edge device computing. PPG data obtained from a device located on the wrist usually allows obtaining a reliable heart rate only when the device is stable. If a person is engaged in physical exercise, obtaining heart rate from PPG data produces poor results unless combined with inertial sensor data.

[0644] In some implementations, an integrated circuit based on a combination of convolutional neural networks and LSTM layers can be used to accurately determine pulse rate based on data from a photoplethysmography (PPG) sensor and a 3-axis accelerometer. The integrated circuit can be used to suppress motion artifacts of the PPG data and determine the pulse rate during physical exercise (e.g., jogging, fitness exercises, and stair climbing) with an accuracy of over 90%.

[0645] In some implementations, the input network is trained with PPG data from the PPG-Dalia dataset. Data is collected for 15 individuals performing various physical activities within a predetermined duration (e.g., 1-4 hours each). The training data including wrist-based sensor data contains PPG, 3-axis accelerometer, temperature, and skin galvanic response signals sampled from 4Hz to 64Hz, and reference heart rate data obtained from an ECG sensor at a sampling frequency of approximately 2Hz. The raw data is divided into sequences of 1000 time steps (about 15 seconds) with a shift of 500 time steps, thus generating a total of 16541 samples. The data set is divided into 13233 training samples and 3308 test samples.

[0646] In some implementations, the input network includes: 2 Conv1D layers, each with 16 filters that perform time series convolution; 2 LSTM layers, each with 16 neurons; and 2 Dense layers with 16 and 1 neurons. In some implementations, the network produces an MSE error of less than 6 beats per minute on the test set.

[0647] In some implementations, the network is T-transformed with the following parameters: delay blocks can generate delays of any number of time steps, maximum input and output connections per neuron = 100, and signal limit of 5. In some implementations, the resulting T-network has the following properties: 15 layers, approximately 700 neurons (e.g., 713 neurons), and approximately 12,000 connections (e.g., 12072 connections).

[0648] Example of processing PPG data with a T-transformed LSTM network

[0649] As described above, for recurrent neurons, some implementations use a signal delay block that is added to each recurrent connection of GRU and LSTM neurons. In some implementations, the delay block has an external loop timer (e.g., a digital timer) that activates the delay block with a constant time period dt. This activation produces an output of x(t-dt), where x(t) is the input signal to the delay block. Such an activation frequency can, for example, correspond to the network input signal frequency (e.g., the output frequency of an analog sensor processed by a T-converted network). Typically, all delay blocks are activated simultaneously with the same activation signal. Some blocks can be activated simultaneously at one frequency, while other blocks can be activated at another frequency. In some implementations, these frequencies have a common multiplier and the signals are synchronized. In some implementations, multiple delay blocks are used on one signal, resulting in additional time shifts. Reference above Fig. 13B Describes the example of a delay block, Fig. 13B Two examples of delay blocks according to some implementations are shown.

[0650] According to some implementations, the network for processing PPG data uses one or more LSTM neurons. According to some implementations, the above reference Fig.13A An example of LSTM neuron implementation is described.

[0651] The network also uses Conv1D, a convolution performed on the time coordinate. According to some implementations, the above reference Fig.15A and Fig. 15B An example of Conv1D implementation is described.

[0652] Details of PPG data are described herein, according to some implementations. PPG is an optically obtained plethysmogram that can be used to detect changes in blood volume in the microvascular bed of tissue. PPG is often obtained by using a pulse oximeter that illuminates the skin and measures changes in light absorption. PPG is often processed in a device (e.g., a fitness tracker) to determine heart rate. Deriving heart rate (HR) from a PPG signal is an important task in edge device computing.

[0653] Some implementations use PPG data from the Capnobase PPG dataset. The data contains raw PPG signals (sampled at 300 samples per second) for 8 minutes each for 42 individuals, as well as reference heart rate data obtained from an ECG sensor at approximately 1 sample per second. For training and evaluation, some implementations split the raw data into sequences of 6000 time steps with a shift of 1000 time steps, resulting in a total set of 5838 samples in total.

[0654] In some implementations, input based on the trained neural network NN allows for 1%-3% accuracy in deriving heart rate (HR) from PPG data.

[0655] This section describes a relatively simple neural network to demonstrate how T-transformation and analog processing can handle this task. This description is provided as an example, according to some implementations.

[0656] In some implementations, the dataset is divided into 4,670 training samples and 1,168 test samples. The network includes: 1 Conv1D layer with 16 filters and 20 kernels; 2 LSTM layers, each with 24 neurons; 2 dense layers (each with 24 and 1 neurons). In some implementations, after training the network for 200 epochs, the test accuracy is found to be 2.1%.

[0657] In some implementations, the input network is T-transformed with the following parameters: delay blocks with periods of 1, 5, and 10 time steps and the following characteristics: 17 layers, 15,448 connections, and 329 neurons (OP3 neurons and multiplier blocks, not counting delay blocks).

[0658] Example Analog Neuromorphic Integrated Circuit for Object Recognition Based on Pulse Doppler Radar Signals

[0659] In some implementations, an integrated circuit is fabricated based on a multi-scale LSTM neural network, which can be used to classify objects based on pulse Doppler radar signals. The IC can be used to classify different objects (e.g., people, cars, bicycles, scooters) based on Doppler radar signals, remove clutter, and provide noise to Doppler radar signals. In some implementations, the accuracy of object classification using the multi-scale LSTM network exceeds 90%.

[0660] Example Analog Neuromorphic Integrated Circuit for Human Activity Type Recognition Based on Inertial Sensor Data

[0661] In some implementations, a neuromorphic integrated circuit is fabricated and can be used for human activity type recognition based on a multi-channel convolutional neural network with input signals from a 3-axis accelerometer and possibly a magnetometer and / or gyroscope from a fitness tracking device, smart watch, or mobile phone. The multi-channel convolutional neural network can be used to distinguish different types of human activities, such as walking, running, sitting, climbing stairs, exercising, and can be used for activity tracking. The IC can be used to detect abnormal patterns of human activities based on accelerometer data convolved with heart rate data. According to some implementations, such an IC can detect pre-stroke or pre-heart attack states or signals in the case of sudden abnormal patterns caused by injuries or dysfunctions due to medical reasons such as epilepsy.

[0662] In some implementations, the IC is based on a channelized 1D convolutional network discussed in the article “Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors”. In some implementations, the network accepts as input 3-axis accelerometer data, which is sampled at up to 96 Hz. In some implementations, the network is trained on 3 different publicly available datasets, presenting such activities as “open and close the dishwasher”, “drink water while standing”, “close the left hand door”, “jog”, “walk”, “climb stairs”, etc. In some implementations, the network includes 3 channelized convolutional networks, which have convolutional layers with 12 filters and kernels of 64, each convolutional layer is followed by a max pooling (4) layer, and 2 common dense layers with 1024 and N neurons respectively, where N is the number of classes. In some implementations, activity classification is performed with a low error rate (e.g., 3.12% error).

[0663] In some implementations, the network is T-transformed with the following parameters: delay blocks can generate delays of any number of time steps, maximum input and output connections per neuron = 100, output layer of 10 neurons, and signal limit of 5. In some implementations, the resulting T-network has the following properties: 10 layers, approximately 1,200 neurons (e.g., 1296 neurons), and approximately 20,000 connections (e.g., 20022 connections).

[0664] Example transformations for building a modular net structure for a library

[0665] According to some implementations, a modular structure of a converted neural network is described herein. Each module of a modular neural network is obtained after the transformation of (all or part of) one or more trained neural networks. In some implementations, one or more trained neural networks are subdivided into parts and then transformed into equivalent analog networks. The modular structure is unique to some of the currently used neural networks, and the modular division of neural networks corresponds to the trend in the development of neural networks. Each module can have any number of inputs or any number of connections from input neurons to the output neurons of the connected modules and any number of outputs connected to the input layer of the subsequent module. In some implementations, a library of preliminary transformation modules (or seed lists of transformation modules) is developed, including photolithography masks for the manufacture of each module. The final chip design is obtained as a combination of preliminary development modules (or by connecting preliminary development modules). Some implementations perform exchanges between modules. In some implementations, ready-made module design templates are used to convert neurons and connections within modules into chip designs. This significantly simplifies the manufacture of chips that are completed by connecting only corresponding modules.

[0666] Some implementations generate libraries of ready-made T-converted neural networks and / or T-converted modules. For example, one layer of a CNN network is a modular building block, an LSTM chain is another building block, and so on. Larger neural networks NN also have modular structures (e.g., LSTM modules and CNN modules). In some implementations, the library of neural networks is not just a byproduct of the example process, and can be sold independently. For example, a third party can start making a neural network with a simulated circuit, circuit diagram, or design in the library (e.g., using a CADENCE circuit, file, and / or photolithography mask). Some implementations generate T-converted neural networks for typical neural networks (e.g., networks that can be transformed into CADENCE or similar software), and the converted neural networks (or associated information) are sold to third parties. In some instances, according to some implementations, the third party chooses not to disclose the structure and / or purpose of the initial neural network, but uses conversion software (e.g., the SDK described above) to convert the initial network into a trapezoidal network, and passes the transformed network to the manufacturer to manufacture the transformed network, where the weight matrix is ​​obtained using one of the above processes. As another example, where a library of ready-made networks is generated according to the process described herein, corresponding photolithography masks are generated, and a consumer can train one of the available network architectures for his task, perform a lossless transform (sometimes called a T-transform), and provide the weights to a manufacturer for fabricating chips for the trained neural network.

[0667] In some implementations, the modular structure concept is also used in the manufacture of multi-chip systems or multi-level 3D chips, where each layer of the 3D chip represents a module. In the case of a 3D chip, the connection of the output of the module to the input of the connected module will be established through a standard interconnect, which provides ohmic contacts of different layers in a multi-layer 3D chip system. In some implementations, the analog outputs of certain modules are connected to the analog inputs of the connected modules through inter-layer interconnects. In some implementations, modular structures are also used to manufacture multi-chip processor systems. The distinguishing feature of such a multi-chip assembly is the analog signal data line between different chips. The analog signal exchange and de-exchange standard scheme developed in the analog circuit system is used to implement the analog exchange scheme unique to the corresponding de-exchange of several analog signals compressed into one data line and the analog signal at the receiver chip.

[0668] A major advantage of chips manufactured according to the above technology is that analog signal propagation can be extended to multi-layer chips or multi-chip modules, where all signals are interconnected and data lines transmit analog signals without the need for analog-to-digital or digital-to-analog conversion. In this way, analog signal transmission and processing can be extended to 3D multi-layer chips or multi-chip modules.

[0669] Example method for generating libraries for hardware implementation of neural networks

[0670] Figure 32A-Figure 32E A flow chart of a method 3200 for generating (3202) a library for hardware implementation of a neural network according to some implementations is shown. The method is performed (3204) at a computing device 200 (e.g., using a library generation module 254) having one or more processors 202 and a memory 214 storing one or more programs configured for execution by the one or more processors 202.

[0671] The method includes obtaining (3206) a plurality of neural network topologies (eg, topology 224), each neural network topology corresponding to a respective neural network (eg, neural network 220).

[0672] The method also includes transforming (3208) each neural network topology into a corresponding equiva...

Claims

1. A system comprising: A hardware device, the hardware device comprising: a vibration sensor configured to sense vibrations of a vibration source of the machine; an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors, wherein the analog circuit is coupled to the vibration sensor and is configured to: - receiving an analog signal from said vibration sensor; and - calculating an output based on the simulated signal by executing a portion of the trained neural network; a transceiver coupled to the analog circuit and configured to receive the output from the analog circuit and transmit the output over a low power wide area network (LPWAN); and a digital circuit communicatively coupled to the transceiver of the hardware device via the LPWAN, wherein the digital circuit is configured to: - receiving said output from said analog circuit; and - Predict the status of the machine based on the output for maintenance.

2. The system according to claim 1, wherein: The digital circuit includes one or more digital computing units selected from the group consisting of: CPU, GPU, RISC, FPGA, and ASIC.

3. The system according to claim 1, wherein: The digital circuit includes a processor configured to perform data classification.

4. The system according to claim 3, wherein: The data classification is performed by a neural network different from the trained neural network.

5. The system according to claim 3, wherein: The data classification is performed using k-nearest neighbors (k-NN).

6. The system according to claim 1, wherein: The output of the analog circuit represents an embedding, and the digital circuit is configured to classify the analog signal using the embedding.

7. A hardware device comprising: a vibration sensor configured to sense vibration of a vibration source; an analog circuit comprising a plurality of operational amplifiers and a plurality of resistors, wherein the analog circuit is coupled to the vibration sensor and is configured to: receiving an analog signal from the vibration sensor; and An output is calculated based on the simulated signal by executing a portion of the trained neural network.

8. The hardware device according to claim 7, further comprising: A transceiver is coupled to the analog circuit, wherein the transceiver is configured to receive the output from the analog circuit and transmit the output via a low power wide area network (LPWAN).

9. The hardware device according to claim 7, wherein: The vibration sensor is disposed adjacent to a movable component of a machine, and the vibration sensor is configured to collect vibration data of the movable component.

10. The hardware device according to claim 9, wherein: The movable parts include ball bearings of the machine.

11. The hardware device according to claim 7, wherein: The output of the simulation circuit represents an embedding for at least one of defining a vibration source, predicting a failure of a machine coupled to the vibration source, and generating a recommendation for maintenance of the machine.

12. The hardware device according to claim 7, wherein: The vibration sensor is disposed in or on a tire and is configured to collect vibration data of the tire.

13. The hardware device according to claim 7, wherein: The output represents an embedding for predicting at least one of: a road surface, a physical condition, a tire condition, a suspension system condition, or a time to failure of the vibration source.

14. The hardware device according to claim 7, wherein: The vibration sensor is configured to sample signals in the range of 0 to 20 kilohertz (kHz).

15. The hardware device according to claim 7, wherein: The vibration sensor is configured to sample signals up to 41 kHz.

16. The hardware device according to claim 7, wherein: The vibration sensor is configured to sample signals below a Nyquist sampling rate for fault detection and classification in a compressed domain.

17. The hardware device according to claim 7, wherein: The vibration sensor is configured to sample a signal for compression sensing (CS) for condition classification of a rolling element support in a rotating machine.

18. The hardware device according to claim 7, wherein: The trained neural network includes a plurality of neuron layers, the plurality of neuron layers includes a first set of layers and a second set of layers, and the analog circuit is configured to implement the first set of layers.

19. The hardware device according to claim 18, wherein: The second set of layers consists of the last layer of the trained neural network.

20. The hardware device according to claim 7, wherein: The trained neural network includes a deep neural network for unsupervised learning based on a sparse autoencoder.

21. The hardware device according to claim 7, wherein: The trained neural network includes a ResNet convolutional neural network (CNN) with global average pooling (GAP) for feature learning and fault diagnosis of rolling bearings.

22. The hardware device according to claim 7, wherein: The vibration sensor is configured to sample and output a one-dimensional time domain signal of a rolling bearing failure signal.

23. The hardware device according to claim 7, wherein: The trained neural network includes stacked denoising autoencoders.

24. The hardware device according to claim 7, wherein: The analog circuit is configured to be powered by vibrations of the vibration source.

25. The hardware device of claim 24, further comprising a power harvesting circuit configured to harvest power from the vibration of the vibration source and supply power to the analog circuit.

26. The hardware device according to claim 7, wherein: The plurality of operational amplifiers are configured to implement neurons of a portion of the trained neural network, and wherein the plurality of resistors are configured to implement connections between neurons of a portion of the trained neural network.

27. The hardware device according to claim 7, wherein: The analog circuit is configured to implement an optimized neural network corresponding to the trained neural network.

28. The hardware device according to claim 7, wherein: The values ​​of the plurality of resistors are based on weights of the connections of the trained neural network.

29. The hardware device according to claim 7, wherein: The plurality of resistors are configured to connect the plurality of operational amplifiers.

30. The hardware device according to claim 7, wherein: The analog circuit includes a resistor in a back end of line (BEOL).

31. The hardware device of claim 7, wherein: (i) the trained neural network is an autoencoder comprising an encoder part and a decoder part; (ii) the encoder part reconstructs the input vector at the output layer after the non-linear transformation performed by the hidden layer; (iii) the analog circuit corresponds to the encoder portion of the autoencoder, the encoder portion including the hidden layer; and (iv) The analog circuit is configured to compute a representation of the input vector in fewer dimensions than an input space of the input vector.

32. The hardware device according to claim 7, wherein: The analog circuit is configured to generate compressed data encoding vibration sensor data based on vibration characteristics from the vibration sensor.

33. A method comprising: Using a vibration sensor to sense the vibration of the vibration source to obtain an analog signal; calculating an output based on the analog signal by executing a portion of the trained neural network using an analog circuit including a plurality of operational amplifiers and a plurality of resistors; and The output is transmitted via a low power wide area network (LPWAN) using a transceiver.

Images