Analog hardware implementation of a trained neural network for speech intelligibility

Abstract

A system and method for an analog hardware implementation of a convolutional neural network for speech intelligibility are provided. The method includes the step of obtaining the neural network topology and weights of a trained neural network. The method also includes the step of converting the neural network topology into an equivalent analog network of analog components. The method also includes the step of calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents one or more connections between the analog components of the equivalent analog network. The method also includes the step of generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application is a continuation of PCT application PCT / RU2020 / 000306, filed June 25, 2020, titled "Analog Hardware Realization of Neural Networks," and a continuation of U.S. application No. 17 / 189,109, filed March 1, 2021, titled "Analog Hardware Realization of Neural Networks," claiming priority to the latter, the full text of which is incorporated herein by reference. U.S. application No. 17 / 189,109 is also a partial continuation of PCT application PCT / EP2020 / 067800, filed June 25, 2020, titled "Analog Hardware Realization of Neural Networks," the full text of which is incorporated herein by reference.

[0003] Technology field

[0004] The disclosed implementations generally relate to neural networks, and more specifically to a system and method for a hardware implementation of a trained neural network for speech intelligibility. Background Technology

[0005] Existing hardware has failed to keep pace with the innovations in neural networks and the growing popularity of machine learning-based applications. As the advancement of digital microprocessors stagnates, the complexity of neural networks continues to outpace the computing power of CPUs and GPUs. Neuromorphic processors based on spiked neural networks, such as Loihi and True North, are limited in their applications. For GPU-like architectures, their power and speed are limited by data transfer rates. Data transfer can consume up to 80% of chip power and significantly impact computation speed. Edge applications require low power consumption, yet there are currently no known high-performance hardware implementations that consume less than 50 milliwatts.

[0006] Memristor-based architectures using cross-bar technology remain impractical for manufacturing recurrent and feedforward neural networks. For example, memristor-based cross-bars have several drawbacks, including long latency and current leakage during operation, which render them impractical. Additionally, there are reliability issues when manufacturing memristor-based cross-bars, particularly when the neural network contains both negative and positive weights. In the case of large-scale neural networks with many neurons, at high dimensions, memristor-based cross-bars cannot be used for the simultaneous propagation of other signals, which ultimately complicates the summation of signals when neurons are represented by operational amplifiers. Furthermore, memristor-based analog integrated circuits have many limitations, such as a small number of resistive states, the first-cycle problem when forming memristors, the complexity of channel formation when training memristors, unpredictable dependence on memristor dimensions, slow operation of memristors, and drift of resistive states.

[0007] Additionally, the training process required for neural networks presents unique challenges regarding their hardware implementation. Trained neural networks are used for specific inference tasks, such as classification. Once a neural network is trained, the corresponding hardware is manufactured. If the neural network is retrained, the hardware manufacturing process is repeated, increasing costs. While some reconfigurable hardware solutions exist, such hardware cannot be easily mass-produced and costs significantly more than non-reconfigurable hardware (e.g., more than five times the cost). Furthermore, edge environments, such as smart home applications, do not require this reprogrammability. For instance, since 85% of all neural network applications do not require retraining during operation, on-chip learning is not particularly useful. Moreover, edge applications involve noisy environments that can render reprogrammable hardware unreliable.

[0008] Voice transmission accounts for the majority of communication between humans and human-machine interfaces, far surpassing video and hand-type communication. The clarity of voice transmission must be maintained while compressing or digitizing voice signals for transmission. Traditionally, multiple noise suppression and noise filtering methods and devices process unclear voice signals and remove at least some of the unwanted noise. Some conventional techniques use microphones that capture noise and generate sound that effectively removes unwanted noise detected around the listener. These techniques are more commonly used in headphones, particularly noise-canceling headphones. There are also techniques that suppress specific noise based on the spectral quality of a specific noise source or by using more sophisticated algorithms, such as Markov processes, Fast Fourier Transform methods, and various noise detection adaptation algorithms.

[0009] More specifically, neural networks have been used to analyze signals containing a mixture of speech and noise, and to effectively extract signals that contain mostly speech based on specific features attributed to speech. These neural networks require training and are effectively implemented as programs running on powerful computers. Such computers consume significant power and computing power. Conventional solutions are often limited by training features, fail to provide real-time processing, and have limitations in processing specific recorded speech signals. Currently, voice communication is primarily conducted via mobile phones or landlines. Conventional equipment lacks the computing power and / or power to effectively process speech signals and suppress unwanted noise. No matter how sophisticated the noise-canceling technology, the types of noise that can be effectively suppressed are significantly limited. It is common for unwanted disturbances, such as dog barking, door slamming, emergency sirens, car horns, and similar unpredictable interference, to occur, and these serve as background noise that hinders the transmission of clear speech signals.

[0010] Non-voice noise or signals can originate near speakers, microphones, or other devices used to convert sound into electrical signals. This noise is generally referred to as background noise. In the case of these noise signals, background conversation or human voices located far from the microphone can be considered noise. Other non-voice noise can occur during signal processing and transmission, such as compression, analog-to-digital conversion, spectrum limiting, or the splitting of packets limited by length, spectrum, or information size. This noise occurs not only during transmission but also in the corresponding reverse phase. When multiple voice / noise signals are mixed together, such as in conference calls or multi-party communication, the noise associated with each signal becomes intermingled, further complicating the tasks and challenges of speech clarification. Additionally, if voice signals are further processed to generate actual sound near the recipient's ear via speakers, headphones, or other devices or methods (e.g., in human-to-human communication), additional noise or unwanted signals may be introduced by the surrounding environment near the recipient. The problem to be solved

[0011] Therefore, methods, circuits, and / or interfaces are required to address at least some of the defects identified above. Analog circuits modeled from trained neural networks and manufactured according to the techniques described herein can provide enhanced performance per watt benefits, be useful for implementing hardware solutions in edge environments, and handle various applications such as drone navigation and autonomous vehicles. The cost benefits provided by the proposed manufacturing methods and / or analog network architectures are much more pronounced in larger neural networks. Furthermore, analog hardware implementations of neural networks provide improved parallel processing and neuromorphism. In addition, neuromorphic analog components are insensitive to noise and temperature variations compared to their digital counterparts.

[0012] Chips manufactured according to the technologies described herein provide a significant improvement over existing systems in terms of size, power, and performance, and are ideal for edge environments, including for retraining purposes. These analog neuromorphic chips can be used to implement edge computing applications or Internet of Things (IoT) environments. Due to the analog hardware, initial processing (e.g., descriptor formation for image recognition), which can consume more than 80 to 90 percent of power, can be moved to the chip, thereby reducing energy consumption and network load, which can open up new markets for applications.

[0013] Various edge applications can benefit from the use of such analog hardware. For example, for video processing, the technologies described herein can be used to include direct connections to CMOS sensors without digital interfaces. Various other video processing applications include road sign recognition for automobiles, camera-based real depth and / or simultaneous local application and mapping for robots, server-free room access control, and always-on solutions for security and medical applications. These chips can be used for data processing and low-level data fusion of radar and lidar. These technologies can be used to implement battery management functions for high-capacity battery packs, sound / voice processing not connected to data centers, voice recognition for mobile devices, speech instruction wake-up for IoT sensors, translators for translating one language into another, and process control configurable with large sensor arrays and / or hundreds of sensors in IoT with low signal strength. means of solving the problem

[0014] According to some implementations, neuromorphic analog chips can be mass-produced after standard software-based neural network simulation / training. Customer neural networks can be easily ported to custom chip design and production, regardless of the network structure. Additionally, according to some implementations, libraries are provided ready to create on-chip solutions (network emulators). These solutions require only training and a single change to the lithography mask, after which the chips can be mass-produced. For example, only a portion of the lithography mask needs to be changed during chip production.

[0015] The techniques described herein can be used to design and / or manufacture analog neuromorphic integrated circuits that are mathematically equivalent to a trained neural network (feedforward or recurrent neural network). According to some implementations, this process begins with a trained neural network that is first converted into a transformed network composed of standard elements. The behavior of the transformed network is simulated using software containing known models representing the standard elements. The software simulation is used to determine individual resistance values ​​for each of the resistors in the transformed network. Lithography masks are laid out based on an array of standard elements of the transformed network. Each standard element is laid out in the masks using an existing circuit library corresponding to the standard elements to simplify and speed up the process. In some implementations, resistors are laid out in one or more separate masks containing other elements of the transformed network (e.g., operational amplifiers). In this way, when the neural network is retrained, only masks containing resistors or other types of fixed resistance elements representing the new weights of the retrained neural network need to be regenerated, which simplifies and speeds up the process. Then, the lithography masks are sent to a fab to manufacture analog neuromorphic integrated circuits.

[0016] In one embodiment, a method for a hardware implementation of a neural network is provided according to some implementations. The method includes the step of obtaining a neural network topology and weights of a trained neural network. The method also includes the step of converting the neural network topology into an equivalent analog network of analog components. The method also includes the step of calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between the analog components of the equivalent analog network. The method also includes the step of generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components.

[0017] In some implementations, the step of generating a rough model includes the step of generating a resistance matrix for a weight matrix. Each element of the resistance matrix corresponds to a respective weight in the weight matrix and represents a resistance value.

[0018] In some implementations, the method further includes the steps of obtaining new weights for a trained neural network, calculating a new weight matrix for an equivalent analog network based on the new weights, and generating a new resistance matrix for the new weight matrix.

[0019] In some implementations, the neural network topology comprises one or more layers of neurons, each layer of neurons computes outputs based on a respective mathematical function, and the step of converting the neural network topology into an equivalent analog network of analog components comprises, for each layer of the one or more layers of neurons, (i) identifying one or more function blocks based on a respective intermediate mathematical function for each layer. Each function block has a respective schematic implementation having block outputs that correspond to the outputs of the respective mathematical function. and (ii) generating a multilayer network of each analog neuron based on arranging the one or more function blocks. Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of the first layer of the multilayer network is connected to one or more analog neurons of the second layer of the multilayer network.

[0020] In some implementations, one or more function blocks are (i) block output As a weighted summation block having, ReLu is a rectified linear unit (ReLU) activation function or a similar activation function, and V i represents the i-th input, and w i The weight summation block, wherein represents the weight corresponding to the i-th input, bias represents the bias value, and ∑ represents the summation operation; (ii) block output As a single multiplier block having, V i represents the i-th input, and V j The single multiplier block, wherein represents the j-th input and coeff is a predetermined coefficient; (iii) block output A sigmoid active block having, wherein V represents the input, and A and B are predetermined coefficient values ​​of the sigmoid active block; (iv) block output As a hyperbolic tangent activation block having, V in A hyperbolic tangent activation block, wherein represents an input, and A and B are predetermined coefficient values ​​of a sigmoid activation block; and (v) a signal delay block having block output U(t) = V(t-dt), wherein t represents a current time period, V(t-dt) represents the output of the signal delay block for a previous time period (t-dt), and dt is a delay value, comprising one or more basic function blocks selected from a group of said signal delay blocks.

[0021] In some implementations, the step of identifying one or more function blocks includes the step of selecting one or more function blocks based on the type of each layer.

[0022] In some implementations, the neural network topology comprises one or more layers of neurons, each neuron layer computes respective outputs based on a respective mathematical function, and the step of converting the neural network topology into an equivalent analog network of analog components comprises: (i) decomposing a first layer of the neural network topology into a plurality of sublayers, comprising decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions; each sublayer implements an intermediate mathematical function; and (ii) for each sublayer of the first layer of the neural network topology, (a) selecting one or more sub-function blocks based on each intermediate mathematical function for each sublayer; and (b) generating a multilayer analog subnet of each analog neuron based on arranging one or more sub-function blocks. Each analog neuron implements a respective function of one or more sub-function blocks, and each analog neuron of the first layer of the multilayer analog subnet is connected to one or more analog neurons of the second layer of the multilayer analog subnet.

[0023] In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and the step of decomposing the mathematical function includes the step of adjusting one or more weights to generate a mathematical function by combining one or more intermediate functions.

[0024] In some implementations, the method further comprises: (i) generating an equivalent digital network of digital components for one or more output layers of the neural network topology; and (ii) connecting the outputs of one or more layers of the equivalent analog network to the equivalent digital network of digital components.

[0025] In some implementations, the analog components include multiple operational amplifiers and multiple resistors, each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.

[0026] In some implementations, the step of selecting component values ​​of analog components includes the step of performing gradient descent to identify possible resistance values ​​for the plurality of resistors.

[0027] In some implementations, the neural network topology includes one or more GRU or LSTM neurons, and the step of transforming the neural network topology includes the step of generating one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons.

[0028] In some implementations, the one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency for the neural network topology.

[0029] In some implementations, the neural network topology comprises one or more layers of neurons performing unlimited activation functions, and the step of transforming the neural network topology comprises applying one or more transformations selected from a group consisting of: (i) replacing the unlimited activation functions with restricted activation functions; and (ii) adjusting the connections or weights of the equivalent analog network such that, for one or more predetermined inputs, the difference between the outputs of the trained neural network and the equivalent analog network is minimized.

[0030] In some implementations, the method further includes the step of generating one or more lithographic masks for manufacturing a circuit that implements an equivalent analog network of the analog components based on a resistance matrix.

[0031] In some implementations, the method further comprises: (i) obtaining new weights for a trained neural network; (ii) calculating a new weight matrix for an equivalent analog network based on the new weights; (iii) generating a new resistance matrix for the new weight matrix; and (iv) generating a new lithographic mask for manufacturing a circuit that implements an equivalent analog network of analog components based on the new resistance matrix.

[0032] In some implementations, the trained neural network is trained using software simulations to generate weights.

[0033] In another embodiment, a method for a hardware implementation of a neural network is provided according to some implementations. The method includes the step of obtaining a neural network topology and weights of a trained neural network. The method also includes the step of calculating one or more connection constraints based on analog integrated circuit (IC) design constraints. The method also includes the step of converting the neural network topology into an equivalent sparsely connected network of analog components that satisfies one or more connection constraints. The method also includes the step of calculating a weight matrix for an equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between the analog components of the equivalent sparsely connected network.

[0034] In some implementations, the step of converting the neural network topology into an equivalent sparsely connected network of analog components depends on one or more connection constraints, the possible degree of input connections (N i ) and output connection degree (N o Includes a step of deriving ).

[0035] In some implementations, the neural network topology includes at least one densely connected layer having K inputs, L outputs, and a weight matrix (U). In this case, the step of transforming at least one densely connected layer is such that the input connection degree is N i Not exceeding, and the output connection degree is N o So as not to exceed, K inputs, L inputs and It includes the step of constructing an equivalent sparsely connected network having layers.

[0036] In some implementations, the neural network topology includes at least one densely connected layer having K inputs, L outputs, and a weight matrix (U). In this case, the step of transforming at least one densely connected layer involves K inputs, L outputs, and The method includes the step of constructing an equivalent sparsely connected network having layers. Each layer (m) has a corresponding weight matrix (U m It is represented as ), where the case of no connection is represented as 0, and the input connection degree is N i The output connection degree does not exceed N o Ensure that it does not exceed. Equation It is satisfied with a predetermined precision.

[0037] In some implementations, the neural network topology has K inputs and L outputs, and a maximum input connection degree (P i ), maximum output connection degree (P o It includes a single sparsely connected layer having a weight matrix (U) and ) and where connections are represented as 0. In this case, the step of transforming the single sparsely connected layer involves input connectivity N i The output connection degree does not exceed N o K inputs, L outputs, so as not to exceed A step of constructing an equivalent sparsely connected network having layers, wherein each layer (m) has a corresponding weight matrix (U m It includes the step of configuring the above, expressed as ), wherein thereis no connection is expressed as 0. Equation It is satisfied with a predetermined precision.

[0038] In some implementations, the neural network topology includes a convolutional layer having K inputs and L outputs. In this case, the step of converting the neural network topology into a sparsely connected network equivalent to analog components involves K inputs, L outputs, and a maximum input connection degree (P i ) and maximum output connection degree (P o ·P i ≤ Ni and P o ≤ N o It includes the step of decomposing into a single sparsely connected layer having ).

[0039] In some implementations, a weight matrix is ​​used to generate a schematic model for implementing an equivalent sparsely connected network.

[0040] In some implementations, the neural network topology includes recurrent neural layers. In such cases, the step of converting the neural network topology into a sparsely connected network equivalent to analog components includes converting the recurrent neural layers into one or more dense or sparsely connected layers having diagonal connections.

[0041] In some implementations, the neural network topology includes recurrent neural layers. In such cases, the step of converting the neural network topology into a sparsely connected network equivalent to analog components includes the step of decomposing the recurrent neural layer into several layers, at least one of which is a dense or sparsely connected layer having K inputs and L outputs and a weight matrix (U), and is represented as 0 if there are no connections.

[0042] In some implementations, the neural network topology consists of K inputs and a weight matrix (U ∈ R K It includes a single-layer perceptron having a computational neuron having ) and an activation function (F). In this case, the step of converting the neural network topology into an equivalent sparsely connected network of analog components comprises: (i) deriving a degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints; (ii) an equation The method comprises the steps of: calculating the number of layers (m) for an equivalent sparsely connected network using [the method]; and (iii) constructing an equivalent sparsely connected network having K inputs, m layers, and a degree of connectivity (N). The equivalent sparsely connected network includes one or more analog neurons in each of the m layers, each analog neuron in the first m-1 layers implements an identity transformation, and the analog neuron in the last layer implements the activation function (F) of a computational neuron of a single-layer perceptron. Additionally, in this case, the step of calculating the weight matrix for the equivalent sparsely connected network includes the step of calculating the weight vector (W) for the connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector (U). The system of equations includes K equations having S variables, where S is the equation It is calculated using .

[0043] In some implementations, the neural network topology comprises K inputs, a single-layer perceptron with L computational neurons, and a weight matrix (V) containing weight rows for each of the L computational neurons. In this case, the step of converting the neural network topology into an equivalent sparsely connected network of analog components comprises: (i) deriving a degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints; (ii) an equation A step of calculating the number of layers (m) for an equivalent sparsely connected network using [a]; (iii) a step of decomposing a single-layer perceptron into L single-layer perceptron networks, wherein each single-layer perceptron network comprises a respective computational neuron of L computational neurons; (iv) for each single-layer perceptron network of L single-layer perceptron networks, (a) a step of constructing an equivalent pyramidal subnetwork for each single-layer perceptron network having K inputs, m layers, and a degree of connectivity (N), wherein the equivalent pyramidal subnetwork comprises one or more respective analog neurons in each of the m layers, wherein each analog neuron of the first m-1 layers implements an identity transformation, and the analog neuron of the last layer implements an activation function of the respective computational neuron corresponding to each single-layer perceptron; and (b) a step of constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork, including connecting the inputs of each equivalent pyramidal subnetwork for the L single-layer perceptron networks to form an input vector having L*K inputs. Additionally, in this case, the step of calculating a weight matrix for the equivalent sparsely connected network comprises, for each single-layer perceptron network of the L signal layer perceptron networks, (i) a weight vector (U=V i (ii) a step of setting ) wherein the i-th row of the weight matrix (V) corresponds to each computational neuron corresponding to each single-layer perceptron network; and (ii) solving an equation system based on the weight vector (U) to obtain a weight vector (W) for the connections of each equivalent pyramidal sub-network. i It includes a step of calculating ). The equation system includes K equations with S variables, where S is an equation It is calculated using .

[0044] In some implementations, the neural network topology includes a multilayer perceptron with K inputs and S layers, wherein each layer (i) of the S layers has a corresponding set of computational neurons (L i ) and L i Corresponding weight matrices (V) containing weight rows for each computational neuron of the computational neurons i ...includes ). In this case, the step of converting the neural network topology into an equivalent sparsely connected network of analog components comprises: (i) deriving a degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints; (ii) a multilayer perceptron A step of decomposing into Q single-layer perceptron networks, wherein each single-layer perceptron network comprises Q computational neurons, and the step of decomposing into a multilayer perceptron comprises a step of duplicating one or more of the K inputs shared by the Q computational neurons; (iii) for each single-layer perceptron network of the Q single-layer perceptron networks, (a) equation A step of calculating the number of layers (m) for each equivalent pyramidal subnetwork using K i,j is the number of inputs to each computational neuron of the multilayer perceptron, the computational step, and (b) K i,jA step of constructing an equivalent pyramidal subnetwork for each single-layer perceptron network having inputs, m layers, and degree of connection (N), wherein the equivalent pyramidal subnetwork comprises one or more analog neurons in each of the m layers, wherein each analog neuron in the first m-1 layers implements an identity transformation, and the analog neuron in the last layer implements an activation function of a computational neuron corresponding to each single-layer perceptron network; and (iv) Q*K i, j The method includes the step of constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork, including connecting the inputs of each equivalent pyramidal subnetwork for the Q single-layer perceptron networks to form an input vector having inputs. Additionally, in this case, the step of calculating a weight matrix for the equivalent sparsely connected network comprises, for each single-layer perceptron network of the Q signal layer perceptron networks, (i) a weight vector ( (ii) a step of establishing ), wherein the i-th row of the weight matrix (V) corresponds to each computational neuron corresponding to each single-layer perceptron network, where j is the layer corresponding to each computational neuron of the multilayer perceptron, the calculating step, and (ii) solving a system of equations based on the weight vector (U) to obtain a weight vector (W) for the connections of each equivalent pyramidal sub-network. i It includes a step of calculating ). The equation system has S variables K i, j It includes equations, and S is an equation It is calculated using .

[0045] In some implementations, the neural network topology includes a convolutional neural network (CNN) with K inputs and S layers, where each layer (i) of the S layers has a corresponding set of computational neurons (L i ) and L i Corresponding weight matrices (V) containing weight rows for each computational neuron of the computational neurons i ...includes ). In this case, the step of converting the neural network topology into an equivalent sparsely connected network of analog components comprises: (i) a step of deriving a degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints; (ii) the CNN A step of decomposing into Q single-layer perceptron networks, wherein each single-layer perceptron network comprises Q computational neurons, and the step of decomposing the CNN comprises a step of duplicating one or more of the K inputs shared by the Q computational neurons; (iii) for each single-layer perceptron network of the Q single-layer perceptron networks, (a) equation A step of calculating the number of layers (m) for each equivalent pyramidal subnetwork using, where j is the layer corresponding to each computational neuron in the CNN, and K i, j is the number of inputs for each computational neuron in the CNN, the calculating step; and (b) K i,jA step of constructing an equivalent pyramidal subnetwork for each single-layer perceptron network having inputs, m layers, and degree of connection (N), wherein the equivalent pyramidal subnetwork comprises one or more analog neurons in each of the m layers, wherein each analog neuron in the first m-1 layers implements an identity transformation, and the analog neuron in the last layer implements an activation function of a computational neuron corresponding to each single-layer perceptron network; and (iv) Q*K i, j The method includes the step of constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork, including connecting the inputs of each equivalent pyramidal subnetwork for the Q single-layer perceptron networks to form an input vector having inputs. Additionally, in this case, the step of calculating a weight matrix for the equivalent sparsely connected network comprises, for each single-layer perceptron network of the Q signal layer perceptron networks, (i) a weight vector ( (ii) a step of setting ) wherein the i-th row of the weight matrix (V) corresponds to each computational neuron corresponding to each single-layer perceptron network, where j is the layer corresponding to each computational neuron of the CNN; and (ii) solving an equation system based on the weight vector (U) to obtain a weight vector (W) for the connections of each equivalent pyramidal sub-network. i It includes a step of calculating ). The equation system is K having S variables i,j It includes equations, and S is an equation It is calculated using .

[0046] In some implementations, the neural network topology is a layer (L) having K inputs and K neurons. P ), a layer having L neurons (L n ) and weight matrix (W ∈ R LΥK Includes ), where R is a set of real numbers, and hierarchy (L p Each neuron of ) is in the hierarchy (L n It is connected to each neuron of ), and the hierarchy (L n Each neuron of ) performs an activation function (F), and the layer (L n The output of ) is the equation for input x It is to be calculated using . In this case, the step of converting the neural network topology into a sparsely connected network equivalent to analog components comprises: (i) the degree of possible input connections (N) according to one or more connection constraints. I > 1) and possible output connection extent (N O Step of deriving >1); (ii) Based on the decision, a layer (LA) having K analog neurons performing an identity activation function p ), performing the identity activation function A layer (LA) having 2 analog neurons h A layer (LA) having L analog neurons that perform ), and an activation function (F). o Includes ) but hierarchy (LA p Each analog neuron of ) is N O Having outputs, layer (LA h Each analog neuron of ) is N I Number of inputs and N O It has fewer than 1 output, and the hierarchy (LA o Each analog neuron of ) N I It includes the step of performing a trapezoidal transformation, which includes the step of constructing a 3-layer analog network having inputs. Additionally, in some cases, the step of calculating a weight matrix for an equivalent sparsely connected network includes layers (LA o The total output of ) is the equation To be calculated using, of the variables Matrix equations containing k equations Solving for the sparse weight matrix (W o and W h Includes the step of generating ). sparse weight matrix (W o ∈ R K×M ) is a hierarchy (LA p ) and hierarchy (LA h Represents the connections between ), and the sparse weight matrix (W h ∈ R M×L ) is a hierarchy (LA h ) and hierarchy (LA o Represents the connections between ).

[0047] In some implementations, the step of performing the trapezoidal transformation is, In accordance with the decision, (i) A sublayer (L) having as many as K' neurons p1 ) and a sublayer (L) having (K-K') neurons p2 To obtain ) hierarchy (L p (ii) a step of dividing ); (ii) a sublayer (L) having K' neurons p1 For ), a step of performing a configuring step and a generating step; and (iii) a sublayer having K-K' neurons (L p2 Regarding ), it further includes a step of repeatedly performing the steps of dividing, configuring, and generating.

[0048] In some implementations, the neural network topology includes a multilayer perceptron network. In such cases, the method further includes the step of iteratively performing a trapezoidal transformation for each consecutive pair of layers of the multilayer perceptron network and calculating a weight matrix for an equivalent sparsely connected network.

[0049] In some implementations, the neural network topology comprises a recurrent neural network (RNN) comprising (i) the computation of a linear combination of two fully connected layers, (ii) element-wise addition, and (iii) the computation of a non-linear function. In such cases, the method further comprises the step of performing a trapezoidal transformation on (i) the two fully connected layers and (ii) the computation of the non-linear function, and computing a weight matrix for an equivalent sparsely connected network.

[0050] In some implementations, the neural network topology comprises a Long Short-Term Memory (LSTM) network or a Gated Recurrence Unit (GRU) network comprising (i) a linear combination of multiple fully connected layers, (ii) element-wise addition, (iii) Hadamard products, and (iv) multiple non-linear function calculations. In such cases, the method further comprises the step of performing a trapezoidal transformation on (i) two fully connected layers and (ii) multiple non-linear function calculations and calculating a weight matrix for an equivalent sparsely connected network.

[0051] In some implementations, the neural network topology comprises a convolutional neural network (CNN) comprising (i) a plurality of partially connected layers and (ii) one or more fully connected layers. In such cases, the method further comprises: (i) transforming the plurality of partially connected layers into equivalent fully connected layers by inserting missing connections having zero weights; and (ii) for each consecutive pair of equivalent fully connected layers and one or more fully connected layers, iteratively performing a ladder transformation and calculating a weight matrix for an equivalent sparsely connected network.

[0052] In some implementations, the neural network topology consists of K inputs, L outputs, and a weight matrix (U ∈ R LΥKIt includes ), where R is a set of real numbers, and each output neuron performs an activation function (F). In this case, the step of converting the neural network topology into a sparsely connected network equivalent to analog components comprises: (i) the degree of possible input connections (N) according to one or more connection constraints. I > 1) and possible output connection extent (N O >1) Step of deriving; (ii) set A step of selecting a parameter (p) from; (iii) depending on the decision that p > 0, the pyramidal neural network in its output layer A step (28186) of configuring the first p layers of an equivalent sparsely connected network to have 10 neurons, wherein each neuron of the pyramidal neural network performs an identity function; and (iv) N p The step of performing an approximation transformation includes the step of constructing a trapezoidal neural network having inputs and L outputs. Each neuron in the last layer of the trapezoidal neural network performs an activation function (F), and all other neurons perform an identity function. In this case, the step of calculating the weight matrix for an equivalent sparsely connected network comprises (i) the step of generating weights for a pyramidal neural network, wherein (a) the following rules (a) As, here C is a non-zero constant, and Physician, the above rule, and (b) K i For all weights (j) of the neuron excluding, (a) a step of setting the weights of all neurons (i) of the first layer of the pyramidal neural network according to the above rule; and (b) a step of setting all other weights of the pyramidal neural network to 1, comprising the step of generating the above; and (ii) a step of generating weights for the trapezoidal neural network, wherein (a) an equation The step of generating the above includes: (b) setting the weights of each neuron (i) of the first layer of the trapezoidal neural network according to the above; and (b) setting the other weights of the trapezoidal neural network to 1.

[0053] In some implementations, the neural network topology has K inputs at the i-th layer, S layers, and weight matrix for computational neurons and the i-th layer ( ...includes ). In this case, the step of converting the neural network topology into an equivalent sparsely connected network of analog components comprises, for each layer (j) of the S layers of the multilayer perceptron, (i) for each layer (j) of the S layers of the multilayer perceptron, L j-1 inputs, L j output neurons and weight matrix (U j By performing an approximation transformation on each single-layer perceptron composed of ), each pyramid-trapezoidal network (PTNNX j (ii) a step of constructing ) and (ii) a step of constructing an equivalent sparsely connected network by stacking each pyramid-trapezoidal network.

[0054] In another aspect, a method for a hardware implementation of a neural network is provided according to some implementations. The method includes the step of obtaining a neural network topology and weights of a trained neural network. The method also includes the step of converting the neural network topology into an equivalent analog network of analog components comprising a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes the step of calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network. The calculating step, wherein each element of the weight matrix represents a respective connection; the method also includes the step of generating a resistor matrix for the weight matrix. Each element of the resistor matrix corresponds to a respective weight of the weight matrix and represents a resistance value.

[0055] In some implementations, the step of generating a resistance matrix for a weight matrix comprises: (i) possible resistance values ​​{R min , R max Obtain a predetermined range of} and within the predetermined range, the initial base resistance value (R base As a step of selecting ), (ii) within a predetermined range, within a limited set of lengths of resistance values ​​{R i , R j For all combinations of}, the range [-R base , R base Possible weights within ] A step of selecting a limited set of lengths of the resistance values ​​that provides the most uniform distribution of the resistance values; (iii) = this To make the resistor set value closest to , the maximum weight (w) of the input connections and bias of each neuron or layer of the equivalent analog network maxBased on ), for each analog neuron or each layer of the equivalent analog network, from a limited set of lengths of resistance values, resistance value = A step of selecting; and (iv) for each element of the weight matrix, an equation for all possible values ​​of R1 and R2 within a predetermined range of possible resistance values A step of selecting each first resistance value (R1) and each second resistance value (R2) that minimizes the error according to, wherein w is each element of the weight matrix and r err It includes the step of selecting, which is a predetermined relative tolerance value for possible resistance values.

[0056] In some implementations, a predetermined range of possible resistance values ​​includes resistances according to the nominal series E24 in the range of 100 KΩ to 1 MΩ.

[0057] In some implementations, R+ and R- are selected independently for each layer of the equivalent analog network.

[0058] In some implementations, R+ and R- are selected independently for each analog neuron of the equivalent analog network.

[0059] In some implementations, one or more first weights and one or more first inputs of a weight matrix represent one or more connections to a first operational amplifier of an equivalent analog network. In such cases, the method further comprises, before generating a resistance matrix, (i) modifying the one or more first weights by a first value; and (ii) configuring the first operational amplifier to multiply the first value by a linear combination of the one or more first weights and the one or more first inputs before performing an activation function.

[0060] In some implementations, the method further comprises: (i) obtaining a predetermined weight range; and (ii) updating the weight matrix according to the predetermined weight range so that the equivalent analog network generates an output similar to that of the trained neural network for the same input.

[0061] In some implementations, the trained neural network is trained so that each layer of the neural network topology has quantized weights.

[0062] In some implementations, the method further includes a step of retraining the trained neural network to reduce sensitivity to errors in weights or resistance values ​​that cause the equivalent analog network to produce different outputs compared to the trained neural network.

[0063] In some implementations, the method further includes the step of retraining the trained neural network to minimize the weight of any layer that is greater than the average absolute weight for that layer by a predetermined threshold.

[0064] In another aspect, a method for a hardware implementation of a neural network is provided according to some implementations. The method includes the step of obtaining a neural network topology and weights of a trained neural network. The method also includes the step of converting the neural network topology into an equivalent analog network of analog components comprising a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes the step of calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. The method also includes the step of generating a resistor matrix for the weight matrix. Each element of the resistor matrix corresponds to a respective weight of the weight matrix. The method also includes the step of pruning the equivalent analog network to reduce the number of operational amplifiers or resistors based on the resistor matrix in order to obtain an optimized analog network of analog components.

[0065] In some implementations, the step of pruning an equivalent analog network includes replacing resistors corresponding to one or more elements of a resistance matrix, where the resistance values ​​are less than a predetermined minimum threshold resistance value, with conductors.

[0066] In some implementations, the step of pruning an equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of a resistance matrix that exceed a predetermined maximum threshold resistance value.

[0067] In some implementations, the step of pruning an equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of a weight matrix that are approximately zero.

[0068] In some implementations, the step of pruning an equivalent analog network further includes the step of removing one or more analog neurons of the equivalent analog network that have no input connections.

[0069] In some implementations, the step of pruning an equivalent analog network comprises: (i) ranking the analog neurons of the equivalent analog network based on detecting the use of the analog neurons when performing calculations on one or more data sets; (ii) selecting one or more analog neurons of the equivalent analog network based on the ranking; and (iii) removing one or more analog neurons from the equivalent analog network.

[0070] In some implementations, the step of detecting the use of analog neurons includes: (i) constructing a model of an equivalent analog network using modeling software; and (ii) using the model to measure the propagation of analog signals to generate calculations for one or more data sets.

[0071] In some implementations, the step of detecting the use of analog neurons includes: (i) constructing a model of an equivalent analog network using modeling software; and (ii) using the model to measure the output signals of the model to generate calculations for one or more data sets.

[0072] In some implementations, the step of detecting the use of analog neurons includes: (i) constructing a model of an equivalent analog network using modeling software; and (ii) using the model to measure the power consumed by the analog neurons and generating calculations for one or more data sets.

[0073] In some implementations, the method further includes the step of recalculating a weight matrix for an equivalent analog network and updating a resistance matrix based on the recalculated weight matrix after the step of pruning the equivalent analog network and before generating one or more lithographic masks for manufacturing a circuit implementing the equivalent analog network.

[0074] In some implementations, the method further comprises, for each analog neuron of an equivalent analog network, (i) calculating a bias value for each analog neuron based on the weights of the trained neural network while calculating a weight matrix; (ii) removing each analog neuron from the equivalent analog network based on a determination that each bias value exceeds a predetermined maximum bias threshold; and (iii) replacing each analog neuron with a linear junction of the equivalent analog network based on a determination that each bias value is less than a predetermined minimum bias threshold.

[0075] In some implementations, before generating the weight matrix, the method further includes a step of decreasing the number of neurons in the equivalent analog network by increasing the number of connections from one or more analog neurons in the equivalent analog network.

[0076] In some implementations, the method further includes the step of pruning a trained neural network to update the neural network topology and the weights of the trained neural network before transforming the neural network topology, using pruning techniques for the neural network such that the equivalent analog network contains fewer than a predetermined number of analog components.

[0077] In some implementations, pruning is performed iteratively, taking into account the level of agreement or accuracy of the output between the trained neural network and the equivalent analog network.

[0078] In some implementations, the method further includes a step of performing network knowledge extraction before converting the neural network topology into an equivalent analog network.

[0079] In another aspect, an integrated circuit according to some implementations is provided. The integrated circuit is an analog network of analog components, comprising: (i) obtaining a neural network topology and weights of a trained neural network; (ii) converting the neural network topology into an equivalent analog network of analog components comprising a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents a respective analog neuron and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron; (iii) calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection; (iv) generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix; (v) generating one or more lithographic masks for manufacturing a circuit that implements the equivalent analog network of analog components based on the resistance matrix; and (vi) an analog network of analog components manufactured by a method comprising the step of manufacturing the circuit based on one or more lithographic masks using a lithographic process.

[0080] In some implementations, the integrated circuit further includes one or more digital-to-analog converters configured to generate an analog input to an equivalent analog network of analog components based on one or more digital signals.

[0081] In some implementations, the integrated circuit further includes an analog signal sampling module configured to process one-dimensional or two-dimensional analog inputs at a sampling frequency based on the number of inferences of the integrated circuit.

[0082] In some implementations, a voltage converter module is further included to scale down or scale up analog signals to match the operating range of multiple operational amplifiers.

[0083] In some implementations, a tact signal processing module configured to process one or more frames acquired from a CCD camera is further included.

[0084] In some implementations, the trained neural network is a Long Short-Term Memory (LSTM) network. In these cases, the integrated circuit further includes one or more clock modules that synchronize signal tactics and allow time-series processing.

[0085] In some implementations, the integrated circuit further includes one or more analog-to-digital converters configured to generate a digital signal based on the output of an equivalent analog network of analog components.

[0086] In some implementations, the integrated circuit further includes one or more signal processing modules configured to process one-dimensional or two-dimensional analog signals acquired in edge applications.

[0087] In some implementations, the trained neural network is trained using training datasets containing signals from a gas sensor array for different gas mixtures to selectively detect different gases in a gas mixture containing a predetermined amount of gases to be detected. In this case, the neural network topology is a 1D deep convolutional neural network designed to detect three binary gas components based on measurements by 16 gas sensors, and includes 16 sensor-specific 1D convolutional blocks, 3 shared or common 1D convolutional blocks, and 3 dense layers. In this case, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) delay blocks to generate delays of any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections.

[0088] In some implementations, the trained neural network is trained using training datasets containing thermal aging time series data for different MOSFETs that predict the remaining useful lifetime (RUL) of the MOSFET device. In this case, the neural network topology includes four LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively. In this case, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) 3,000 to 3,200 analog neurons, and (v) 123,000 to 124,000 connections.

[0089] In some implementations, a trained neural network is trained using a training dataset containing time-series data including discharge and temperature data during continuous use of various commercially available lithium-ion batteries to monitor the state of life (SOH) and state of charge (SOC) of lithium-ion batteries to be used in a battery management system (BMS). In this case, the neural network topology includes an input layer, two LSTM layers with 64 neurons in each layer, and a subsequent output dense layer with two neurons to generate SOC and SOH values. In this case, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) 1,200 to 1,300 analog neurons, and (v) 51,000 to 52,000 connections.

[0090] In some implementations, a trained neural network is trained using a training dataset containing time-series data including discharge and temperature data during continuous use of various commercially available lithium-ion batteries to monitor the state of life (SOH) and state of charge (SOC) of lithium-ion batteries to be used in a battery management system (BMS). In this case, the neural network topology includes an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron. In this case, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) 200 to 300 analog neurons, and (v) 2,200 to 2,400 connections.

[0091] In some implementations, the trained neural network is trained using training datasets containing speech commands to identify speech commands. In this case, the neural network topology is a depth-separable convolutional neural network (DS-CNN) with one neuron. In this case, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections.

[0092] In some implementations, a trained neural network is trained using a training dataset that includes photoplethysmography (PPG) data, accelerometer data, temperature data, skin electroreaction signal data, and reference heart rate data obtained from an ECG sensor for various individuals performing various physical activities for a predetermined time, in order to determine the heart rate during physical exercise based on PPG sensor data and 3-axis accelerometer data. In this case, the neural network topology includes two Conv1D layers with 20 kernels and 16 filters each performing time-series convolution, two LSTM layers with 16 neurons each, and two dense layers with 16 neurons and 1 neuron. In this case, the equivalent analog network comprises (i) delay blocks for generating any number of time steps, (ii) up to 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) 700 to 800 analog neurons, and (vi) 12,000 to 12,500 connections.

[0093] In some implementations, the trained neural network is trained to classify different objects based on pulse Doppler radar signals. In this case, the neural network topology includes a multi-scale LSTM neural network.

[0094] In some implementations, the training network is trained to perform human activity type recognition based on inertial sensor data. In this case, the neural network topology comprises three channel-wise convolution networks, each comprising a convolution layer with 12 filters and 64 kernel dimensions, followed by a max pooling layer and two common dense layers consisting of 1024 neurons and N neurons (where N is the number of classes). In this case, the equivalent analog network comprises (i) delay blocks for generating time steps of arbitrary coefficients, (ii) up to 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) a signal limit of 5, (v) 10 layers, (vi) 1,200 to 1,300 analog neurons, and (vi) 20,000 to 21,000 connections.

[0095] In some implementations, the trained neural network is further trained to detect abnormal patterns of human activity based on accelerometer data merged with heart rate data using convolution operations.

[0096] In another embodiment, a method for generating a library for a hardware implementation of a neural network is provided. The method comprises the step of acquiring a plurality of neural network topologies, each neural network topology corresponding to a respective neural network. The method also comprises the step of converting each neural network topology into an equivalent analog network of each analog component. The method further comprises the step of generating a plurality of lithographic masks for manufacturing a plurality of circuits, wherein each circuit implements an equivalent analog network of each analog component.

[0097] In some implementations, the method further includes the step of obtaining a neural network topology and weights of a trained neural network. The method also includes the step of selecting one or more lithographic masks from a plurality of lithographic masks based on comparing a new neural network topology with a plurality of neural network topologies. The method also includes the step of calculating a weight matrix for a new equivalent analog network based on the weights. The method also includes the step of generating a resistance matrix for the weight matrix. The method also includes the step of generating a new lithographic mask for manufacturing a circuit that implements a new equivalent analog network based on the resistance matrix and one or more lithographic masks.

[0098] In some implementations, the new neural network topology includes multiple sub-network topologies, and the step of selecting one or more lithographic masks is further based on comparing each sub-network topology with each network topology of the multiple network topologies.

[0099] In some implementations, one or more of the multiple sub-network topologies fail to compare with any of the multiple network topologies. In such cases, the method further comprises: (i) converting each of the one or more sub-network topologies into an equivalent analog sub-network of each of the analog components; and (ii) generating one or more lithographic masks for manufacturing one or more circuits (3228), wherein each of the one or more circuits implements an equivalent analog sub-network of each of the analog components.

[0100] In some implementations, the step of converting each network topology into each equivalent analog network comprises: (i) decomposing each network topology into a plurality of sub-network topologies; (ii) converting each sub-network topology into each equivalent analog sub-network of analog components; and (iii) configuring each equivalent analog sub-network to obtain each equivalent analog network.

[0101] In some implementations, the step of decomposing each network topology includes the step of identifying one or more layers of each network topology into multiple sub-network topologies.

[0102] In some implementations, each circuit is obtained by (i) generating a schematic diagram for each equivalent analog network of the analog components; and (ii) generating a circuit layout design based on the schematic diagrams.

[0103] In some implementations, the method further includes the step of combining one or more circuit layout designs before generating a plurality of lithographic masks for manufacturing a plurality of circuits.

[0104] In another embodiment, a method for optimizing the energy efficiency of analog neuromorphic circuits according to some implementations is provided. The method comprises the step of converting a neural network topology into an equivalent analog network of analog components comprising a plurality of operational amplifiers and a plurality of resistors. The analog network represents a trained neural network, each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron. The method also comprises the step of generating inferences using an integrated circuit for a plurality of test inputs, including simultaneously transmitting signals from one layer of the analog network to a subsequent layer. The method further comprises, while generating inferences using the integrated circuit: (i) determining whether the signal output levels of the plurality of operational amplifiers are balanced; and (ii) depending on the determination that the signal output levels are balanced, (a) determining a set of active analog neurons of the analog network that influences signal formation for the propagation of signals; and (during a predetermined time period, turning off power to one or more analog neurons of the analog network that are distinct from the set of active analog neurons).

[0105] In some implementations, the step of determining the set of active analog neurons is based on calculating the delay of signal propagation through the analog network.

[0106] In some implementations, the step of determining the set of active analog neurons is based on detecting the propagation of signals through the analog network.

[0107] In some implementations, the trained neural network is a feedforward neural network, the set of active analog neurons belongs to the active layer of the analog network, and the step of turning off power includes the step of turning off power to one or more layers prior to the active layer of the analog network.

[0108] In some implementations, a predetermined time period is calculated based on simulating the propagation of signals through an analog network, taking signal delay into account.

[0109] In some implementations, the trained neural network is a recurrent neural network (RNN), and the analog network further includes one or more analog components in addition to a plurality of operational amplifiers and a plurality of resistors. In this case, the method further includes the step of turning off power to one or more analog components for a predetermined time period, based on a determination that the signal output level is balanced.

[0110] In some implementations, the step of turning on power to one or more analog neurons of the analog network after a predetermined time period is further included.

[0111] In some implementations, the step of determining whether the signal output levels of multiple operational amplifiers are balanced is based on the detection of cases where one or more operational amplifiers of an analog network are outputting above a predetermined threshold signal level.

[0112] In some implementations, the method further includes the step of repeating the step of turning off a set of active analog neurons for a predetermined period and turning on a set of active analog neurons for a predetermined period while generating inference.

[0113] In some embodiments, the method further comprises: (i) determining, during each inference cycle, that the signal output level is balanced, (a) determining a first analog neuron layer of the analog network that influences signal formation for the propagation of signals; and (b) turning off power to one or more first analog neurons of the analog network prior to the first layer for a predetermined time period; and (ii) turning off power to one or more second analog neurons including one or more first analog neurons of the analog network and the first analog neuron layer for a predetermined time period during a second time period after the first time period.

[0114] In some implementations, one or more analog neurons are composed of analog neurons of one or more first layers of an analog network, and an active set of analog neurons is composed of analog neurons of a second layer of an analog network, and the second layer of the analog network is distinct from the layers of one or more first layers.

[0115] According to some embodiments, a method and apparatus are provided for clearing a speech signal from unwanted noise to clarify transmission for the benefit of a receiver (e.g., a human or machine interface) receiving such a clear signal. The technique described herein may be applied at the source, intermediate transmission point(s), or near the receiver. To process speech clarification, a neural network is trained to separate the speech signal from other unwanted signals in an input signal. This neural network is converted into an equivalent analog neural network using the technique described herein. This converted equivalent analog neural network is implemented in the form of an analog integrated circuit.

[0116] In another embodiment, a method for implementing an analog hardware convolutional neural network trained for speech intelligibility is provided. The method includes the step of obtaining the neural network topology and weights of the trained neural network. The method also includes the step of converting the neural network topology into an equivalent analog network of analog components. The method also includes the step of calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents one or more connections between the analog components of the equivalent analog network. For example, in the case of a dense layer, one weight matrix element represents a single connection. On the other hand, in the case of a convolutional layer, one weight matrix element represents multiple connections. To illustrate further, it is assumed that a layer multiplies N input signals by a single weight value (w). In this case, the input layer size is N, the output layer size is N, and there are N connections, each with weight w. In this way, one weight value represents multiple connections. The method also includes the step of generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components.

[0117] In some implementations, the neural network topology includes a Fourier transform layer and an inverse Fourier transform layer. The Fourier transform layer is useful for all speech-based applications because speech signals are frequency-based, allowing frequencies to be extracted via the Fast Fourier Transform (FFT). In some implementations, the FFT is implemented using a dense layer.

[0118] In some implementations, the neural network topology includes one or more of convolution layers, max pooling layers, and densely connected layers.

[0119] In some implementations, the neural network topology includes convolution layers. In such cases, the step of transforming the neural network topology includes, for each output of the convolution layer, defining a dependency relationship between each output and an associated input subset. The associated input subset is defined by the filter, kernel, padding, and stride parameters of the convolution layer; the step of defining a weight subset according to the dependency relationship of each output; and the step of configuring an analog neuron layer such that (i) each analog neuron corresponds to each output of the convolution layer, (ii) each analog neuron is connected to an associated input subset of a previous analog neuron layer of the equivalent analog network, and (iii) the input connection to each analog neuron is weighted according to a weight subset of the corresponding output of the convolution layer.

[0120] To illustrate these steps, consider a convolutional layer where each output depends on an input subset, and each input subset influences the output subset. In other words, unlike the all-to-all relationship of a dense layer, this is a many-to-many relationship. For a single batch, assume there is an input matrix (which is a 2D matrix for 1D convolution; a 3D matrix for 2D convolution) and an output matrix with the same number of dimensions as the input matrix. The first and second dimensions are also called spatial dimensions, and the last dimension is called the channel. Assume that a kernel size is provided (e.g., a 1D kernel for 1D convolution, a 2D kernel for 2D convolution). The weight matrix is ​​defined and sized using the equation: (kernel size multiplied by channel multiplied by filter). Here, the kernel size and filter are parameters of the weight matrix, and the channel is a measure of the input shape. For each filter (F), a subset of the weight matrix [:,:,F] is applied over the input data and slid along the spatial dimension using a step defined by the stride parameter. For 1D convolution, the kernel size is 3, the spatial dimension X is 7, and the stride = 2. The weight matrix is ​​applied over the spatial coordinates {1,2,3}, {3,4,5}, {5,6,7} and the channel dimension. Whenever the weight matrix is ​​applied, a single output element is computed. This element concatenation is weighted by the weight matrix subset as previously described and connected to the input data subset. For more filters, some implementations leave the input as is and duplicate the kernel and output for each filter using different kernel values. It should be noted that the description herein is for illustrative purposes only and various other implementations are possible.

[0121] In some implementations, the neural network topology includes a max pooling layer. In this case, the step of transforming the neural network topology includes the step of generating a multilayer network of analog neurons for a max pooling layer having a maximum number of inputs. In some implementations, the step of generating a multilayer network of analog neurons includes the step of generating a 2-input scheme containing two SNMs arranged in two layers, wherein the SNM of the last layer has a maximum of two inputs. In some implementations, the step of generating a multilayer network of analog neurons includes the step of generating a 3-input scheme containing three SNMs arranged in three layers, wherein the SNM of the last layer has a maximum of three inputs. In some implementations, the step of generating a multilayer network of analog neurons includes the step of generating a 4-input scheme containing four SNMs arranged in three layers, wherein the SNM of the last layer has a maximum of four inputs. In some implementations, the method comprises a 2-input scheme in which each node of the computation tree contains two SNMs arranged in two layers, wherein the SNM of the last layer has a maximum of two inputs; The method further includes the step of transforming a max pooling layer into a computation tree selected from a group consisting of: a 3-input scheme comprising three SNMs arranged in three layers having at most three inputs, and a 4-input scheme comprising four SNMs arranged in three layers having at most four inputs, wherein the last layer's SNM is an SNM. In some implementations, the method further includes the step of minimizing the number of layers in the computation tree. In some implementations, the method further includes the step of prioritizing the use of 4-input SNMs over the use of 3-input SNMs and 2-input SNMs.

[0122] In some implementations, the method further includes (i) a step of defining an analog neuron in the last layer of a multilayer neural network to perform an activation function other than ReLU, and (ii) a step of defining all other neurons in the multilayer analog network to perform ReLU without changing the final output of the multilayer network.

[0123] In some implementations, each layer of the trained neural network computes each output based on a respective mathematical function. In such cases, the step of converting the neural network topology into an equivalent analog network of analog components includes, for each layer of the trained neural network, identifying one or more function blocks based on a respective mathematical function for each layer. Each function block has a respective schematic implementation having a block output corresponding to the output of the respective mathematical function; and the step of generating a multilayer network of each analog neuron based on arranging one or more function blocks, wherein each analog neuron implements a respective function of one or more function blocks, and each analog neuron of the first layer of each multilayer network is connected to one or more analog neurons of the second layer of each multilayer network. In some implementations, one or more function blocks are block outputs It includes a weighted summation block having, where ReLU is a Rectified Linear Unit (ReLU) activation function or a similar activation function, Vi represents the i-th input, wi represents the weight corresponding to the i-th input, Bias represents the bias value, and Σ is the summation operator. In some implementations, one or more function blocks are block outputs It includes a weighted summation block having, where ReLU_X is a rectified linear unit (ReLU) activation function or a similar activation function that limits the output signal to a positive X, Vi represents the i-th input, wi represents the weight corresponding to the i-th input, bias represents the bias value, and Σ is the summation operator.

[0124] In some implementations, the neural network topology includes a convolutional layer having K inputs and L outputs. In such cases, the step of converting the neural network topology into an equivalent analog network involves, based on one or more connection constraints derived from analog integrated circuit (IC) design constraints, a possible degree of input connection (N i ) and output connection degree (N o The step of deriving ); and the step of transforming the convolutional layer involves transforming the convolutional layer into one with K inputs, L outputs, and a maximum input connection degree P i , and maximum output connection degree P o (Here, P i ≤N i and P o ≤N o It includes the step of decomposing into a single sparsely connected layer having ).

[0125] In some implementations, the analog component includes multiple operational amplifiers and multiple resistors. Each operational amplifier represents an analog neuron of an equivalent analog network, and each resistor represents a connection between two analog neurons. The step of generating a schematic model includes the step of generating a resistance matrix from a weight matrix. Each element of the resistance matrix represents (i) a respective resistance value and (ii) corresponds to a respective weight of the weight matrix. The step of selecting component values ​​of the analog component includes the step of performing gradient descent to identify possible resistance values ​​for multiple resistors.

[0126] In some implementations, the method further comprises the steps of: generating an equivalent digital network of digital components for one or more output layers of a neural network topology; and connecting the outputs of one or more layers of an equivalent analog network to the equivalent digital network of digital components.

[0127] In another embodiment, a system for a hardware implementation of a neural network is provided. The system includes one or more processors and memory for storing one or more programs configured to be executed by one or more processors. One or more programs include instructions for obtaining a neural network topology and weights of a trained neural network; converting the neural network topology into an equivalent analog network of analog components; and calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents one or more connections of the analog component asses of the equivalent analog network; and the method includes the step of generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components.

[0128] In another embodiment, a voice transmission device is provided and includes an integrated circuit for voice clarification. The integrated circuit includes an analog network of analog components manufactured by a method comprising: obtaining a neural network topology and weights of a trained neural network; converting the neural network topology into an equivalent analog network of analog components; calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values ​​for the analog components; and manufacturing the circuit according to the schematic model using a lithographic process.

[0129] In some implementations of a voice transmission device, the step of generating a schematic model further includes the step of generating a resistance matrix for a weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix; and further includes the step of generating one or more lithographic masks for manufacturing a circuit that implements an equivalent analog network of analog components based on the resistance matrix.

[0130] In some implementations of the voice transmission device, the voice transmission device is integrated into the mobile phone.

[0131] In some implementations, input from the mobile phone's microphone is input into an integrated circuit.

[0132] In some implementations of voice transmission devices, the output from the integrated circuit is input into the speaker of the mobile phone.

[0133] In some implementations of voice transmission devices, the integrated circuit is coupled to one or more other noise canceling devices.

[0134] In some implementations of voice transmission devices, an integrated circuit is coupled to one or more noise reduction software programs running on the voice transmission device.

[0135] In some embodiments, the computer system has one or more processors, memory, and a display. One or more programs include instructions for performing any one of the methods described herein.

[0136] In some embodiments, a non-transient computer-readable storage medium stores one or more programs configured to be executed by a computer system having one or more processors, memory and a display. One or more programs include instructions for performing any one of the methods described herein.

[0137] Accordingly, a method, system, and device used for hardware implementation of trained neural networks are disclosed. Brief explanation of the drawing

[0138] To better understand the aforementioned systems, methods, and graphical user interfaces, as well as additional systems, methods, and graphical user interfaces that provide data visualization analysis and data preparation, you should refer to the description of the implementations below, along with the following drawings, in which similar reference numbers indicate corresponding parts throughout the drawings. FIG. 1a is a block diagram of a system for a hardware implementation of a neural network trained using analog components according to some implementations. FIG. 1b is a block diagram of an alternative representation of the system of FIG. 1a for a hardware implementation of a neural network trained using analog components according to some implementations. FIG. 1c is a block diagram of yet another representation of the system of FIG. 1a for a hardware implementation of a neural network trained using analog components according to some implementations. FIG. 2a is a system diagram of a computing device according to some implementations. FIG. 2b illustrates optional modules of a computing device according to some implementations. FIG. 3a illustrates an exemplary process for generating schematic models of analog networks corresponding to a trained neural network, according to some implementations. FIG. 3b illustrates an exemplary manual prototyping process used to generate a target chip model, according to some implementations. Figures 4a, 4b, and 4c illustrate examples of neural networks that are converted into mathematically equivalent analog networks according to some implementations. Figure 5 illustrates an example of a mathematical model for a neuron according to some implementations. FIGS. 6a through 6c illustrate an exemplary process for an analog hardware implementation of a neural network for calculating the XOR of input values ​​according to some implementations. FIG. 7 illustrates an example of an exemplary perceptron according to some implementations. Figure 8 illustrates an exemplary pyramidal neural network according to some implementations. Figure 9 illustrates an exemplary pyramidal single neural network according to some implementations. FIG. 10 illustrates an example of a transformed neural network according to some implementations. FIGS. 11a through 11c illustrate the application of a T-transform algorithm to a single-layer neural network according to some implementations. FIG. 12 illustrates an exemplary recurrent neural network (RNN) according to some implementations. Figure 13a is a block diagram of an LSTM neuron according to some implementations. FIG. 13b illustrates delay blocks according to some implementations. Figure 13c is a neuron regime for an LSTM neuron according to some implementations. Figure 14a is a block diagram of a GRU neuron according to some implementations. Figure 14b is a neuron regime for a GUR neuron according to some implementations. Figures 15a and 15b are neuron regimes for variations of a single Conv1D filter according to some implementations. FIG. 16 illustrates an exemplary architecture of a transformed neural network according to some implementations. FIGS. 17a through 17c provide exemplary charts showing the dependency between output error and classification error or weighting error according to some implementations. FIG. 18 provides an exemplary scheme of a neuron model used for resistor quantization according to some implementations. FIG. 19a illustrates a schematic diagram of an operational amplifier made in CMOS according to some implementations. FIG. 19b illustrates a descriptive table for the exemplary circuit shown in FIG. 19a according to some implementations. FIGS. 20a through 20e illustrate schematic diagrams of LSTM blocks according to some implementations. FIG. 20f illustrates a descriptive table for the exemplary circuits illustrated in FIGS. 20a through 20d according to some implementations. FIGS. 21a through 21i illustrate schematic diagrams of a multiplier block according to some implementations. FIG. 21j illustrates an explanatory table for the schematic diagrams illustrated in FIGS. 21a through 21i according to some implementations. FIG. 22a illustrates a schematic diagram of a sigmoid neuron according to some implementations. FIG. 22b illustrates an explanatory table for the schematic diagram shown in FIG. 22a according to some implementations. FIG. 23a illustrates a schematic diagram of a hyperbolic tangent function block according to some implementations. FIG. 23b illustrates an explanatory table for the schematic diagram shown in FIG. 23a according to some implementations. FIGS. 24a through 24c illustrate schematic diagrams of a single neuron CMOS operational amplifier according to some embodiments. FIG. 24d illustrates an explanatory table for the schematic diagrams illustrated in FIGS. 24a through 24c according to some embodiments. FIGS. 25a through 25d illustrate schematic diagrams of variations of single-neuron CMOS operational amplifiers according to some embodiments. FIG. 25e illustrates an explanatory table for the schematic diagrams illustrated in FIGS. 25a through 25d according to some embodiments. FIGS. 26a through 26k illustrate exemplary weight distribution histograms according to some implementations. FIGS. 27a to 27j illustrate a flowchart of a hardware implementation method of a neural network according to some implementations. FIGS. 28a to 28s illustrate a flowchart of a hardware implementation method of a neural network according to hardware design constraints according to some implementations. FIGS. 29a to 29f illustrate a flowchart of a hardware implementation method of a neural network according to hardware design constraints according to some implementations. FIGS. 30a to 30m illustrate a flowchart of a hardware implementation method of a neural network according to hardware design constraints according to some implementations. FIGS. 31a through 31q illustrate a flowchart of a method for manufacturing an integrated circuit including an analog network of analog components according to some implementations. FIGS. 32a to 32e illustrate a flowchart of a method for creating a library for a hardware implementation of a neural network according to some implementations. FIGS. 33a through 33k illustrate a flowchart of a method for optimizing the energy efficiency of analog neuromorphic circuits (modeling a trained neural network) according to some implementations. Figure 34 illustrates a table describing the MobileNet v1 architecture according to some implementations. FIG. 35 illustrates an example of a 1-D convolutional neural network used for speech clarification according to some implementations. FIG. 36 illustrates an exemplary T-transform of a fully connected layer of the neural network illustrated in FIG. 35 according to some implementations. FIG. 37 illustrates an exemplary T-transform of the 1-D convolution of the neural network illustrated in FIG. 35 according to some implementations. FIG. 38 illustrates an exemplary T-transform of the max pooling operator of the neural network illustrated in FIG. 35 according to some implementations. Reference to implementations will now be made, examples of which are illustrated in the accompanying drawings. In the following detailed description, a number of specific details are described to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without requiring these specific details. Specific details for implementing the invention

[0139] FIG. 1a is a block diagram of a system (100) for a hardware implementation of a trained neural network using analog components according to some implementations. The system includes converting (126) the trained neural network (102) into an analog neural network (104). In some implementations, an analog integrated circuit constraint (184) constrains (146) the conversion (126) to create the analog neural network (104). Subsequently, the system derives (calculates or generates) weights (106) for the analog neural network (104) by a process often referred to as weight quantization (128). In some implementations, the analog neural network includes a plurality of analog neurons, each analog neuron is represented by an analog component such as an operational amplifier, and each analog neuron is connected to another analog neuron through a connection. In some implementations, the connection is represented using resistors that reduce the current flow between two analog neurons. In some implementations, the system converts (148) the weights (106) into resistance values ​​(112) for the connection. The system then generates one or more schematic models (108) for implementing an analog neural network (104) based on weights (106) (130). In some implementations, the system optimizes resistance values ​​(112) (or weights (106)) to form an optimized analog neural network (114) that is further used to generate the schematic models (108) (150). In some implementations, the system generates a lithographic mask (110) for connections (132) and / or generates a lithographic mask (120) for analog neurons (136). In some implementations, the system manufactures (134 and / or 138) an analog integrated circuit (118) for implementing the analog neural network (104). In some implementations, the system generates (152) a library of lithographic masks (116) based on the lithographic mask for connections (110) and / or the lithographic mask (120) for analog neurons.In some implementations, the system uses a library of lithographic masks (116) (154) to manufacture an analog integrated circuit (118). In some implementations, when a trained neural network (142) is retrained (142), the system regenerates (or recalculates) (144) lithographic masks for resistance values ​​(112) (and / or weights (106)), a schematic model (108), and / or connections (110). In some implementations, the system reuses lithographic masks (120) for analog neurons (120). In other words, in some implementations, only lithographic masks for weights (106) (or resistance values ​​(112) corresponding to changed weights) and / or connections (110) are regenerated. Since only the connections, weights, schematic model, and / or corresponding lithographic mask for the connections are regenerated as indicated by the dotted line (156), the process (or path) for manufacturing an analog integrated circuit for a retrained neural network is significantly simplified compared to existing technologies for hardware implementation of a neural network, and the time to market for retraining the neural network hardware is shortened.

[0140] FIG. 1b is a block diagram of an alternative representation of a system (100) for a hardware implementation of a neural network trained using analog components according to some implementations. The system includes neural network training (156) in software, determining connection weights, generating an electronic circuit equivalent to the neural network (158), calculating resistance values ​​corresponding to the weights of each connection (160), and subsequently generating a lithographic mask using the resistance values ​​(162).

[0141] FIG. 1c is a block diagram of another representation of a system (100) for a hardware implementation of a trained neural network using analog components, according to some implementations. The system is distributed as a software development kit (SDK) (180), according to some implementations. A user develops and trains a neural network (164) and inputs the trained neural network (166) into the SDK (180). The SDK estimates the complexity (168) of the trained neural network (166). If the complexity of the trained neural network can be reduced (e.g., some connections and / or neurons can be removed, some layers can be reduced, or the density of neurons can be changed), the SDK (180) cuts the trained neural network (178) and retrains the neural network (182) to obtain an updated trained neural network (166). Once the complexity of the trained neural network is reduced, the SDK (180) converts the trained neural network (166) into a sparse network of analog components (e.g., a pyramidal or trapezoidal network) (170). The SDK (180) also generates a circuit model (172) of the analog network. In some implementations, the SDK uses software simulation to estimate (176) the deviation of the output generated by the circuit model (172) for the trained neural network for the same input. If the estimated error exceeds a threshold error (e.g., a value set by the user), the SDK (180) prompts the user to reconfigure, redevelop, and / or retrain the neural network. In some implementations, although not illustrated, the SDK automatically reconfigures the trained neural network (166) to reduce the estimated error. This process is repeated several times until the error is reduced to below the threshold error. In FIG. 1c, the dotted line from block (176) ("Estimation of error occurring in circuit part") to block (164) ("Development and training of neural network") represents a feedback loop.For example, if a pruned network does not exhibit the desired accuracy, some implementations prune the network differently until the accuracy exceeds a predetermined threshold (e.g., 98% accuracy) for a given application. In some implementations, since pruning involves retraining the entire network, this process includes weight recalculation.

[0142] In some implementations, the components of the system (100) described above are implemented as computing modules in one or more computing devices or server systems. FIG. 2a is a system diagram of a computing device (200) according to some implementations. As used herein, the term “computing device” includes both personal devices (102) and servers. A computing device (200) generally includes one or more processing units / cores (CPUs) (202) for performing processing operations by executing modules, programs, and / or instructions stored in memory (214); one or more network or other communication interfaces (204); memory (214); and one or more communication buses (212) for interconnecting these components. The communication buses (212) may include circuits that interconnect and control communication between system components. A computing device (200) may include a user interface (206) comprising a display device (208) and one or more input devices or mechanisms (210). In some implementations, the input device / mechanism (210) includes a keyboard; in some implementations, the input device / mechanism includes a “soft” keyboard, which is displayed on the display device (208) as needed to allow the user to “press a key” appearing on the display (208). In some implementations, the display (208) and the input device / mechanism (210) include a touch screen display (also called a touch-sensitive display). In some implementations, the memory (214) includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, the memory (214) includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.In some implementations, memory (214) includes one or more storage devices located remotely from the CPU(s) (202). Memory (214), or alternatively, non-volatile memory device(s) within memory (214), includes a computer-readable storage medium. In some implementations, memory (214) or the computer-readable storage medium of memory (214) stores the following programs, modules, and data structures, or a subset thereof:

[0143] An operating system (216) including procedures for processing various basic system services and performing hardware-dependent tasks;

[0144] A communication module (218) used to connect a computing device (200) to other computers and devices through one or more communication network interfaces (204) (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan networks, etc.;

[0145] A trained neural network (220) comprising weights (222) and a neural network topology (224). Examples of input neural networks are described below with reference to FIGS. 4a through 4c, FIGS. 12, FIGS. 13a and 14a, according to some implementations;

[0146] A neural network conversion module (226) comprising a converted analog neural network (228), a mathematical formula (230), a basic function block (232), an analog model (234) (often referred to as a neuron model), and / or an analog integrated circuit (IC) design constraint (236). Exemplary operations of the neural network conversion module (226) are described below with reference to the flowcharts illustrated in at least FIGS. 5, 6a to 6c, 7, 8, 9, 10, and 11a to 11c, and FIGS. 27a to 27j, and FIGS. 28a to 28s; and / or

[0147] A weight matrix calculation (often referred to as weight quantization) module (238) comprising weights (272) of the transformed network and optionally a resistance calculation module (240) and a resistance value (242). Exemplary operations of the weight matrix calculation module (238) and / or weight quantization are described with reference to at least FIGS. 17a through 17c, FIGS. 18 and FIGS. 29a through 29f, according to some implementations.

[0148] Some implementations include one or more optional modules (244) as shown in FIG. 2b. Some implementations include an analog neural network optimization module (246). Examples of analog neural network optimization are described below with reference to FIG. 30a through 30m according to some implementations.

[0149] Some implementations include a lithographic mask generation module (248) that further includes a lithographic mask (250) for resistors (corresponding to connections) and / or a lithographic mask for analog components other than resistors (e.g., operational amplifiers, multipliers, delay blocks, etc.). In some implementations, the lithographic mask is generated based on the chip design layout according to the chip design using Cadence, Synopsys, or Mentor Graphics software packages. In some implementations, a design kit from a silicon wafer manufacturing plant (often referred to as a fab) is used. The lithographic mask is intended for use at a specific fab that provides the design kit (e.g., TSMC 65 nm design kit). The generated lithographic mask file is used to manufacture the chip at the fab. In some implementations, the chip design based on the Cadence, Mentor Graphics, or Synopsys software packages is generated semi-automatically in the SPICE or Fast SPICE (Mentor Graphics) software packages. In some implementations, users with chip design skills drive the conversion from SPICE or Fast SPICE circuits to Cadence, Mentor Graphics, or Synopsis chip designs. Some implementations combine Cadence design blocks for single-neuron units to establish appropriate interconnections between the blocks.

[0150] Some implementations include a library generation module (254) that further includes a library of lithographic masks (256). Examples of library generation are described below with reference to FIGS. 32a through 32e according to some implementations.

[0151] Some implementations include an integrated circuit (IC) fabrication module (258) further comprising an analog-to-digital converter (ADC), a digital-to-analog converter (DAC), or similar other interface (260), and / or a fabricated IC or model (262). Exemplary integrated circuits and / or related modules according to some implementations are described below with reference to FIGS. 31a through 31q.

[0152] Some implementations include an energy efficiency optimization module (264) further comprising an inference module (266), a signal monitoring module (268), and / or a power optimization module (270). Examples of energy efficiency optimizations according to some implementations are described below with reference to FIGS. 33a through 33k.

[0153] Each of the executable modules, applications, or sets of procedures identified above may be stored in one or more of the memory devices mentioned above and corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) identified above do not need to be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or rearranged in various implementations. In some implementations, memory (214) stores a subset of the modules and data structures identified above. Also, in some implementations, memory (214) stores additional modules or data structures not described above.

[0154] FIG. 2a illustrates a computing device (200), but FIG. 2a is intended more as a functional description of various features that may exist rather than a structural schematic diagram of the embodiments described herein. In practice, and as recognized by those skilled in the art, items depicted separately may be combined and some items may be separated.

[0155] An exemplary process for creating a schematic model of an analog network

[0156] FIG. 3a illustrates an exemplary process (300) for generating schematic models of analog networks corresponding to a trained neural network, according to some implementations. As illustrated in FIG. 3a, a trained neural network (302) (e.g., MobileNet) is converted (322) into a target or equivalent analog network (304) (using a process often called a T-transform). The target neural network (often called a T-network) (304) is exported to SPICE (324) using a single neuron model (SNM) (324), which is then exported from SPICE to CADENCE, and the entire on-chip design uses a CADENCE model (308). The CADENCE model (308) is cross-validated (328) against the initial neural network for one or more validation inputs.

[0157] In the descriptions above and below, a mathematical neuron is a mathematical function that receives one or more weighted inputs and generates a scalar output. In some implementations, mathematical neurons may have memory (e.g., long short-term memory (LSTM), recurrent neurons). A trivial neuron is a function representing an 'ideal' mathematical neuron, It is a mathematical neuron that performs [the function], where f(x) is the activation function. An SNM is a schematic model containing analog components (e.g., operational amplifiers, resistors (R1, ..., Rn), and other components) that represent a specific type of mathematical neuron (e.g., a trivial neuron) in a schematic form. The SNM output voltage depends on the K input voltages and the corresponding formula for the SNM component values. It is expressed as such. According to some implementations, if the component values ​​are appropriately selected, the SNM formula is identical to the mathematical neuron formula with the desired set of weights. In some implementations, the set of weights is entirely determined by the resistances used in the SNM. The target (analog) neural network (304) (often called a T-network) is a set of mathematical neurons that define the SNM representation and the weighted connections between them, forming the neural network. The T-network is subject to several constraints, such as an inbound limit (the maximum limit of inbound connections for all neurons in the T-network), an outbound limit (the maximum limit of outbound connections for all neurons in the T-network), and a signal range (e.g., all signals must be within a predefined signal range). The T-transform (322) is the process of converting some desired neural network, such as MobileNet, into the corresponding T-network. The SPICE model (306) is a SPICE neural network model of the T-network (304), where each mathematical neuron is replaced by one or more corresponding SNMs. The Cadence NN model (310) is a Cadence model of a T-network (304), wherein each mathematical neuron is replaced by one or more corresponding SNMs. Additionally, as described herein, two networks (L and M) for all neuron outputs of these networks In this case, they have mathematical equivalence, where eps is relatively small (e.g., between 0.1 and 1% of the operating voltage range). Additionally, the two networks (L and M) are given a verification input data set {I1, ..., I n Regarding}, when the classification results are mostly identical, i.e. In the case where eps is relatively small, they have functional equivalence.

[0158] FIG. 3b illustrates an exemplary manual prototyping process used to generate a target chip model (320) based on an SNM model (314) on Cadence, depending on some implementations. The following description uses Cadence, but depending on some implementations, alternative tools from Mentor Graphic Design or Synopsys (e.g., Synopsys Design Kit) may be used instead of the Cadence tool. The process includes selecting SNM constraints, including inbound and outbound limits and signal limits; selecting analog components for connections between neurons (e.g., resistors including specific resistor array techniques); and developing the Cadence SNM model (314). A prototype SNM model (316) (e.g., a PCB prototype) is developed (330) based on the SNM model (314) on Cadence. The prototype SNM model (316) is compared with a SPICE model for equivalence. In some implementations, when the neural network meets the equivalence requirements, the neural network for the on-chip prototype is selected. Because the size of the neural network is small, the T-transform can be hand-verified for equivalence. Subsequently, an on-chip SNM model (318) is created (332) based on the SNM model prototype (316). The on-chip SNM model is optimized as much as possible depending on some implementations. In some implementations, the on-chip density for the SNM model is calculated after the SNM is completed and before creating a target chip model (320) based on the on-chip SNM model (318) (334). During the prototyping process, the physician may iterate on selecting a neural network task or application and a specific neural network (e.g., a neural network with about 1 million to 1.1 million neurons), performing the T-transform, building the Cadence neural network model, and designing the interface and / or target chip model.

[0159] Exemplary input neural network

[0160] FIGS. 4a, 4b, and 4c illustrate examples of trained neural networks (e.g., neural network (220)) that are input into the system (100) and converted into mathematically equivalent analog networks, depending on some implementations. FIG. 4a illustrates an exemplary neural network (often referred to as an artificial neural network) composed of artificial neurons that receive inputs, combine inputs using an activation function, and generate one or more outputs. Inputs include data such as images, sensor data, and documents. Typically, each neural network performs a specific task, such as object recognition. Networks include connections between neurons, where each connection provides the output of a neuron as input to another neuron. After training, a corresponding weight is assigned to each connection. As illustrated in FIG. 4a, neurons are typically composed of multiple layers, and each neuron layer is connected only to the neuron layer immediately before and after it. The input layer of the input neuron layer (402) receives external inputs (e.g., inputs (X1, X2, ..., X nIt receives )). The input layer (402) is followed by one or more hidden neuron layers (e.g., layers (404 and 406)), and is followed by an output layer (408) that generates outputs (410). Various types of connection patterns connect neurons in successive layers, such as a fully-connected pattern that connects all neurons in one layer to all neurons in the next layer, or a pooling pattern that connects the output of a group of neurons in one layer to a single neuron in the next layer. Unlike the neural network illustrated in FIG. 4a, also known as a feedforward network, the neural network illustrated in FIG. 4b includes one or more connections from neurons in one layer to other neurons in the same layer or to neurons in the previous layer. The example illustrated in FIG. 4b is an example of a recurrent neural network, comprising two input neurons (412 (receiving input (X1)) and 414 (receiving input (X2))) in an input layer followed by two hidden layers. The first hidden layer comprises fully connected neurons (416 and 418) with the neurons of the input layer, and neurons (420, 422 and 424) of the second hidden layer. The output of the neuron (420) of the second hidden layer is connected to the neuron (416) of the first hidden layer to provide a feedback loop. The hidden layer comprising the neurons (420, 422 and 424) is input to a neuron (426) of the output layer that generates an output (y).

[0161] FIG. 4c illustrates an example of a convolutional neural network (CNN) according to some implementations. Unlike the neural networks illustrated in FIG. 4a and 4b, the example illustrated in FIG. 4c includes different types of neural network layers, including first-stage layers for feature learning and second-stage layers for classification tasks such as object recognition. The feature learning stage includes a convolution and Rectified Linear Unit (ReLU) layer (430), followed by a pooling layer (432), followed by another convolution and ReLU layer (434), followed in turn by another pooling layer (436). The first layer (430) extracts features from the input (428) (e.g., an input image or part thereof), performs a convolution operation on the input, and performs one or more non-linear operations (e.g., ReLU, tanh, or sigmoid). A pooling layer, such as layer (432), reduces the number of parameters when the inputs are large. The output of the pooling layer (436) is flattened by layer (438) and input into a fully connected neural network having one or more layers (e.g., layers (440 and 442)). The output of the fully connected neural network is input into a softmax layer (444) to classify the output of layer (442) of the fully connected network to generate one of many different outputs (446) (e.g., an object class or type of the input image (428)).

[0162] Some implementations store the layout or configuration of input neural networks, including the number of neurons in each layer, the total number of neurons, the operation or activation function of each neuron, and / or the connections between neurons, in memory (214) as a neural network topology (224).

[0163] FIG. 5 illustrates an example of a mathematical model (500) for a neuron according to some implementations. The mathematical model includes incoming signals (502) that are input by multiplying synaptic weights (504) and summing by a unit sum (506). According to some implementations, the result of the unit sum (506) is input to a non-linear transformation unit (508) to generate an output signal (510).

[0164] FIGS. 6a through 6c illustrate an exemplary process for an analog hardware implementation of a neural network for calculating the XOR of input values ​​(classification of XOR results), according to some implementations. FIG. 6a illustrates a table (600) of possible input values ​​(X1 and X2) along the x-axis and y-axis, respectively. Expected result values ​​are indicated by hollow circles (representing a value of 1) and filled or dark circles (representing a value of 0)—this is a typical XOR problem with two input signals and two classes. If only one of the X1 and X2 values ​​is 1, but not both, the expected result is 1; otherwise, it is 0. The training set consists of four possible input signal combinations (binary values ​​for the X1 and X2 inputs). FIG. 6b illustrates a ReLU-based neural network (602) for solving the XOR classification of FIG. 6a, according to some implementations. The neurons use ReLU activation without using any bias values. The inputs (604 and 606) (corresponding to X1 and X2, respectively) are input to the first ReLU neuron (608-2). The inputs (604 and 606) are also input to the second ReLU neuron (608-4). The results of the two ReLU neurons (608-2 and 608-4) are input to a third neuron (608-6) that performs a linear sum of the input values ​​to generate an output value (510) (Out value). The neural network (602) has weights -1 and 1 for the ReLU neuron (608-2) (for input values ​​X1 and X2, respectively), weights 1 and -1 for the ReLU (for input values ​​X1 and X2, respectively), and weights 1 and 1 (for the outputs of the ReLU neurons (608-2 and 608-4), respectively. In some implementations, the weights of the trained neural networks are stored in memory (214) as weights (222).

[0165] FIG. 6c illustrates an exemplary equivalent analog network for network (602) according to some implementations. The analog equivalent inputs (614 and 616) of the X1 and X2 inputs (604 and 606) are input to the analog neurons N1 (618) and N2 (620) of the first layer. The neurons (N1 and N2) are densely connected to the neurons (N3 and N4) of the second layer. The neurons of the second layer (i.e., neurons N3 (622) and N4 (624)) are connected to an output neuron N5 (626) that produces an output (Out) (identical to the output (610) of the network (602)). The neurons (N1, N2, N3, N4, and N5) have a ReLU (maximum value = 1) activation function.

[0166] Some implementations use Keras learning that converges in about 1000 iterations and generate weights for the connections. In some implementations, the weights are stored in memory (214) as part of the weights (222). In the following example, the data format is 'neuron[first link weight, second link weight, bias]'.

[0167] N1 [-0.9824321, 0.976517, -0.00204677];

[0168] N2 [1.0066702, -1.0101418, -0.00045485];

[0169] N3 [1.0357606, 1.0072469, -0.00483723];

[0170] N4 [-0.07376373, -0.7682612, 0.0]; and

[0171] N5 [1.0029935, -1.1994369, -0.00147767].

[0172] Next, to calculate the resistance values ​​for the connections between neurons, some implementations calculate the resistance range. In some implementations, nominal resistance values ​​(R+, R-) of 1 MΩ, a possible resistance range of 100 KΩ to 1 MΩ, and a nominal series E24 are set. Some implementations calculate the resistance values ​​w1, w2, and wbias for each connection as follows. For each weight value wi (e.g., weights (222)), some implementations evaluate all possible (Ri-, Ri+) resistance pair options within the selected nominal series and the minimum error value, Select a pair of resistors that generate. The following table provides exemplary values ​​for the weights (w1, w2, and bias) for each connection according to some implementations.

[0173]

[0174] Exemplary Benefits of Transformed Neural Networks

[0175] Before describing examples of transformations, it is worth mentioning some advantages of transformed neural networks compared to existing architectures. As described herein, input-trained neural networks are transformed into pyramidal or trapezoidal analog networks. Some of the advantages of pyramidal or trapezoidal networks over crossbar networks include low latency, simultaneous analog signal propagation, manufacturability using standard integrated circuit (IC) design elements including resistors and operational amplifiers, high parallelism in computation, high accuracy (e.g., accuracy increases with the number of layers compared to existing methods), tolerance for error(s) at each weight and / or each connection (e.g., pyramids balance errors), low RC (low resistive capacitance delay associated with signal propagation through the network), and / or the ability to manipulate the biases and functions of each neuron in each layer of the transformed network. Furthermore, pyramids are excellent computational blocks in themselves because they are multi-level perceptrons capable of modeling the entire neural network with a single output. Networks with multiple outputs are implemented using different pyramid or trapezoidal geometric structures depending on the implementation. A pyramid can be thought of as a multilayer perceptron with one output and multiple layers (e.g., N layers), where each neuron has n inputs and one output. Similarly, a trapezoidal is a multilayer perceptron where each neuron has n inputs and m outputs. Each trapezoidal is a pyramidal network where each neuron has n inputs and m outputs, where n and m are limited by IC analog chip design constraints depending on the implementation.

[0176] Some implementations perform lossless conversion of any trained neural network into pyramidal or trapezoidal subsystems. Therefore, pyramidal and trapezoidal structures can be used as general-purpose building blocks for converting any neural network. The advantage of pyramidal or trapezoidal neural networks is the possibility of realizing any neural network using standard IC analog elements (e.g., operational amplifiers, resistors, and signal delay lines in the case of recurrent neurons) using standard lithography techniques. It is also possible to limit the weights of the converted networks to a fixed interval. In other words, according to some implementations, lossless conversion is performed with weights limited to some predefined range. Another advantage of using pyramidal or trapezoidal structures is that they provide low latency through high levels of parallel processing or the simultaneous propagation of analog signals during signal processing that is continuously accelerating. Furthermore, many modern neural networks are sparsely connected networks, and they perform much better when converted to pyramids than to crossbars (e.g., being more compact, having lower RC values, and having no leakage current), and pyramid and trapezoidal networks are relatively more compact than crossbar-based memristor networks.

[0177] Furthermore, analog neuromorphic trapezoidal chips possess several properties not typically found in analog devices. For example, the signal-to-noise ratio does not increase with the number of cascades in the analog chip, external noise is suppressed, and the influence of temperature is significantly reduced. These characteristics make trapezoidal analog neuromorphic chips similar to digital circuits. For instance, individual neurons level the signal based on operational amplifiers and operate at frequencies ranging from 20,000 to 100,000 Hz; according to some implementations, they are unaffected by noise or signals at frequencies higher than their operating range. Trapezoidal analog neuromorphic chips also perform output signal filtration due to the characteristics of how the operational amplifiers function. These trapezoidal analog neuromorphic chips suppress synphase noise. Noise is also significantly reduced due to the low resistance outputs of the operational amplifiers. Due to signal leveling at each operational amplifier output and the synchronization of the amplifiers, parameter drift caused by temperature does not affect the signals of the final outputs. Trapezoidal analog neuromorphic circuits are resistant to errors and noise in input signals, and to deviations in resistor values ​​corresponding to the neural network's weight values. Trapezoidal analog neuromorphic networks are also resistant to all kinds of system errors, such as errors in resistor value settings, provided that these errors are the same across all resistors, due to the very nature of analog neuromorphic trapezoidal circuits based on operational amplifiers.

[0178] Exemplary lossless transformation (T-transform) of a trained neural network

[0179] In some implementations, the exemplary transformations described herein are performed by a neural network transformation module (226) that transforms neural networks (220) trained based on mathematical formulas (230), basic function blocks (232), analog component models (234), and / or analog design constraints (236) to obtain transformed neural networks (228).

[0180] FIG. 7 illustrates an example of an exemplary perceptron (700) according to some implementations. The perceptron includes K = 8 inputs and 8 neurons (702-2, ..., 702-16) in an input layer receiving 8 inputs. There is an output layer with 4 neurons (704-2, ..., 704-8) in an output layer corresponding to L = 4 outputs. The neurons in the input layer are fully connected to the neurons in the output layer, forming 8 × 4 = 32 connections. It is assumed that the weights of the connections are represented by a weight matrix (WP) (element WP i,j (This corresponds to the weight of the connection between the i-th neuron of the input layer and the j-th neuron of the output layer). It is assumed that each neuron performs an activation function (F).

[0181] FIG. 8 illustrates an exemplary Pyramid-Neural Network (P-NN) (800), which is a type of Target-Neural Network (T-NN or TNN) identical to the perceptron shown in FIG. 7, according to some implementations. To perform this conversion of the perceptron (Fig. 7) to the PN-NN architecture (Fig. 8), it is assumed that for the T-NN, the number of inputs is limited to Ni = 4 and the number of outputs is limited to No = 2. The T-NN includes an input layer (LTI) of neurons (802-2, ..., 802-34), which is formed by concatenating two copies of the input layer of neurons (802-2, ..., 802-16) and has a total of 2 × 8 = 16 input neurons. A set of neurons (804) containing neurons (802-20, ..., 802-34) is a copy of neurons (802-2, ..., 802-18), and the input is duplicated. For example, the input to neuron (802-2) is also input to neuron (802-20), and the input (20) to neuron (802-4) is also input to neuron (802-22). FIG. 8 also includes a hidden layer (LTH1) of linear neurons, neurons (806-02, ..., 806-16) ((2 x 8) / 4 = 8 neurons). Each group of Ni neurons in the input layer (LTI) is fully connected to two neurons in the LTH1 layer. FIG. 8 also includes an output layer (LTO) with (2 x 8) / 4 = 4 neurons (808-02, ..., 808-08), each neuron performing an activation function (F). Each neuron in the layer (LTO) is connected to distinct neurons in different groups of the layer (LTH1). The network illustrated in FIG. 8 includes 40 connections. Some implementations perform weight matrix calculations for the P-NN of FIG. 8 as follows.The weights (WTH1) for the hidden layer (LTH1) are calculated from the weight matrix (WP), and the weights (WTO) corresponding to the output layer (LTO) form a sparse matrix in which the elements are equal to 1.

[0182] FIG. 9 illustrates a Pyramid Single Neural Network (PSNN) (900) corresponding to the output neurons of FIG. 8, according to some implementations. The PSNN includes a layer (LPSI) of input neurons (902-02, ..., 902-16) (corresponding to the 8 input neurons of the network (700) of FIG. 7). The hidden layer (LPSH1) includes 8 / 4 = 2 linear neurons (904-02 and 904-04), and each group of Ni neurons of the LTI is connected to one neuron of the LPSH1 layer. The output layer (LPSO) consists of one neuron (906) having an activation function (F) connected to both of the neurons (904-02 and 904-04) of the hidden layer. To compute the weight matrix for the PSNN (900), some implementations compute a vector WPSH1 for the LPSH1 layer that is equal to the first row of WP. For the LPSO layer, some implementations compute a weight vector WPSO with two elements, each element equal to 1. This process is repeated for the first, second, third, and fourth output neurons. A P-NN, such as the network illustrated in Fig. 8, is a union of PSNNs (for 4 output neurons). The input layer of every PSNN is an individual copy of the input layer of P. In this example, the P-NN (800) includes an input layer with 8 × 4 = 32 inputs, a hidden layer with 2 × 4 = 8 neurons, and an output layer with 4 neurons.

[0183] An exemplary transformation using target neurons with N inputs and 1 output

[0184] In some implementations, the exemplary transformations described herein are performed by a neural network transformation module (226) that transforms neural networks (220) trained based on mathematical formulas (230), basic function blocks (232), analog component models (234), and / or analog design constraints (236) to obtain transformed neural networks (228).

[0185] A single-layer perceptron with one output

[0186] Assume that a single-layer perceptron SLP(K,1) includes K inputs and 1 output neuron with an activation function (F). Additionally, U ∈ R K It is assumed that is a vector of weights for SLP(K,1). The following algorithm, Neuron2TNN1, constructs a T-neural network (referred to as TN(N,1)) from T-neurons having N inputs and 1 output.

[0187] Algorithm Neuron2TNN1

[0188] 1. Construct an input layer for T-NN including all inputs of SLP(K,1).

[0189] 2. If K > N,

[0190] a. K input neurons Divide into groups such that all groups consist of fewer than N inputs.

[0191] b. The first hidden layer (LTH1) of the T-NN is constructed from m1 neurons, and each neuron performs an identity activation function.

[0192] c. Connect all input neurons of the group to the corresponding neurons in the next layer. Therefore, all neurons in LTH1 have fewer than N input connections.

[0193] d. Set the weights for the new connections according to the following equation:

[0194]

[0195]

[0196] 3. Otherwise (i.e., when K <= N),

[0197] a. Construct an output layer having one neuron that computes an activation function (F).

[0198] b. Connect input neurons to a single output neuron. There are K≤N connections.

[0199] c. Set the weights of the new connections using the following equation:

[0200]

[0201] d. Terminate the algorithm.

[0202] 4. Set l=1.

[0203] 5. m l In the case of N,

[0204] a. m l dogs' neurons Divided into groups, and each group consists of N or fewer neurons.

[0205] b. m l+1 Hidden layer (LTH) of T-NN from several neurons l+1 It consists of, and every neuron has an identity activation function.

[0206] c. Connect the input neurons of all groups to the corresponding neurons in the next layer.

[0207] d. Set the weights of the new connections according to the following equation:

[0208]

[0209]

[0210] e. Set l = l + 1.

[0211] 6. Other cases (where m >= N),

[0212] a. Construct an output layer having one neuron that computes an activation function (F).

[0213] b. All LTH l Connect the input neurons to a single output neuron.

[0214] c. Set the weights of the new connections according to the following equation:

[0215] a.

[0216] d. Terminate the algorithm.

[0217] 7. Repeat steps 5 and 6.

[0218] Here, [x] - the minimum integer is greater than or equal to x. The number of T-NN layers constructed by the algorithm Neuron2TNN1 is The total number of weights in the T-NN is as follows:

[0219]

[0220] Figure 10 illustrates an example of a T-NN configured according to some implementations. All layers except the first layer perform identity transformations on their inputs. The weight matrices of the configured T-NN have the following forms depending on some implementations.

[0221] Layer 1 (e.g., Layer (1002):

[0222]

[0223] Layers i = 2, 3, ..., h (e.g., layers(1004, 1006, 1008, and 1010)):

[0224]

[0225] The output value of the T-NN is calculated according to the following formula:

[0226]

[0227] The output of the first layer is calculated as an output vector according to the following formula:

[0228]

[0229] Multiply the acquired vector by the second layer weight matrix:

[0230]

[0231] All subsequent layers output a vector having components identical to a linear combination of some subvectors of x.

[0232] Finally, the output of the T-NN is as follows:

[0233]

[0234] This is the same value calculated in SLP(K,1) for the same input vector x. Therefore, the output values ​​of SLP(K,1) and the constructed T-NN are identical.

[0235] A single-layer perceptron with multiple outputs

[0236] Assume there is a single-layer perceptron SLP(K, L) with K inputs and L output neurons, and each neuron performs an activation function (F). Additionally, U ∈ R LΥK Assume that is the weight matrix for SLP(K, L). The following algorithm Layer2TNN1 constructs a T-neural network from neurons TN(N, 1).

[0237] Algorithm Layer2TNN1

[0238] 1. For all output neurons i=1, ..., L,

[0239] a. K inputs, 1 output neuron, and weight vector (U ij SLP composed of ), j=1,2, ..., K i Apply the Neuron2TNN1 algorithm to (K, 1). Consequently, the TNN i It is composed of.

[0240] 2. All TNNs iConstruct a PTNN by configuring it into a single neural network:

[0241] a. All TNNs i By concatenating the input vectors, the PTNN input has L groups of K inputs, and each group is a copy of the input layer of SLP(K, L).

[0242] All pairs SLP i (K, 1) and TNN i Since the output of is the same, the output of PTNN is equal to the output of SLP(K, L) for the same input vector.

[0243] Multilayer perceptron

[0244] A multilayer perceptron (MLP) is an MLP(K, S, L1, ... L S K inputs to the i-th layer, S layers, and L, represented as ). i Assume that it includes 2 computational neurons. Assume this is the weight matrix for the i-th layer.

[0245] The following is an exemplary algorithm for constructing a T-neural network from neurons TN(N, 1) according to some implementations.

[0246] MLP2TNN1 algorithm

[0247] 1. For all layers i=1, ..., S,

[0248] a. L i-1 inputs, L i output neurons, weight matrix (U i SLP composed of ) i (L i-1 , L i By applying the Layer2TNN1 algorithm to ), the result is PTNN i Constitutes.

[0249] 2. All PTNNs in a single neural network i Constructing an MTNN by stacking; TNN i-1 The output of is TNNi It is set as the input of.

[0250] All pairs SLP i (L i-1 , L i ) and PTNN i Since the output is identical, the output of the MTNN is the MLP(K, S, L1, ..., L for the same input vector. S It is the same as the output of ).

[0251] N I n inputs and N O An exemplary T-transform using target neurons having outputs

[0252] In some implementations, the exemplary transformations described herein are performed by a neural network transformation module (226) that transforms neural networks (220) trained based on mathematical formulas (230), basic function blocks (232), analog component models (234), and / or analog design constraints (236) to obtain transformed neural networks (228).

[0253] An exemplary transformation of a single-layer perceptron with multiple outputs

[0254] Assume that a single-layer perceptron SLP(K, L) comprises K inputs and L output neurons, and that each neuron performs an activation function (F). Additionally, U ∈ R L×K It is assumed that is the weight matrix for SLP(K, L). The following algorithm, depending on some implementations, describes the neurons TN(N I , N O Construct a T-neural network from ).

[0255] Algorithm Layer2TNNX

[0256] 1. Construct a PTNN from SLP(K,L) using the Layer2TNN1 algorithm (see description above). The PTNN has an input layer consisting of L groups of K inputs.

[0257] 2. From L groups Constructs n subsets. Each subset contains N O Includes groups of fewer than 10 input vector copies.

[0258] 3. Replace all subset groups with a single copy of the input vector.

[0259] 4. N from all input neurons O Construct PTNNX by reconstructing the connections of all subsets by creating output connections.

[0260] According to some implementations, the output of PTNNX is calculated by the same formulas as for PTNN (described above), so the outputs are identical.

[0261] FIGS. 11a through 11c illustrate two output neurons and a TN (N IThe application (1100) of the above algorithm to a single-layer neural network (NN) having , 2) is illustrated. FIG. 11a illustrates an example of a source or input NN according to some implementations. K inputs are input to two neurons 1 and neuron 2 belonging to layer (1104). FIG. 11b illustrates a PTNN configured after the first step of the algorithm according to some implementations. The PTNN consists of two parts implementing subnets corresponding to the output neurons 1 and neuron 2 of the NN illustrated in FIG. 11a. In FIG. 11b, the input (1102) is duplicated and input to two sets of input neurons (1106-2 and 1106-4). Each set of input neurons is connected to a subsequent neuron layer having two sets of neurons (1108-2 and 1108-4), each neuron set containing m1 neurons. The input layer is followed by identity transformation blocks (1110-2 and 1110-4), each block containing one or more layers having an identity weight matrix. The output of the identity transformation block (1110-2) is connected to an output neuron (1112) (corresponding to output neuron 1 in FIG. 11a), and the output of the identity transformation block (1110-4) is connected to an output neuron (1114) (corresponding to output neuron 1 in FIG. 11a). FIG. 11c illustrates the application of the final steps of the algorithm, which include replacing two copies of the input vectors (1106-2 and 1106-4) with a single vector (1116) (step 3), and reconstructing the connections of the first layer (1118) by creating two output links from all input neurons as follows: one link is connected to the subnet associated with output 1, and the other link is connected to the subnet of output 2.

[0262] Exemplary transformation of a multilayer perceptron

[0263] A multilayer perceptron (MLP) is an MLP(K, S, L1, ..., L S K inputs to the i-th layer, S layers, and L, represented as ). iAssume that it includes 2 computational neurons. Assume this is the weight matrix of the i-th layer. The following exemplary algorithm is a neuron TN (N depending on some implementations). I , N O Construct a T-neural network from ).

[0264] Algorithm MLP2TNNX

[0265] 1. For all layers i=1, ..., S,

[0266] a. Algorithm Layer2TNNX L i-1 inputs, L i output neurons and weight matrix (U i SLP composed of ) i (L i-1 , L i Apply to ). Consequently, PTNNX i It is composed of.

[0267] 2. All PTNNX in a single neural network i MTNNX is constructed by stacking them;

[0268] a. TNNX i-1 The output of is TNNX i It is set as the input of.

[0269] According to some implementations, all pair SLPs i (L i-1 , L i ) and PTNNX i Since the outputs are identical, the output of MTNNX is the MLP(K, S, L1, ..., L for the same input vector. S It is the same as the output of ).

[0270] Exemplary transformation of a recurrent neural network

[0271] A Recurrent Neural Network (RNN) includes backward connections that store information. FIG. 12 illustrates an exemplary RNN (1200) according to some implementations. This example has an input X tA block (1204) that performs activation function A is illustrated, which accepts (1206), performs activation function A, and outputs the value ht (1202). According to some implementations, a backward arrow from block (1204) to itself indicates a backward connection. An equivalent network up to the point where the activation block receives input Xt (1206) is illustrated on the right. At time 0, the network receives input X t (1208) is accepted, activation function A (1204) is performed, and value h0 (1210) is output; at time 1, the network accepts input X1 (1212) and the output of the network at time 0, performs activation function A (1204), and outputs value h1 (1214); at time 2, the network accepts input X2 (1216) and the output of the network at time 1, performs activation function A (1204), and outputs value h1 (1218); this process, according to some implementations, is such that the network accepts input X t (1206) and the output of the network at time t-1 are accepted, activation function A (1204) is performed, and the value h t It continues until time t, outputting (1202).

[0272] Data processing in RNNs is performed using the following formula:

[0273]

[0274] In the above equation, x t is the current input vector, and h t-1 is the previous input vector x t-1 This is the output of the RNN for . This expression consists of several operations as follows: linear combination calculations for two fully connected layers and , element-wise addition and non-linear function calculation (f). The first and third operations can be implemented in a trapezoid-based network (one fully connected layer is implemented in a pyramid-based network, which is a special case of a trapezoid network). The second operation is a common operation that can be implemented in networks of all structures.

[0275] In some implementations, layers of RNNs without recurrent connections are transformed through the Layer2TNNX algorithm described above. After the transformation is complete, recurrent links are added between the relevant neurons. Some implementations use the delay blocks described below with reference to Fig. 13b.

[0276] Exemplary transformation of an LSTM network

[0277] Long Short-Term Memory (LSTM) neural networks are a special case of RNNs. The operations of an LSTM network are expressed by the following equations:

[0278] ;

[0279] ;

[0280] ;

[0281] ;

[0282] ; and

[0283]

[0284] In the above equations, W f , W i , W D and W o are trainable weight matrices, and b f , b i , b D and b o are trainable biases, and x tis the current input vector, and h t-1 is the previous input vector(x t-1 It is the internal state of the LSTM calculated for ), and o t is the output for the current input vector. In the equations, the subscript t represents the time instance t, and the subscript t-1 represents the time instance t-1.

[0285] FIG. 13a is a block diagram of an LSTM neuron (1300) according to some implementations. A sigmoid (σ) block (1318) receives inputs h t-1 (1330) and x t Process (1332) and output f t (1336) is generated. The second sigmoid (σ) block (1320) is input h t-1 (1330) and x t Process (1332) and output i t Generates (1338). The hyperbolic tangent (tanh) block (1322) is input h t-1 (1330) and x t Process (1332) and output D t (1340) is generated. The third sigmoid (σ) block (1328) is inputs h t-1 (1330) and x t Process (1332) and output O t (1342) is generated. The multiplier block (1304) is f t (1336) and, output i t (1338) and D t Output C is generated by generating outputs that are sequentially summed by the summing block (1306) together with the output of the second multiplier block (1314) that multiplies (1340). t (1310) of the summation block (1306) that generates (previous time instance C t-1 Process the output (from 1302). Output C t (1310) is the output h t Output O to generate (1334) tIt is input to another tanh block (1312) that generates the output of multiplying (1342) and the third multiplier block (1316).

[0286] There are several types of operations used in these expressions, such as: (i) calculation of linear combinations of multiple fully connected layers, (ii) element-wise addition, (iii) Hadamard product, and (iv) calculation of non-linear functions (e.g., sigmoid (σ) and hyperbolic tangent (tanh)). Some implementations implement operations (i) and (iv) using trapezoid-based networks (where one fully connected layer is implemented by a pyramid-based network, which is a special case of a trapezoid network). Some implementations use networks of various structures for the common operations (ii) and (iii).

[0287] According to some implementations, layers of LSTM layers without recurring links are transformed using the Layer2TNNX algorithm described above. According to some implementations, after the transformation is complete, recurring links are added between the relevant neurons.

[0288] FIG. 13b illustrates delay blocks according to some implementations. As described above, some expressions of the equations for the LSTM operation depend on storing, restoring, and / or retrieving the output of a previous time instance. For example, the multiplier block (1304) processes the output of the summation block (1306) (from the previous time instance Ct-1 (1302)). FIG. 13b illustrates two examples of delay blocks according to some implementations. Example (1350) is input x at time t t Accept (1352) and output x t-dt It includes a delay block (1354) on the left that outputs the input after a delay of dt indicated by (1356). The example on the right (1360) is, according to some implementations, an output x t-2dtInput x after a time delay of 2 units, denoted by (1368) t Illustrates cascade (or multiple) delay blocks (1364 and 1366) that output (1362).

[0289] FIG. 13c is a neuron regime for an LSTM neuron according to some implementations. The regime includes weighted summer nodes (also called adder blocks) (1372, 1374, 1376, 1378, and 1396), multiplier blocks (1384, 1392, and 1394), and delay blocks (1380 and 1382). Input x t (1332) is connected to adder blocks (1372, 1374, 1376, and 1378). Previous input x t-1 Output h for t-1 (1330) is also input to the adder blocks (1372, 1374, 1376, and 1378). The adder block (1372) outputs f t It generates an output that is input to a sigmoid block (1394-2) that generates (1336). Likewise, the adder block (1374) generates an output that is input to a sigmoid block (1386) that generates an output (1338). Likewise, the adder block (1376) generates output D t It generates an output that is input to the hyperbolic tangent block (1388) that generates (1340). Likewise, the adder block (1378) produces an output O t It generates an output that is input to a sigmoid block (1390) that generates (1342). The multiplier block (1392) generates outputs i t (1338), f t (1336) and previous time instance C t-1 A first output is generated using the output of the adder block (1396) from (1302). The multiplier block (1394) output i t (1338) and D t(1340) is used to generate a second output. The adder block (1396) sums the first output and the second output to produce output C t Generates (1310). Output C t (1310) is the output O of the sigmoid block (1390) for the multiplier block (1384) to generate the output h (1334). t It is input to a hyperbolic tangent block (1398) that generates an output input together with (1342). The delay block (1382) is used to retrieve (e.g., save and restore) the output of the adder block (1396) from a previous time instance. Likewise, the delay block (1380) is used to retrieve the previous input x (e.g., from a previous time instance). t-1 It is used to call, store, and restore the output of the multiplier block (1384) for. Examples of delay blocks are described above with reference to FIG. 13b depending on some implementations.

[0290] Exemplary transformation of the GRU network

[0291] Gated Recurrent Unit (GRU) neural networks are a special case of RNNs. The operations of an RNN are expressed by the following equations:

[0292] ;

[0293] ;

[0294] ;

[0295] .

[0296] In the above equation, x t is the current input vector, and h t-1 is the previous input vector x t-1 This is the output calculated for.

[0297] FIG. 14a is a block diagram of a GRU neuron according to some implementations. The sigmoid (σ) block (1418) is input h t-1 (1402) and x t Process (1422) to output r t (1426) is generated. The second sigmoid (σ) block (1420) is input ht-1 (1402) and x t Process (1422) and output z t (1428) is generated. The multiplier block (1412) outputs r t (1426) and input h t-1 Multiply by (1402) (input x t (1422) together) generates an output input to the hyperbolic tangent (tanh) block (1424) to produce output j t (1430) is generated. The second multiplier block (1414) outputs j t (1430) and output z t Multiply by (1428) to generate the first output. Block (1410) is 1 - output h t Input h to generate a product that is input to the adder block (1406) along with the first output (from the multiplier block (1414)) to generate (1408). t-1 output z to generate an output that is input to a third multiplier block (1404) that multiplies (1402) and the output t Calculate (1428). Input h t-1 (1402) is the output of the GRU neuron from the previous time interval output t-1.

[0298] FIG. 14b is a neuron scheme for a GUR neuron (1440) according to some implementations. The scheme includes weighted summer nodes (also called adder blocks) (1404, 1406, 1410, 1406, and 1434), multiplier blocks (1404, 1412, and 1414), and a delay block (1432). Input x t (1422) is connected to adder blocks (1404, 1410, and 1406). Previous input xt-1 Output h for t-1 (1402) is also input to the adder blocks (1404 and 1406) and the multiplier blocks (1404 and 1412). The adder block (1404) outputs Z t It generates an output that is input to the sigmoid block (1418) that generates (1428). Likewise, the adder block (1406) generates an output r that is input to the multiplier block (1412). t It generates an output that is input to a sigmoid block (1420) that generates (1426). The output of the multiplier block (1412) is input to an adder block (1410) of the output that is input to a hyperbolic tangent block (1424) that generates an output (1430). In addition to the output (1430), the output of the sigmoid block (1418) is input to a multiplier block (1414). The output of the sigmoid block (1418) is input to a multiplier block (1404) that multiplies the input from the delay block (1432) and the output to generate a first output. The multiplier block generates a second output. The adder block (1434) sums the first output and the second output to produce an output h t (1408) is generated. The delay block (1432) is used to retrieve (e.g., save and restore) the output of the adder block (1434) from the previous time instance. Examples of delay blocks are described above with reference to FIG. 13b depending on some implementations.

[0299] Since the types of operations used in GRU are the same as those of LSTM networks (described above), GRU is converted into a trapezoid-based network according to the principles described above for LSTMs, depending on some implementations (e.g., using the Layer2TNNX algorithm).

[0300] Exemplary transformation of a convolutional neural network

[0301] Generally, Convolutional Neural Networks (CNNs) involve several basic operations, such as convolution (a linear combination set of kernel and image (or internal map) fragments), activation functions, and pooling (e.g., maximization, averaging, etc.). All computational neurons in a CNN follow a linear combination of some input and subsequent computation of an activation function, which is the general processing method for neurons in an MLP. Therefore, depending on the implementation, CNNs are converted into multilayer perceptrons using the MLP2TNNX algorithm described above.

[0302] Conv1D is a convolution performed over time coordinates. FIGS. 15a and 15b show neuron regimes for variations of a single Conv1D filter according to some implementations. In FIG. 15a, the weighted summing node (1502) (referred to as an adder block and marked with '+') has 5 inputs to correspond to a 1D convolution with a kernel of 5. The inputs are x from time t t (1504) (obtained by inputting the delay block (1506)) and x from time t-2 t-2 (1516) (obtained by inputting the output of delay block (1506) to another delay block (1508)), and x from time t-3 t-3 (1516) (obtained by inputting the output of delay block (1508) to another delay block (1510)), and x from time t-4 t-4 (1516) (obtained by inputting the output of delay block (1510) to another delay block (1512). For large kernels, it is sometimes advantageous to utilize different frequency delay blocks so that some blocks generate larger delays. Some implementations replace multiple small delay blocks with one large delay block, as shown in FIG. 15b. In addition to the delay blocks of FIG. 15a, an example is x from time t-3. t-3 Delay_3 block (1524) generating (1518), and x from time t-5 t-5Another delay block (1526) is used to generate (1522). This operation does not reduce the total number of blocks, but depending on some implementations, it can reduce the total number of resulting operations performed on the input signal and reduce error accumulation.

[0303] In some implementations, convolutional layers are represented by trapezoidal neurons, and fully connected layers are represented by crossbars of resistors. Some implementations use crossbars and calculate the resistance matrix for the crossbars.

[0304] An exemplary approximation algorithm for a single-layer perceptron with multiple outputs

[0305] In some implementations, the exemplary transformations described herein are performed by a neural network transformation module (226) that transforms trained neural networks (220) and / or analog neural network optimization modules (246) based on mathematical formulas (230), basic function blocks (232), analog component models (234), and / or analog design constraints (236) to obtain transformed neural networks (228).

[0306] A single-layer perceptron SLP(K, L) is assumed to have K inputs and L output neurons, and each output neuron performs an activation function (F). U ∈ R LΥK It is additionally assumed that is the weight matrix for SLP(K, L). The following is the TN(N neurons) using the approximation algorithm Layer2TNNX_Approx, depending on some implementations. I , N OThis is an example of constructing a T-neural network from ). The algorithm applies the Layer2TNN1 algorithm (described above) in the first step to reduce the number of neurons and connections, and then applies Layer2TNNX to process the reduced-size inputs. The outputs of the resulting neural network are calculated using the shared weights of the layers constructed by the Layer2TNN1 algorithm. The number of these layers is determined by the value p, which is a parameter of the algorithm. If p is 0, only the Layer2TNNX algorithm is applied and the transformation is identical. If p > 0, p layers have shared weights and the transformation is an approximation.

[0307] Algorithm Layer2TNNX_Approx

[0308] 1. Set {0, 1, ..., Set the parameter (p) that has a value from 1}.

[0309] 2. If p > 0, neuron TN(N I Algorithm with , 1) Layer2TNN1 Apply and configure the first p layers of the resulting subnet (PNN). The net PNN at the output layer It has a number of neurons.

[0310] 3. Neuron TN(N I , N O Apply the Layer2TNNX algorithm having ) and N p Construct a neural subnet TNN with n inputs and L outputs.

[0311] 4. Set the weights of the PNN network. The weights of all neurons (i) in the first layer of the PNN are based on the rule As, here, C is an arbitrary constant not equal to 0, and In, the above rule, and k i For all weights (j) of these neurons excluding, It is set according to the rule. All other weights of the PNN are set to 1. is a neuron (i) and a neuron (k) in the first layer. i It represents the weight for the first layer for the connection between ) (indicated by the superscript (1)).

[0312] 5. Set the weights of the TNN subnet. The weights of all neurons (i) in the first layer of the TNN (which is the (p+1)th layer when considering the entire network) are given by the equation It is set according to. All other weights of the TNN are set to 1.

[0313] 6. Set the activation function for all neurons in the last layer of the TNN subnet to F. The activation functions of all other neurons are identifiable.

[0314] FIG. 16 illustrates an exemplary architecture (1600) of a resulting neural network according to some implementations. This example includes a PNN (1602) connected to a TNN (1606). The PNN (1602) includes layers for K inputs and N p It generates L outputs, which are connected to the TNN (1606) as inputs (1612). The TNN (1606) generates L outputs (1610), depending on some implementations.

[0315] Approximation algorithm for a multilayer perceptron with multiple outputs

[0316] A multilayer perceptron (MLP) is an MLP(K, S, L1, ... L S K inputs to the i-th layer, S layers, and L, represented as ). i Assume that it includes 2 computational neurons. It is additionally assumed that is the weight matrix for the i-th layer. The following exemplary algorithm, depending on some implementations, describes a neuron TN(N I , N O Construct a T-neural network from ).

[0317] Algorithm MLP2TNNX_Approx

[0318] 1. For all layers i=1, ..., S,

[0319] a. Algorithm Layer2TNNX_Approx (described herein) L i-1 inputs, L i output neurons and weight matrix (U i SLP composed of ) i (L i-1 , L i Apply to ). If i=1, L0=K. At this stage, as a result, PTNNX i Assume that it constitutes.

[0320] 2. All PTNNX i It constructs an MTNNX (Multilayer Perceptron) by stacking them into a single neural network, where TNNX i-1 The output of is TNNX i It is set as input for.

[0321] Exemplary compression method for transformed neural networks

[0322] In some implementations, the exemplary transformations described herein are performed by a neural network transformation module (226) that transforms trained neural networks (220) and / or analog neural network optimization modules (246) based on mathematical formulas (230), basic function blocks (232), analog component models (234), and / or analog design constraints (236) to obtain transformed neural networks (228).

[0323] This section describes exemplary methods for compressing transformed neural networks according to certain implementations. Some implementations compress analog pyramidal neural networks to minimize the number of operational amplifiers and resistors required to implement the analog network on a chip. In some implementations, the compression method for analog neural networks is pruning similar to that used for software neural networks. Nevertheless, there are several characteristics to the compression of pyramidal analog networks that are feasible in hardware as IC analog chips. Since the number of elements such as operational amplifiers and resistors defines the weights in analog-based neural networks, it is important to minimize the number of operational amplifiers and resistors placed on the chip. This will also help minimize the chip's power consumption. Modern neural networks, such as convolutional neural networks, can be compressed 5 to 200 times without significant loss of network accuracy. Often, entire blocks of modern neural networks can be pruned without significant loss of accuracy. The conversion of dense neural networks into sparsely connected pyramidal, trapezoidal, or crossbar neural networks provides an opportunity to prune sparsely connected pyramidal or trapezoidal analog networks, which is represented by operational amplifiers and resistors on analog IC chips. In some implementations, these techniques are applied in addition to existing neural network compression techniques. In some implementations, compression techniques are applied based on the specific architecture of the input neural network and / or the converted neural network (e.g., pyramidal versus trapezoidal versus crossbar).

[0324] For example, since networks are implemented through analog elements such as operational amplifiers, some implementations determine whether knots (operational amplifiers) are needed for the entire chip by determining the current flowing through the operational amplifiers when a standard training dataset is provided. Some implementations analyze the chip's SPICE model and determine knots and connections where no current flows and no power is consumed. Some implementations determine the current flow through the analog IC network, thereby determining the knots and connections, and then pruning them. Furthermore, some implementations remove connections if the connection weight is too high, and / or replace resistors with direct connectors if the connection weight is too low. In some implementations, if all connections leading to this knot have weights lower than a predetermined threshold (e.g., close to zero), the knot is pruned; connections where the operational amplifier always provides zero at the output are removed; and / or the operational amplifier is converted into a linear junction if the amplifier provides a linear function without amplification.

[0325] Some implementations apply specific compression techniques to pyramidal, trapezoidal, or crossbar-type neural networks. Some implementations minimize the number of pyramidal or trapezoidal layers by generating pyramidal or trapezoidal networks with a larger amount of input (than when not compressing). Some implementations create more compact trapezoidal networks by maximizing the number of outputs from each neuron.

[0326] Exemplary creation of an optimal resistor set

[0327] In some implementations, the exemplary calculations described herein are performed by a weight matrix calculation or weight quantization module (238) (e.g., using a resistance calculation module (240)) that calculates weights (272) for connections of the transformed neural network and / or corresponding resistance values ​​(242) for the weights (272).

[0328] This section describes an example of generating an optimal set of resistors for a trained neural network according to certain implementations. An exemplary method is provided for converting connection weights to resistor nominals to implement a neural network (also known as an NN model) on a microchip where resistor nominals may be smaller and allowed resistor variation may be higher.

[0329] The test set 'Test' contains approximately 10,000 input vector values ​​(x and y coordinates), and it is assumed that both coordinates vary in increments of 0.01 within the range [0;1]. It is assumed that the network NN output for a given input X is given as Out = NN(X). Additionally, it is assumed that the input value class is determined as follows: Class_nn(X) = NN(X) > 0.61 ? 1:0

[0330] The following compares the mathematical network model (M) with the schematic network model (S). The schematic network model includes the possible resistor variance of rv, processes the 'Test' set, and generates a different vector of output values ​​S(Test) = Out_s each time. The output error is defined by the following equation:

[0331]

[0332] Classification error is defined by the following equation:

[0333]

[0334] In some implementations, the desired classification error is set to 1% or less.

[0335] Exemplary error analysis

[0336] FIG. 17a illustrates an exemplary chart (1700) illustrating the dependency between the output error and the classification error on an M network according to some implementations. In FIG. 17a, the x-axis corresponds to the classification margin (1704), and the y-axis corresponds to the total error (1702) (see description above). The graph shows the total error (the difference between the output of the model (M) and the actual data) for different classification margins of the output signal. In this example, according to the chart, the optimal classification margin (1706) is 0.610.

[0337] Assuming that another network (O) generates output values ​​with a constant shift versus related M output values, there will be a classification error between O and M. To keep the classification error less than 1%, this shift must be in the range [-0.045, 0.040]. Therefore, the possible output error for S is 45 mV.

[0338] The possible weight error is determined by analyzing the dependency between the relative weight / bias error and the output error for the entire network. The charts (1710 and 1720) shown in FIGS. 17b and 17c, respectively, are obtained by averaging 20 randomly modified networks through a 'test' set according to some implementations. In these charts, the x-axis represents the absolute weight error (1712) and the y-axis represents the absolute output error (1714). As can be seen in the chart, the output error limit of 45 mV (y = 0.045) allows for a relative error or an absolute error value (x value) of 0.01 for each weight. The maximum weight coefficient for the neural network (the maximum absolute value of the weights among all weights) is 1.94.

[0339] Exemplary process for selecting a resistor set

[0340] Resistors configured with pairs {R+, R-} selected from this set have a value function for the required weighting range [-wlim; wlim] with a certain amount of resistor error (r_err). In some implementations, the value function of the resistor set is calculated as follows:

[0341] The array of possible weight options is calculated along with the weighted average error according to the resistor error;

[0342] The array's weight options are restricted to the required weight range [-wlim; wlim];

[0343] Values ​​that are worse than neighboring values ​​in terms of weighted error are removed;

[0344] An array of distances between adjacent values ​​is calculated;

[0345] The value function is a combination of the squared mean or maximum value of the distance array.

[0346] Some implementations iteratively search for the optimal set of resistors by successively adjusting each resistor value of the set at a learning rate value. In some implementations, the learning rate changes over time. In some implementations, the initial set of resistors is selected as uniform (e.g., [1;1;...;1]) such that the selected minimum and maximum resistor values ​​are within a two-digit range (e.g., [1;100] or [0.1;10]). Some implementations select R+ = R-. In some implementations, the iterative process converges to a local minimum. In one case, the process generated the following set, [0.17, 1.036, 0.238, 0.21, 0.362, 1.473, 0.858, 0.69, 5.138, 1.215, 2.083, 0.275]. This is a local optimal resistor set consisting of 12 resistors for a weighted range [-2; 2] with rmin = 0.1 (minimum resistance), rmax = 10 (maximum resistance), and r_err = 0.001 (resistance estimation error). In some implementations, the entire available range [rmin; rmax] is not used to find a good local optimal value. Only a portion of the available range (e.g., in this case [0.17; 5.13]) is used. The resistor set values ​​are relative rather than absolute. In this case, a relative value range of 30 is sufficient for the resistor set.

[0347] In one example, the following set of resistors of length 20 for the parameters mentioned above, [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] is obtained. In this example, the value 1.763 is also an R- = R+ value. This set is then used to generate weights for the NN to create the corresponding model (S). When the relative resistor error is close to zero, the mean squared output error of model S is 11 mV, which is more than what is required for a set of 20 resistors. The maximum error for the input data set was calculated to be 33 mV. In one example, S, DAC, and ADC converters with 256 levels were analyzed as separate models, and the results showed a mean squared output error of 14 mV and a maximum output error of 49 mV. In NN, an output error of 45 mV corresponds to a relative recognition error of 1%. The 45 mV output error value also corresponds to a relative error or an absolute weighted error of 0.01, which is acceptable. The maximum weighting factor of NN is 1.94. In this way, the optimal (or near-optimal) set of resistors is determined using an iterative process based on the desired weighting range [-wlim; wlim], the resistor error (relative), and the possible resistor range.

[0348] In general, a very wide set of resistors is not very useful unless different precisions (e.g., 1 to 1 / 5 digits are sufficient) are required within different layers or parts of the weight spectrum. For example, assuming the weights are in the [0, 1] range but most weights are in the [0, 0.001] range, better precision is needed within that range. In the example described above, if the relative resistance error is close to zero, a set of 20 resistors is sufficient to quantize the NN network with a given precision. In one example, in a set of registers [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] (reference values ​​are relative), an average S output error of 11 mV was obtained.

[0349] An exemplary process for the quantization of resistor values

[0350] In some implementations, the exemplary calculations described herein are performed by a weight matrix calculation or weight quantization module (238) (e.g., using a resistance calculation module (240)) that calculates weights (272) for connections of the transformed neural network and / or corresponding resistance values ​​(242) for the weights (272).

[0351] This section describes an exemplary process for quantizing resistance values ​​corresponding to the weights of a trained neural network, depending on the implementation. The exemplary process significantly simplifies the process of manufacturing chips using analog hardware components to implement the neural network. As described above, in some implementations, resistors are used to represent neural network weights and / or biases for operational amplifiers representing analog neurons. The exemplary process described herein reduces complexity, particularly when manufacturing sets of resistors for a chip using lithography. Through the procedure of quantizing the resistor values, only the resistance values ​​required for chip manufacturing need to be selected. In this way, the exemplary process simplifies the entire chip manufacturing process and enables the automatic manufacturing of resistor lithographic masks upon request.

[0352] FIG. 18 provides an exemplary scheme of a neuron model (1800) used for resistor quantization according to some implementations. In some implementations, the circuit is based on an operational amplifier (1824) (e.g., AD824 series precision amplifier) ​​that receives input signals from negative weighting fixed resistors (R1-(1804), R2-(1806), Rb-bias (1816), Rn-(1818) and R-(1812)) and positive weighting fixed resistors (R1+(1808), R2+(1810), Rb+bias (1820), Rn+(1822) and R+(1814)). Positive weighting voltages are supplied to the direct input of the operational amplifier (1824), and negative weighting voltages are supplied to the inverse input of the operational amplifier (1824). The operational amplifier (1824) is used to allow a weighted sum operation of the weighted outputs from each resistor, where negative weights are subtracted from the positive weights. The operational amplifier (1824) also amplifies the signal to the extent required for circuit operation. In some implementations, the operational amplifier (1824) also performs a RELU conversion of the output signal in the output cascade.

[0353] The following equations determine weights based on resistance values.

[0354] The voltage at the output of a neuron is determined by the following equation:

[0355]

[0356] The weights of each connection are determined by the following equation:

[0357]

[0358] The following exemplary optimization procedure quantizes the values ​​of each resistor according to some implementations and minimizes the error in the neural network output.

[0359] 1. Obtain a set of connection weights and bises {w1, ..., wn, b}.

[0360] 2. Obtain the possible minimum and maximum resistance values ​​{Rmin, Rmax}. These parameters are determined based on the technology used in manufacturing. In some implementations, TaN or tellurium high resistivity materials are used. In some implementations, the minimum value of the resistor is determined by the least squares that can be formed by lithography. The maximum value is determined by the length that can be allowed for the resistors (e.g., resistors made of TaN or tellurium) to fit into the desired area, which in turn is determined by the operational amplifier rectangular area of ​​the lithographic mask. In some implementations, the area of ​​the resistor arrays is smaller than the area of ​​a single operational amplifier because the resistor arrays are stacked (e.g., one in BEOL and the other in FEOL).

[0361] 3. Assume that each resistor has a relative tolerance value of r_err.

[0362] 4. The goal is to select a set of resistor values ​​{R1, ..., Rn} of a given length N within the defined [Rmin; Rmax] based on the values ​​{w1, ..., wn, b}. An exemplary search algorithm for finding sub-options {R1, ..., Rn} set based on specific optimality criteria is provided below.

[0363] 5. Another algorithm selects {Rn, Rp, Rni, Rpi} for the network once {R1..Rn} is determined.

[0364] Exemplary {R1, ..., Rn} search algorithm

[0365] Some implementations use an iterative approach to search for a set of resistors. Some implementations select an initial (random or uniform) set {R1, … , Rn} within a defined range. Some implementations select one of the resistor elements set to R- = R+. Some implementations change each resistor in the set by the current learning rate value until these changes generate a 'better' set (according to a value function). This process is repeated for all resistors in the set with various different learning rate values ​​until further improvement is impossible.

[0366] Some implementations define the value function of a resistor set as follows:

[0367] The possible weight options are calculated according to the formula (explained above):

[0368]

[0369] The expected error value for each weighting option is estimated based on the potential resistance relative error (r_err) determined by the IC manufacturing technology.

[0370] The list of weight options is limited or restricted to the range [-wlim; wlim]

[0371] Some values ​​expected to have errors exceeding a high threshold (e.g., 10 times r_err) are removed.

[0372] The value function is calculated as the mean squared distance between two adjacent weight options. Therefore, the value function is minimized when the weight options are uniformly distributed within the range [-wlim; wlim].

[0373] The required weight range [-wlim; wlim] for the model is set to [-5; 5], and other parameters include N = 20, r_err = 0.1%, rmin = 100 KΩ, rmax = 5 MΩ. Here, rmin and rmax are the minimum and maximum values ​​for the resistors, respectively.

[0374] In one example, the following set of resistors of length 20 for the parameters mentioned above, [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] MΩ was obtained. R- = R+ = 1.763 MΩ.

[0375] Exemplary {Rn, Rp, Rni, Rpi} search algorithm

[0376] Some implementations determine Rn and Rp using iterative algorithms similar to the one described above. Some implementations set Rp = Rn (the tasks of determining Rn and Rp are symmetric—the two quantities generally converge to similar values). Then, for each weight (wi), some implementations select a resistance pair {Rni, Rpi} that minimizes the estimated weight error value:

[0377]

[0378] Some implementations construct neural network schematics using the subsequently set {Rni; Rpi; Rn; Rp} values. In one example, the schematics, according to some implementations, generated a mean squared output error of 11 mV (also known as the S mean squared output error described above) and a maximum error of 33 mV for a set of 10,000 uniformly distributed input data samples. In one example, the S model was analyzed as a separate model with 256 levels, along with digital-to-analog converters (DACs) and analog-to-digital converters (ADCs). According to some implementations, this model generated a mean squared output error of 14 mV and a maximum output error of 49 mV on the same dataset. DACs and ADCs have levels because they convert analog values ​​into bit values ​​or vice versa. An 8-bit digital value is equivalent to 256 levels. For an 8-bit ADC, precision cannot be better than 1 / 256.

[0379] In some implementations, Mathcad or other similar software is used to calculate the resistance values ​​of analog IC chips when the weights of the connections are known, based on Kirchhoff's electrical circuit laws and the basic principles of operational amplifiers (described below with reference to Fig. 19a). In some implementations, operational amplifiers are used for both signal amplification and transformation according to activation functions (e.g., ReLU, sigmoid, tangent hyperbolic, or linear mathematical equations).

[0380] Some implementations fabricate resistors on a lithographic layer where the resistors are formed as cylindrical holes in a SiO2 matrix, and the resistance value is determined by the diameter of the hole. In some implementations, amorphous TaN, TiN, or tellurium are used as high-resistance materials to create high-density resistor arrays. Certain ratios of Ta to N, Ti to N, and Cr to N provide high resistance for creating ultra-high-density, high-resistivity element arrays. For example, in the case of TaN, Ta5N6, and Ta3N5, the higher the ratio of N to Ta, the higher the resistivity. Some implementations use Ti2N, TiN, CrN, or Cr5N, and determine the ratio accordingly. TaN deposition is a standard procedure used in chip manufacturing and can be used by all major foundries.

[0381] Exemplary operational amplifier

[0382] FIG. 19a illustrates a schematic diagram of an operational amplifier made of CMOS (COMS Op-amp) (1900) according to some implementations. In FIG. 19a, In+ (positive input or pos) (1404), In- (negative input or neg) (1406), and Vdd- (positive supply voltage to GND) (1402) are contact inputs. Contact Vss- (negative supply voltage or GND) is labeled (1408). The circuit output is Out (1410) (contact output). The parameters of the CMOS transistors are determined by the ratio of the following dimensions: L (gate channel length) to W (gate channel width), as shown in the table illustrated in FIG. 19b (described below), which is an example. A current mirror is formed on NMOS transistors M11 (1944), M12 (1946) and resistor R1 (1921) (having an exemplary resistance value of 12 kΩ) and provides an offset current for the differential pair (M1 (1926) and M3 (1930)). A differential amplifier stage (differential pair) is formed on NMOS transistors M1 (1926) and M3 (1930). Transistors M1 and M3 are amplified, and PMOS transistors M2 (1928) and M4 (1932) act as active current loads. From transistor M3, a signal is input to the gate of output PMOS transistor M7 (1936). From transistor M1, a signal is input to the active loads of PMOS transistor M5 (inverter) (1934) and NMOS transistor M6 (1934). The current flowing through transistor M5 (1934) is set for NMOS transistor M8 (1938). Transistor M7 (1936) is included in a setup having a common source for a positive half-wave signal. Transistor M8 (1938) is enabled by a common source circuit for a negative half-wave signal.To increase the total load capacity of the operational amplifier, the outputs of M7 (1936) and M8 (1938) include an inverter of transistors M9 (1940) and M10 (1942). Capacitors C1 (1912) and C2 (1914) are blocked.

[0383] FIG. 19b illustrates a descriptive table (1948) for the exemplary circuit illustrated in FIG. 19a according to some implementations. Values ​​for the parameters are provided as examples, and various other configurations are possible. The transistors (M1, M3, M6, M8, M10, M11, and M12) are N-channel MOSFET transistors with explicit substrate connections. The other transistors (M2, M4, M5, M7, and M9) are P-channel MOSFET transistors with explicit substrate connections. The table shows exemplary shutter ratios of length (L, column 1) and width (W, column 2) provided for each transistor (column 3).

[0384] In some implementations, operational amplifiers such as the example described above are used as the basic elements of integrated circuits for hardware implementations of neural networks. In some implementations, the size of the operational amplifiers is 40 square microns and they are fabricated according to the 45 nm node standard.

[0385] In some implementations, activation functions such as the ReLU, hyperbolic tangent, and sigmoid functions are represented as operational amplifiers with modified output cascades. For example, the RELU, sigmoid, or tangent functions are implemented as output cascades of operational amplifiers (also called OpAmps) using corresponding well-known analog schematics depending on the implementation.

[0386] In the examples described above and below, in some implementations, operational amplifiers are replaced with inverters, current mirrors, 2-quadrant or 4-quadrant multipliers and / or other analog function blocks that allow weighted sum operations.

[0387] Exemplary scheme of an LSTM block

[0388] FIGS. 20a through 20e illustrate schematic diagrams of an LSTM neuron (20000) according to some implementations. The inputs to the neuron are Vin1 (20002) and Vin2 (20004), which are values ​​in the range [-0.1, 0.1]. The LSTM neuron also inputs the result of computing the neuron at time H(t-1) (previous value; see above description for LST neuron) (20006) and the neuron's state vector at time C(t-1) (previous value). The outputs of the neuron LSTM (see FIG. 20b) include the result of computing the neuron at the current time H(t) (20118) and the neuron's state vector at the current time C(t) (20120). The system includes the following:

[0389] "Neuron O" assembled in operational amplifiers U1 (20094) and U2 (20100) as shown in FIG. 20a. Resistors R_Wo1 (20018), R_Wo2 (20016), R_Wo3 (20012), R_Wo4 (20010), R_Uop1 (20014), R_Uom1 (20020), Rr (20068) and Rf2 (20066) set the weights of the connections of a single "Neuron O". "Neuron O" uses a sigmoid (module X1 (20078), FIG. 20b) as a non-linear function;

[0390] "Neuron C" assembled in operational amplifiers U3 (20098) (shown in FIG. 20c) and U4 (20100) (shown in FIG. 20a). Resistors R_Wc1 (20030), R_Wc2 (20028), R_Wc3 (20024), R_Wc4 (20022), R_Ucp1 (20026), R_Ucm1 (20032), Rr (20122), and Rf2 (20120) set the weights of the connections of "Neuron C". "Neuron C" uses the hyperboline tangent (module X2 (22080), FIG. 2b) as a nonlinear function;

[0391] "Neuron I" assembled in operational amplifiers U5 (20102) and U6 (20104) as shown in FIG. 20c. Resistors R_Wi1 (20042), R_Wi2 (20040), R_Wi3 (20036), and R_Wi4 (20034), R_Uip1 (20038), R_Uim1 (20044), Rr (20124), and Rf2 (20126) set the weights of the combinations of "Neuron I". "Neuron I" uses a sigmoid (module X3 (20082)) as a non-linear function;

[0392] "Neuron f" assembled in operational amplifiers U7 (20106) and U8 (20108) as shown in FIG. 20d. Resistors R_Wf1 (20054), R_Wf2 (20052), R_Wf3 (20048), R_Wf4 (20046), R_Ufp1 (20050), R_Ufm1 (20056), Rr (20128) and Rf2 (20130) set the weights of the combinations of "Neuron f". "Neuron f" uses a sigmoid (module X4 (20084)) as a non-linear function.

[0393] The outputs of module X2 (20080) (Fig. 20b) and X3 (20082) (Fig. 20c) are input to the X5 multiplier module (20086) (Fig. 20b). The outputs of the buffer for module X4 (20084) (Fig. 20d) and U9 (20010) are input to the multiplier module X6 (20088). The outputs of modules X5 (20086) and X6 (20088) are input to the adder (U10 (20112)). The divider (10) is assembled on resistors R1 (20070), R2 (20072), and R3 (20074). A non-linear function of the hyperbolic tangent (module X7 (20090), Fig. 20b) is obtained by releasing the divisor signal. The output C(t)(20120) (current state vector of the LSTM neuron) is obtained from the buffer-inverter of the U11 (20114) output signal. The outputs of modules X1 (20078) and X7 (20090) are input to a multiplier (module X8 (20092)), and the output is input to the buffer divider of U12 (20116) as 10. The result of calculating the LSTM neuron at the current H(t)(20118) is obtained from the output signal of U12 (20116).

[0394] FIG. 20e illustrates exemplary values ​​for various configurable parameters (e.g., voltages) for the circuits illustrated in FIG. 20a through 20d according to some implementations. According to some implementations, Vdd (20058) is set to +1.5V, Vss (20064) is set to -1.5V, Vdd1 (20060) is set to +1.8V, Vss1 (20062) is set to -1.0V, and GND (20118) is set to GND.

[0395] FIG. 20f illustrates a descriptive table (20132) for the exemplary circuits shown in FIG. 20a through 20d according to some implementations. Values ​​for parameters are provided as examples, and various other configurations are possible. The transistors (U1 through U12) are CMOS OpAmps (described above with reference to FIG. 19a and 19b). X1, X3, and X4 are modules that perform a sigmoid function. X2 and X7 are modules that perform a hyperbolic tangent function. X5 and X8 are modules that perform a multiplier function. Exemplary resistor ratings are as follows: Rw = 10 kΩ and Rr = 1.25 kΩ. Other resistor values ​​are expressed relative to Rw. For example, Rf2 = 12 × Rw, R_Wo4 = 5 × Rw, R_Wo3 = 8 × Rw, R_Uop1 = 2.6 × Rw, R_Wo2 = 12 × Rw, R_W1 = w × Rw, R_Uom1 = 2.3 × Rw, R_wc4 = 4 × Rw, R_Wc3 = 5.45 × Rw, R_Ucp1 = 3 × Rw, R_Wc2 = 12 × Rw, R_Wc1 = 2.72 × Rw, R_Ucm1 = 3.7 × Rw, R_Wi4 = 4.8 × Rw, W_Wi3 = 6 × Rw, W_Uip1 = 2 × Rw, R_Wi2 = 12 × Rw, and R_Wi1 = 3 R_Uim1 = 2.3 × Rw, R_Wf4 = 2.2 × Rw, R_Wf3 = 5 × Rw, R_Wfp = 4 × Rw, R_Wf2 = 2 × Rw, R_Wf1 = 5.7 × Rw, and Rfm1 = 4.2 × Rw.

[0396] Exemplary system of a multiplier block

[0397] FIGS. 21a through 21i illustrate schematic diagrams of a multiplier block (21000) according to some implementations. The neuron (21000) is based on the principle of a 4-quadrant multiplier assembled using operational amplifiers U1 (21040) and U2 (21042) (illustrated in FIG. 21b), U3 (21044) (illustrated in FIG. 21h), U4 (21046) and U5 (21048) (illustrated in FIG. 21i) and CMOS transistors M1 (21052) through M68 (21182). The inputs of the multiplier include V_one (21020 21006) and V_two (21008) (shown in FIG. 21b), contact Vdd (positive supply voltage, e.g., +1.5 V with respect to GND) (21004), and contact Vss (negative supply voltage, e.g., -1.5 V with respect to GND). In this scheme, the following additional supply voltages are used: contact input Vdd1 (positive supply voltage, e.g., +1.8 V with respect to GND), contact Vss1 (negative supply voltage, e.g., -1.0 V with respect to GND). The result of the circuit calculation is output at mult_out (output pin) (21170) (shown in FIG. 21i).

[0398] Referring to FIG. 21b, the input signal (V_one) of V_one (21006) is connected to an inverter with a single gain generated by U1 (21040), and the output forms a signal negA (21006) with the same amplitude but opposite sign to the signal (V_one). Similarly, the signal (V_two) of V_two (21008) is connected to an inverter with a single gain generated by U2 (21042), and the output forms a signal negB (21012) with the same amplitude but opposite sign to the signal (V_two). Pairwise combinations of signals from possible combinations (V_one, V_two, negA, negB) are output to the corresponding mixers of the CMOS transistors.

[0399] Referring again to FIG. 21a, V_two (21008) and negA (21010) are input to a multiplexer assembled with NMOS transistors (M19 (21086), M20 (21088), M21 (21090), M22 (21092)) and PMOS transistors (M23 (1094) and M24 (21096)). The output of this multiplexer is input to NMOS transistor M6 (21060) (Fig. 21d).

[0400] Similar transformations caused by signals include the following:

[0401] negB (21012) and V_one (21020) are input to a multiplexer assembled with NMOS transistors (M11 (21070), M12 (2072), M13 (2074), M14 (21076)) and PMOS transistors (M15 (2078) and M16 (21080)). The output of this multiplexer is input to the M5 (21058) NMOS transistor (shown in FIG. 21d);

[0402] V_one (21020) and negB (21012) are input to a multiplexer assembled with PMOS transistors (M18 (21084), M48 (21144), M49 (21146) and M50 (21148)) and NMOS transistors (M17 (21082), M47 (21142)). The output of this multiplexer is input to the M9 PMOS transistor (21066) (shown in FIG. 21d);

[0403] negA (21010) and V_two (21008) are input to a multiplexer assembled with PMOS transistors (M52 (21152), M54 (21156), M55 (21158) and M56 (21160)) and NMOS transistors (M51 (21150) and M53 (21154)). The output of this multiplexer is input to the M2 NMOS transistor (21054) (shown in FIG. 21c);

[0404] negB (21012) and V_one (21020) are input to a multiplexer assembled with NMOS transistors (M11 (21070), M12 (21072), M13 (21074) and M14 (21076)) and PMOS transistors (M15 (21078) and M16 (21080)). The output of this multiplexer is input to the M10 NMOS transistor (21068) (shown in FIG. 21d);

[0405] negB (21012) and V_one (21020) are input to a multiplexer assembled with NMOS transistors (M35 (21118), M36 (21120), M37 (21122) and M38 (21124)) and PMOS transistors (M39 (21126) and M40 (21128)). The output of this multiplexer is input to the M27 PMOS transistor (21102) (shown in FIG. 21h);

[0406] V_two (21008) and V_one (21020) are input to a multiplexer assembled with NMOS transistors (M41 (21130), M42 (21132), M43 (21134) and M44 (21136)) and PMOS transistors (M45 (21138) and M46 (21140)). The output of this multiplexer is input to M30 NMOS transistor (21108) (shown in FIG. 21h);

[0407] V_one (21020) and V_two (21008) are input to a multiplexer assembled with PMOS transistors (M58 (21162), M60 (21166), M61 (21168) and M62 (21170)) and NMOS transistors (M57 (21160) and M59 (21164)). The output of this multiplexer is input to M34 PMOS transistor (21116) (shown in FIG. 21h);

[0408] negA (21010) and negB (21012) are input to a multiplexer assembled with PMOS transistors (M64 (21174), M66 (21178), M67 (21180) and M68 (21182)) and NMOS transistors (M63 (21172) and M65 (21176)). The output of this multiplexer is input to PMOS transistor M33 (21114) (shown in FIG. 21h).

[0409] The current mirror (transistors M1 (21052), M2 (21053), M3 (21054), M4 (21056)) powers a part of the 4-quadrant multiplier circuit shown on the left, which is made of transistors (M5 (21058), M6 (21060), M7 (219106), M8 (21064), M9 (21066) and M10 (21068)). The current mirrors (for transistors M25 (21098), M26 (21100), M27 (21102) and M28 (21104)) are power supplies for the right side of the 4-quadrant multiplier made of transistors (M29 (21106), M30 (21108), M31 (21110), M32 (21112), M33 (21114) and M34 (21116)). The multiplication result is taken from resistor Ro (21022) enabled in parallel to transistor M3 (21054) and resistor Ro (21188) enabled in parallel to transistor M28 (21104), which is supplied to the adder of U3 (21044). The output of U3 (21044) is supplied to an adder with a gain of 7,1 assembled in U5 (21048), and the second input is compensated to a reference voltage set by resistors R1 (21024) and R2 (21026) and buffer U4 (21046) as shown in FIG. 21i. The result of the multiplication is output through the Mult_Out output (21170) of the output of U5 (21048).

[0410] FIG. 21j illustrates a descriptive table (21198) for the schematic diagrams shown in FIG. 21a through 21i according to some implementations. U1 through U5 are CMOS OpAmps. N-channel MOSFET transistors with explicit substrate connections include transistors (M1, M2, M25, and M26) (where the shutter ratio of length (L) = 2.4u and the shutter ratio of width (W) = 1.26u), transistors (M5, M6, M29, and M30 (where L = 0.36u and W = 7.2u), transistors (M7, M8, M31, and M32) (where L = 0.36u and W = 199.98u), transistors (M11 to M14, M19 to M22, M35 to M38, and M41 to M44 (where L = 0.36u and W = 0.4u)) and transistors (M17, M47, M51, M53, M57, M59, M43, and M64 (where L = 0.36u and W = 0.72u) P-channel MOSFET transistors with explicit substrate connections include transistors (M3, M4, M27, and M28) (when the shutter ratio of length (L) = 2.4u and the shutter ratio of width (W) = 1.26u), transistors (M9, M10, M33, and M34) (when L = 0.36u and W = 7.2u), transistors (M18, M48, M49, M50, M52, M54, M55, M56, M58, M60, M61, M62, M64, M66, M67, and M68) (when L = 0.36u and W = 0.8u), and transistors (M15, M16, M23, M24, M39, M40, M45, and M46) (when L = 0.36u and W = Includes (in the case of 0.72u). According to some implementations, exemplary resistor ratings include Ro = 1kΩ, Rin = 1kΩ, Rf = 1kΩ, Rc4 = 2kΩ and Rc5 = 2kΩ.

[0411] Exemplary system of sigmoid blocks

[0412] FIG. 22a illustrates a schematic diagram of a sigmoid block (2200) according to some implementations. A sigmoid function (e.g., modules X1 (20078), X3 (20082) and X4 (20084), described above with reference to FIG. 20a through 20f) is implemented using operational amplifiers (U1 (2250), U2 (2252), U3 (2254), U4 (2256), U5 (2258), U6 (2260), U7 (2262) and U8 (2264)) and NMOS transistors (M1 (2266), M2 (2268) and M3 (2270)). Contact sigm_in (2206) is a module input, contact input Vdd1 (2222) is a positive supply voltage of +1.8V relative to GND (2208), and contact Vss1 (2204) is a negative supply voltage of -1.0V relative to GND. In this configuration, U4 (2256) has a reference voltage source of -0.2332V, and the voltage is set by dividers R10 (2230) and R11 (2232). U5 (2258) has a reference voltage source of 0.4V, and the voltage is set by dividers R12 (2234) and R13 (2236). U6 (2260) is a reference voltage source of 0.32687V, and the voltage is set by dividers R14 (2238) and R15 (2240). U7 (2262) has a reference voltage source of -0.5V, and the voltage is set by dividers R16 (2242) and R17 (2244). U8 (2264) has a reference voltage source of -0.33V, and the voltage is set by dividers R18 (2246) and R19 (2248).

[0413] The sigmoid function is formed by adding reference voltages corresponding to the differential module assembled in transistors M1 (2266) and M2 (2268). The current mirror for the differential stage is assembled with an active modulation operational amplifier U3 (2254) and an NMOS transistor M3 (2270). The signal of the differential stage is removed by the NMOS transistor M2, and resistor R5 (2220) is input to the adder U2 (2252). The output signal sigm_out (2210) is removed from the output of the U2 adder (2252).

[0414] FIG. 22b illustrates a descriptive table (2278) for the schematic diagram shown in FIG. 22a according to some implementations. U1 through U8 are CMOS OpAmps. M1, M2 and M3 are N-channel MOSFET transistors having a shutter ratio of length (L) = 0.18 u and a shutter ratio of width (W) = 0.9 u according to some implementations.

[0415] Exemplary system of a hyperbolic tangent block

[0416] FIG. 23a illustrates a schematic diagram of a hyperbolic tangent function block (2300) according to some implementations. The hyperbolic tangent function (e.g., modules X2 (20080) and X7 (20090) described above with reference to FIG. 20a through 20f) is implemented using operational amplifiers (U1 (2312), U2 (2314), U3 (2316), U4 (2318), U5 (2320), U6 (2322), U7 (2328) and U8 (2330)) and NMOS transistors (M1 (2332), M2 (2334) and M3 (2336)). In this configuration, contact tanh_in (2306) is a module input, contact input Vdd1 (2304) is a positive supply voltage of +1.8V relative to GND (2308), and contact Vss1 (2302) is a negative supply voltage of -1.0V relative to GND. Also in this configuration, U4 (2318) has a reference voltage source of -0.1V, and the voltage is set by dividers R10 (2356) and R11 (2358). U5 (2320) has a reference voltage source of 1.2V, and the voltage is set by dividers R12 (2360) and R13 (2362). U6 (2322) is a reference voltage source of 0.32687V, and the voltage is set by dividers R14 (2364) and R15 (2366). U7 (2328) has a reference voltage source of -0.5V, and the voltage is set by dividers R16 (2368) and R17 (2370). U8 (2330) has a reference voltage source of -0.33V, and the voltage is set by dividers R18 (2372) and R19 (2374). The hyperbolic tangent function is formed by adding reference voltages corresponding to the differential module created in transistors M1 (2332) and M2 (2334). The current mirror for the differential stage is obtained by an active modulation operational amplifier U3 (2316) and an NMOS transistor M3 (2336). By means of the NMOS transistor M2 (2334) and resistor R5 (2346), the signal is removed from the differential stage and input to the adder U2 (2314).The output signal tanh_out (2310) is removed from the output of the U2 adder (2314).

[0417] FIG. 23b shows a descriptive table (2382) for the schematic diagram shown in FIG. 23a according to some implementations. U1 through U8 are CMOS OpAmps, and M1, M2 and M3 are N-channel MOSFET transistors having a shutter ratio of length (L) = 0.18 u and a shutter ratio of width (W) = 0.9 u.

[0418] Exemplary regime of a single neuron OP1 CMOS OpAmp

[0419] FIGS. 24a through 24c illustrate schematic diagrams of a single-neuron OP1 CMOS OpAmp (2400) according to some implementations. This example is a variation of a single-neuron operational amplifier made in CMOS according to the OP1 scheme described herein. In this scheme, contacts V1 (2410) and V2 (2408) are the inputs of the single neuron, contact bias (2406) is a voltage of +0.4 V relative to GND, contact input Vdd (2402) is a positive supply voltage of +5.0 V relative to GND, contact Vss (2404) is GND, and contact output (2474) is the output of the single neuron. The parameters of the CMOS transistors are determined by the ratio of geometric dimensions: L (length of the gate channel) and W (width of the gate channel). This Op Amp has two current mirrors. Current mirrors on NMOS transistors M3 (2420), M6 (2426), and M13 (2440) provide offset currents for differential pairs on NMOS transistors M2 (2418) and M5 (2424). Current mirrors on PMOS transistors M7 (2428), M8 (2430), and M15 (2444) provide offset currents for differential pairs on PMOS transistors M9 (2432) and M10 (2434). In the first differential amplifier stage, NMOS transistors M2 (2418) and M5 (2424) are amplified, and PMOS transistors M1 (2416) and M4 (2422) act as active current loads. From transistor M5 (2424), a signal is output to the PMOS gate of transistor M13 (2440). From the M2 (2418) transistor, the signal is output to the right input of the second differential amplifier stage of the PMOS transistors M9 (2432) and M10 (2434). The NMOS transistors M11 (2436) and M12 (2438) serve as active current loads for the M9 (2432) and M10 (2434) transistors.The M17 (2448) transistor is switched on according to the common source method for a positive half-wave signal. The M18 (2450) transistor is switched on according to the common source method for a negative half-wave signal. To increase the total load capacity of the Op Amp, the inverters of the M17 (2448) and M18 (2450) transistors are enabled at the outputs of the M13 (2440) and M14 (2442) transistors.

[0420] FIG. 24d illustrates a descriptive table (2476) for the schematic diagrams shown in FIG. 24a through 24c according to some implementations. The weights of the connections of a single neuron (including two inputs and one output) are set by the following resistor ratios: w1 = (Rp / R1+) - (Rn / R1 -); w2 = (Rp / R2+) - (Rn / R2 -); w bias = (Rp / Rbias+) - (Rn / Rbias -). To obtain exact equivalence, the resistors are normalized (Rnorm - and Rnorm+) as follows: (Rn / R1 -) + (Rn / R2 -) + (Rn / Rbias -) + (Rn / Rnorm -) = (Rp / R1 +) + (Rp / R2 +) + (Rp / Rbias+) + (Rp / Rnorm +). N-channel MOSFET transistors with explicit substrate connections include transistors with L = 0.36u and W = 3.6u (M2 and M5), transistors with L = 0.36u and W = 1.8u (M3, M6, M11, M12, M14 and M16), and a transistor with L = 0.36u and W = 18u (M18). P-channel MOSFET transistors with explicit substrate connections include transistors with L = 0.36u and W = 3.96u (M1, M4, M7, M8, M13 and M15), transistors with L = 0.36u and W = 11.88u (M9 and M10), and a transistor with L = 0.36u and W = 39.6u (M17).

[0421] Exemplary regime of a single-neuron OP3 CMOS OpAmp

[0422] FIGS. 25a through 25d illustrate schematic diagrams of variations of a single neuron (25000) on operational amplifiers constructed on CMOS according to the OP3 method, according to some implementations. According to some implementations, the single neuron consists of three simple operational amplifiers (OpAmps). The unit neuron adder is performed on two Opamps with bipolar power supplies, and the RELU activation function is performed on unipolar power supplies and OpAmps of which 10 are. Transistors M1 (25028) through M16 (25058) are used to sum the negative connections of the neuron. Transistors M17 (25060) through M32 (25090) are used to add the positive connections of the neuron. The RELU activation function is performed on transistors M33 (25092) through M46 (25118). In this regime, contacts V1 (25008) and V2 (25010) are the inputs of a single neuron, contact bias (25002) is a voltage of +0.4V relative to GND, contact input Vdd (25004) is a positive supply voltage of +2.5V relative to GND, contact Vss (25006) is a negative supply voltage of -2.5V, and contact output (25134) is the output of the single neuron. The parameters of the CMOS transistors used in the single neuron are determined by the ratio of the following geometric dimensions: L (length of the gate channel) and W (width of the gate channel). Consider the operation of the simplest OpAmp included in the single neuron. Each OpAmp has two current mirrors. The current mirror of NMOS transistors M3 (25032) (M19 (25064), M35 (25096)), M6 (25038) (M22 (25070), M38 (25102)) and M16 (25058) (M32 (25090), M48 (2512)) provides an offset of NMOS transistors M2 (25030) (M18 (25062), M34 (25094)) and M5 (25036) (M21 (25068), M35 (25096)).The current mirrors of PMOS transistors M7 (25040) (M23 (25072), M39 (25104)), M8 (25042) (M24 (25074), M40 (25106)) and M15 (25056) (M31 (2588)) provide differential pair offset currents in PMOS transistors M9 (25044) (M25 (25076), M41 (25108)) and M10 (25046) (M26 (25078), M42 (25110)). In the first differential amplifier stage, NMOS transistors M2 (25030) (M18 (25062), M34 (25094)) and M5 (25036) (M21 (25068), M37 (25100)) are amplified, and PMOS transistors M1 (25028) (M17 (25060), M33 (25092)) and M4 (25034) (M20 (25066), M36 (25098)) act as active current loads. From transistor M5 (25036) (M21 (25068), M37 (25100)), a signal is input to the PMOS gate of transistor M13 (25052) (M29 (25084), M45 (25116)). From transistor M2 (25030) (M18 (25062), M34 (25094)), the signal is input to the right input of the second differential amplifier stage of PMOS transistors M9 (25044) (M25 (25076), M41 (25108)) and M10 (25046) (M26 (25078), M42 (25110)). NMOS transistors M11 (25048) (M27 (25080), M43 (25112)) and M12 (25048) (M28 (25080), M44 (25114)) serve as active current loads for transistors M9 (25044) (M25 (25076), M41 (5108)) and M10 (25046) (M26 (25078), M42 (25110)). Transistor M13 (25052) (M29 (25082), M45 (25116)) is included in the regime with a common source for positive half-wave signals.Transistor M14 (25054) (M30 (25084), M46 (25118)) is switched on according to the common source regime for a negative half-wave signal.

[0423] The weights of the connections of a single neuron (containing two inputs and one output) are set by the following resistor ratios: w1 = (R Feedback / R1+) - (R Feedback / R1 -); w2 = (R Feedback / R2+) - (R Feedback / R2 -); wbias = (R Feedback / Rbias+) - (R Feedback / Rbias-); w1 = (R p*K amp / R1+) - (R n*K amp / R1 -); w2 = (R p*K amp / R2+) - (R n*K amp / R2 -); wbias = (R p*K amp / Rbias+) - (R n*K amp / Rbias-), where K amp = R1ReLU / R2ReLU. R Feedback = 100k - used only in the calculation of w1, w2, and wbias. According to some implementations, exemplary values ​​include the following: R feedback = 100k, Rn = Rp = Rcom = 10k, K amp ReLU = 1 + 90k / 10k = 10, w1 = (10k * 10 / 22.1k) - (10k * 10 / 21.5k) = -0.126276, w2 = (10k * 10 / 75k) - (10k * 10 / 71.5k) = - 0.065268, wbias = (10k * 10 / 71.5k) -(10k * 10 / 78.7k) = 0.127953.

[0424] The input to the negative link adder of neurons (M1 to M17) is received from the positive link adder of neurons (M17 to M32) through an Rcom resistor.

[0425] FIG. 25e illustrates a descriptive table (25136) for the schematic diagrams shown in FIG. 25a through 25d according to some embodiments. N-channel MOSFET transistors having explicit substrate connections include transistors with L = 0.36u and W = 3.6u (M2, M5, M18, M21, M34 and M37), and transistors with L = 0.36u and W = 1.8u (M3, M6, M11, M12, M14, M16, M19, M22, M27, M28, M32, M38, M35, M38, M43, M44, M46 and M48). P-channel MOSFET transistors having explicit substrate connections include transistors with L = 0.36u and W = 3.96u (M1, M4, M7, M8, M13, M15, M17, M20, M23, M24, M29, M31, M33, M36, M39, M40, M45, and M47), and transistors with L = 0.36u and W = 11.88u (M9, M10, M25, M26, M41, and M42).

[0426] An exemplary method for analog hardware implementation of a trained neural network

[0427] FIGS. 27a through 27j illustrate a flowchart of a method (2700) for a hardware implementation (2702) of a neural network according to some implementations. The method is performed (2704) in a computing device (200) having one or more processors (202) and a memory (214) for storing one or more programs configured for execution by one or more processors (202) (e.g., using a neural network transformation module (226)). The method includes the step (2706) of obtaining a neural network topology (e.g., topology (224)) and weights (e.g., weights (222)) of a trained neural network (e.g., networks (220)). In some implementations, the trained neural network is trained (2708) using software simulations to generate weights.

[0428] The method also includes the step (2710) of converting the neural network topology into an equivalent analog network of analog components. Next, referring to FIG. 27c, in some implementations, the neural network topology includes one or more neuron layers (2724). Each neuron layer computes each output based on a respective mathematical function. In this case, the step of converting the neural network topology into an equivalent analog network of analog components includes the step (2726) of performing a series of steps for each layer of the one or more neuron layers. The series of steps includes, for each layer, the step (2728) of identifying one or more function blocks based on each mathematical function. Each function block has a respective schematic implementation having block outputs that follow the outputs of each mathematical function. In some implementations, the step of identifying one or more function blocks includes the step (2730) of selecting one or more function blocks based on the type of each layer. For example, a layer may be composed of neurons, and the output of the layer is a linear superposition of its inputs. If the output of a layer is linear nesting or similar pattern identification, the step of selecting one or more function blocks is based on the identification of these layer types. Some implementations determine whether the number of outputs > 1 and then use trapezoidal or pyramidal transformations.

[0429] Next, referring to FIG. 27d, in some implementations, one or more function blocks include (i) block output A weighted summation block (2736) having, wherein ReLU is a rectified linear unit (ReLU) activation function or a similar activation function (e.g., ReLU with a threshold), and V i represents the i-th input, and w i The above weighted summation block, wherein represents a weight corresponding to the i-th input, bias represents a bias value, and ∑ is a summation operator; (ii) block output As a signal multiplier block (2738) having V i represents the i-th input, and V j represents the j-th input, and coeff is a predetermined coefficient, the signal multiplier block; (iii) block output A sigmoid activation block (2740) having V as the input, and A and B as predetermined coefficient values ​​of the sigmoid activation block (e.g., A = -0.1; B = 11.3); (iv) block output As a hyperconvex tangent activation block (2742) having V in The hyperconvex tangent activation block, wherein represents an input and A and B are predetermined coefficient values ​​(e.g., A = 0.1, B = -10.1); and a signal delay block (2744) having a block output U(t) = V(t-dt), wherein t represents a current time period, V(t-1) represents the output of the signal delay during a previous time period (t-1), and dt is a delay value, comprising one or more basic function blocks (e.g., basic function blocks (232)) selected (2734) from a group composed of the signal delay blocks.

[0430] Now, referring again to FIG. 27c, the series of steps also includes the step (2732) of creating a multilayer network of analog neurons based on arranging one or more function blocks. Each analog neuron implements a function of one or more function blocks, and each analog neuron of the first layer of the multilayer network is connected to one or more analog neurons of the second layer of the multilayer network.

[0431] Now, referring again to FIG. 27a, for some networks such as GRU and LSTM, according to some implementations, the step (2710) of converting the neural network topology into an equivalent analog network of analog components requires more complex processing. Next, referring to FIG. 27e, it is assumed that the neural network topology includes one or more neuron layers (2746). It is also assumed that each neuron layer calculates each output based on a respective mathematical function. In this case, the step of converting the neural network topology into an equivalent analog network of analog components includes the following: (i) the step (2748) of decomposing the first layer of the neural network topology into a plurality of sub-layers, including decomposing the mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements the intermediate mathematical function. In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and the step of decomposing the mathematical function includes the step (2750) of adjusting one or more weights to generate a mathematical function by combining one or more intermediate functions; and (ii) the step (2752) of performing a series of steps for each sublayer of the first layer of the neural network topology. The series of steps includes the step (2754) of selecting one or more sub-function blocks based on each intermediate mathematical function for each sublayer; and the step (2756) of creating a multilayer analog subnet of each analog neuron based on arranging one or more sub-function blocks. Each analog neuron implements each function of one or more sub-function blocks, and each analog neuron of the first layer of the multilayer analog subnet is connected to one or more analog neurons of the second layer of the multilayer analog subnet.

[0432] Next, referring to FIG. 27h, it is assumed that the neural network topology includes one or more GRU or LSTM neurons (2768). In that case, the step of transforming the neural network topology includes the step of generating one or more signal delay blocks for each recurrent connection of one or more GRU or LSTM neurons (2770). In some implementations, an external cycle timer activates one or more signal delay blocks for a fixed time period (e.g., 1, 5, or 10 time steps). Some implementations use multiple delay blocks for one signal to generate additional time shifts. In some implementations, the activation frequency of one or more signal delay blocks is synchronized with the network input signal frequency. In some implementations, one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency for the neural network topology (2772). In some implementations, this predetermined input signal frequency may vary depending on the application, such as Human Activity Recognition (HAR) or PPG. For example, the predetermined input signal frequencies are 30 to 60 Hz for video processing, about 100 Hz for HAR and PPG, 16 KHz for sound processing, and about 1 to 3 Hz for battery management. Some implementations enable different signal delay blocks at different frequencies.

[0433] Next, referring to FIG. 27i, it is assumed that the neural network topology includes one or more layers of neurons that perform unlimited activation functions (2774). In some implementations, in this case, the step of transforming the neural network topology includes the step of applying one or more transformations selected from a group consisting of the steps of: replacing unlimited activation functions with limited activation functions (e.g., replacing ReLU with a threshold ReLU) (2778); and adjusting the connections or weights of the equivalent analog network for one or more predetermined inputs so as to minimize the difference in output between the trained neural network and the equivalent analog network (2780).

[0434] Now, referring again to FIG. 27a, the method also includes the step (2712) of calculating a weight matrix for an equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between the analog components of the equivalent analog network.

[0435] The method also includes the step (2714) of generating a schematic model for implementing an equivalent analog network based on a weight matrix, including selecting component values ​​for analog components. Next, referring to FIG. 27b, in some implementations, the step of generating a schematic model includes the step (2716) of generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to each weight of the weight matrix and represents a resistance value. In some implementations, the method includes the step of regenerating only the resistance matrix for the resistors of the retrained network. In some implementations, the method further includes the step (2718) of obtaining new weights for the trained neural network, the step (2720) of calculating a new weight matrix for the equivalent analog network based on the new weights, and the step (2722) of generating a new resistance matrix for the new weight matrix.

[0436] Next, referring to FIG. 27j, in some implementations, the method further comprises the step (2782) of generating one or more lithographic masks for manufacturing a circuit that implements an equivalent analog network of analog components based on a resistance matrix (e.g., generating masks (250 and / or 252) using a mask generation module (248). In some implementations, the method comprises the step of regenerating only the masks for resistors for the trained networks (e.g., masks (250)). In some implementations, the method further comprises the steps of: (i) obtaining new weights for the trained neural network (2784); (ii) calculating a new weight matrix for the equivalent analog network based on the new weights (2786); (iii) generating a new resistance matrix for the new weight matrix (2788); and (iv) generating a new lithographic mask for manufacturing a circuit that implements an equivalent analog network of analog components based on the new resistance matrix (2790).

[0437] Now, referring again to FIG. 27g, the analog components include a plurality of operational amplifiers and a plurality of resistors (2762). Each operational amplifier represents an analog neuron of an equivalent analog network, and each resistor represents a connection between two analog neurons. Some implementations include other analog components such as four-quadrant multipliers, sigmoid and hyperbolic tangent function circuits, delay lines, summers and / or dividers. In some implementations, the step of selecting component values ​​of the analog components (2764) includes the step (2766) of performing gradient descent and / or other weighted quantization methods to identify possible resistance values ​​for the plurality of resistors.

[0438] Now, referring again to FIG. 27f, in some implementations, the method further includes the step of implementing specific activation functions (e.g., softmax) in the output layer of the digital. In some implementations, the method further includes the step of generating an equivalent digital network of digital components for one or more output layers of the neural network topology (2758), and the step of connecting the outputs of one or more layers of the equivalent analog network to the equivalent digital network of digital components (2760).

[0439] An exemplary method for implementing a neural network on limited analog hardware

[0440] FIGS. 28a through 28s illustrate a flowchart of a method (28000) for a hardware implementation (28002) of a neural network according to some implementations and hardware design constraints. The method is performed (28004) in a computing device (200) having one or more processors (202) and a memory (214) for storing one or more programs configured for execution by one or more processors (202) (e.g., using a neural network transformation module (226)). The method includes the step (28006) of obtaining a neural network topology (e.g., topology (224)) and weights (e.g., weights (222)) of a trained neural network (e.g., networks (220)).

[0441] The method also includes the step (28008) of calculating one or more connection constraints based on analog integrated circuit (IC) design constraints (e.g., constraints (236)). For example, IC design constraints may set a current limit (e.g., 1A), and neuron schematics and operational amplifier (OpAmp) designs may set the OpAmp output current to a range of [0 to 10 mA], which thus limits the output neuron connections to 100. This allows current to flow to the next layer through 100 connections since the neuron has 100 outputs, but since the current at the output of the operational amplifier is limited to 10 mA, some implementations use up to 100 outputs (0.1 mA x 100 = 10 mA). In the absence of such constraints, some implementations use current repeaters to increase the number of outputs, for example, to more than 100.

[0442] The method also includes the step (28010) of converting the neural network topology into an equivalent sparsely connected network of analog components that satisfies one or more connection constraints (e.g., using a neural network transformation module (226)).

[0443] In some implementations, the step of transforming the neural network topology, depending on one or more connection constraints, the possible degree of input connections (N i ) and output connection degree (N o It includes a step (28012) of deriving )

[0444] Next, referring to FIG. 28b, in some implementations, the neural network topology includes at least one densely connected layer having K inputs (neurons of the previous layer) and L outputs (neurons of the current layer) and a weight matrix (U) (28018), and the step of transforming at least one densely connected layer (28020) is such that the input connection degree is N i The output connection degree does not exceed N oTo not exceed, K inputs, L outputs and It includes the step (28022) of constructing an equivalent sparsely connected network having layers.

[0445] Next, referring to FIG. 28c, in some implementations, the neural network topology comprises at least one densely connected layer having K inputs (neurons of the previous layer) and L outputs (neurons of the current layer) and a weight matrix (U) (28024), and the step of transforming at least one densely connected layer (28026) comprises K inputs, L outputs and The method includes the step (28028) of constructing an equivalent sparsely connected network having layers. Each layer (m) has a corresponding weight matrix (U m It is represented as ), where the case of no connection is represented as 0, and the input connection degree is N i The output connection degree does not exceed N o Ensure that it does not exceed. Equation ...is satisfied by a predetermined precision. The predetermined precision is a reasonable precision value that statistically guarantees that the changed network output differs from the reference network output by less than or equal to an allowed error value, and this error value varies depending on the task (typically between 0.1% and 1%).

[0446] Next, referring to FIG. 28d, in some implementations, the neural network topology has K inputs and L outputs, and a maximum input connection degree (P i ), maximum output connection degree (P o It includes a single sparsely connected layer (28030) having a weight matrix (U) and ) where the absence of connections is represented as 0. In this case, the step (28032) of transforming the single sparsely connected layer comprises K inputs, L outputs, The method includes the step (28034) of constructing an equivalent sparsely connected network having layers. Each layer (m) has a corresponding weight matrix (U m It is represented as ), where 0 is used when there are no connections, and the input connectivity level is N i The output connection degree does not exceed N o Ensure that it does not exceed, and the equation It is satisfied with a predetermined precision.

[0447] Next, referring to FIG. 28e, in some implementations, the neural network topology includes a convolutional layer (e.g., a depth-wise convolutional layer or a separable convolutional layer) having K inputs (neurons of the previous layer) and L outputs (neurons of the current layer) (28036). In this case, the step (28038) of converting the neural network topology into an equivalent sparsely connected network of analog components comprises converting the convolutional layer into K inputs, L outputs, and a maximum input connection degree (P i ) and maximum output connection degree (P o ) have, but here P i ≤ N i and P o ≤ N o It includes the step (28040) of decomposing into a single sparsely connected layer.

[0448] Referring to FIG. 28a, the method also includes the step (28014) of calculating a weight matrix for an equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network.

[0449] Now, referring to FIG. 28f, in some implementations, the neural network topology includes a recurrent neural layer (28042), and the step of converting the neural network topology into a sparsely connected network equivalent to analog components (28044) includes the step of converting the recurrent neural network into one or more dense or sparsely connected layers having signal delay connections (28046).

[0450] Next, referring to FIG. 28g, in some implementations, the neural network topology includes a recurrent neuron layer (e.g., a long short-term memory (LSTM) layer or a gated recurrent unit (GRU) layer), and the step of converting the neural network topology into a sparsely connected network equivalent to analog components includes the step of decomposing the recurrent neuron layer into several layers, wherein at least one of the layers has K inputs (neurons of the previous layer) and L outputs (neurons of the current layer) and a weight matrix (U), which is equivalent to a dense or sparsely connected layer where the absence of connections is represented as 0.

[0451] Next, referring to FIG. 28h, in some implementations, the method includes the step of performing a transformation of a single-layer perceptron having one computational neuron. In some implementations, the neural network topology has K inputs, a weight matrix (U ∈ R K It includes a single-layer perceptron having a computational neuron having an activation function (F) and (28054). In this case, the step (28056) of converting the neural network topology into an equivalent sparsely connected network of analog components comprises: (i) a step (28058) of deriving a degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints; (ii) an equation The method comprises the step (28060) of calculating the number of layers (m) for an equivalent sparsely connected network using [the method]; and (iii) the step (28062) of constructing an equivalent sparsely connected network having K inputs, m layers, and a degree of connection (N). The equivalent sparsely connected network includes one or more analog neurons in each of the m layers. Each analog neuron in the first m-1 layers implements an identity transformation, and the analog neuron in the last layer implements the activation function (F) of the computational neuron of a single-layer perceptron. Additionally, in this case, the step (28064) of calculating a weight matrix for the equivalent sparsely connected network includes the step (28066) of calculating a weight vector (W) for the connections of the equivalent sparsely connected network by solving a system of equations based on a weight vector (U). The system of equations includes K equations having S variables, where S is an equation It is calculated using .

[0452] Next, referring to FIG. 28i, in some implementations, the method includes the step of performing a transformation of a single-layer perceptron having L computational neurons. In some implementations, the neural network topology includes K inputs, a single-layer perceptron having L computational neurons, and a weight matrix (V) containing a weight row for each of the L computational neurons (28068). In this case, the step of transforming the neural network topology into an equivalent sparsely connected network of analog components (28070) comprises: (i) deriving a degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints (28072); (ii) an equation (iii) a step of calculating the number of layers (m) for an equivalent sparsely connected network using [a method] (28074); (iii) a step of decomposing a single-layer perceptron into L single-layer perceptron networks (28076), wherein each single-layer perceptron network comprises a respective computational neuron of L computational neurons; (iv) a step of constructing an equivalent pyramidal subnetwork for each single-layer perceptron network (28078) of the L single-layer perceptron networks, having K inputs, m layers, and a degree of connectivity (N) (28080), wherein the equivalent pyramidal subnetwork comprises one or more respective analog neurons in each of the m layers, wherein each analog neuron of the first m-1 layers implements an identity transformation, and the analog neuron of the last layer implements an activation function of the respective computational neuron corresponding to each single-layer perceptron; and (v) a step of connecting the inputs of each equivalent pyramidal subnetwork for L single-layer perceptron networks to form an input vector having L*K inputs, and a step of constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork (28082). Additionally, in this case, a step of calculating a weight matrix for the equivalent sparsely connected network (28084) for each single-layer perceptron network (28086) of L signal layer perceptron networks, (i) a weight vector (U=V i A step (28088) of setting ) wherein the i-th row of the weight matrix (V) corresponds to each computational neuron corresponding to each single-layer perceptron network, and (ii) solving an equation system based on the weight vector (U) to obtain a weight vector (W) for the connections of each equivalent pyramidal sub-network. iIt includes a step (28090) of calculating ). The equation system includes K equations having S variables, where S is an equation It is calculated using .

[0453] Next, referring to FIG. 28j, in some implementations, the method includes the step of performing a transformation algorithm for a multilayer perceptron. In some implementations, the neural network topology includes a multilayer perceptron having K inputs and S layers, wherein each layer (i) of the S layers has a corresponding set of computational neurons (L i ) and L i Corresponding weight matrices (V) containing weight rows for each computational neuron of the computational neurons i It includes ) (28092). In this case, the step (28094) of converting the neural network topology into an equivalent sparsely connected network of analog components comprises: (i) a step (28096) of deriving a degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints; (ii) a multilayer perceptron A step of decomposing into 1 single-layer perceptron networks (28098), wherein each single-layer perceptron network comprises a respective computational neuron of Q computational neurons, and the step of decomposing into a multilayer perceptron comprises a step of duplicating one or more of the K inputs shared by the Q computational neurons; (iii) for each single-layer perceptron network (28100) of the Q single-layer perceptron networks, (a) equation A step of calculating the number of layers (m) for each equivalent pyramidal subnetwork using K i,j is the number of inputs to each computational neuron of the multilayer perceptron, the computational step, and (b) K i,jA step (28104) of constructing an equivalent pyramidal subnetwork for each single-layer perceptron network having inputs, m layers, and degree of connection (N), wherein the equivalent pyramidal subnetwork comprises one or more analog neurons in each of the m layers, wherein each analog neuron in the first m-1 layers implements an identity transformation, and the analog neuron in the last layer implements an activation function of a computational neuron corresponding to each single-layer perceptron network; and (iv) Q*K i,j The method comprises the step of constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork, including the step of connecting the inputs of each equivalent pyramidal subnetwork for Q single-layer perceptron networks to form an input vector having inputs. In this case, the step of calculating a weight matrix for the equivalent sparsely connected network (28108) comprises, for each single-layer perceptron network (28110) of the Q single-layer perceptron networks, (i) a weight vector ( A step (28112) of setting ) wherein the i-th row of the weight matrix (V) corresponds to each computational neuron corresponding to each single-layer perceptron network, and j is the layer corresponding to each computational neuron in a multilayer perceptron; and (ii) solving a system of equations based on the weight vector (U) to obtain a weight vector (W) for the connections of each equivalent pyramidal sub-network. i It includes a step (28114) of calculating ). The equation system is K having S variables i,j It includes equations, and S is an equation It is calculated using .

[0454] Next, referring to FIG. 28k, in some implementations, the neural network topology comprises a convolutional neural network (CNN) having K inputs and S layers (28116), wherein each layer (i) of the S layers has a corresponding set of computational neurons (L i ) and L i Corresponding weight matrices (V) containing weight rows for each computational neuron of the computational neurons i It includes ) (28116). In this case, the step (28118) of converting the neural network topology into an equivalent sparsely connected network of analog components comprises: (i) a step (28120) of deriving the degree of connectivity (N) for the equivalent sparsely connected network according to one or more connection constraints; (ii) the CNN A step of decomposing into Q single-layer perceptron networks (28122), wherein each single-layer perceptron network comprises a respective computational neuron of Q computational neurons, and the step of decomposing the CNN comprises a step of duplicating one or more of the K inputs shared by the Q computational neurons; (iii) for each single-layer perceptron network of the Q single-layer perceptron networks, (a) equation A step of calculating the number of layers (m) for each equivalent pyramidal subnetwork using, where j is the layer corresponding to each computational neuron in the CNN, and K i, j is the number of inputs for each computational neuron in the CNN, the calculating step; and (b) K i,jA step of constructing an equivalent pyramidal subnetwork for each single-layer perceptron network having inputs, m layers, and degree of connection (N), wherein the equivalent pyramidal subnetwork comprises one or more analog neurons in each of the m layers, wherein each analog neuron in the first m-1 layers implements an identity transformation, and the analog neuron in the last layer implements an activation function of a computational neuron corresponding to each single-layer perceptron network; and (iv) Q*K i,j The method comprises the step of constructing an equivalent sparsely connected network by connecting each equivalent pyramidal subnetwork, including the step of connecting the inputs of each equivalent pyramidal subnetwork for Q single-layer perceptron networks to form an input vector having inputs. In this case, the step of calculating a weight matrix for the equivalent sparsely connected network (28132) comprises, for each single-layer perceptron network (28134) of the Q signal layer perceptron networks, (i) a weight vector ( A step of setting ) wherein the i-th row of the weight matrix (V) corresponds to each computational neuron corresponding to each layer perceptron network, and j is the layer corresponding to each computational neuron in the CNN; and (ii) the equation system K having S variables i,j It includes equations, and S is an equation It includes the step of calculating using the above.

[0455] Next, referring to FIG. 28l, in some implementations, the method includes the step of converting two layers into a trapezoid-based network. In some implementations, the neural network topology is a layer (L) having K inputs and K neurons. P ), a layer having L neurons (L n ) and weight matrix (W ∈ RLΥK Includes ) (28140), where R is a set of real numbers, and hierarchy (L p Each neuron of ) is in the hierarchy (L n It is connected to each neuron of ), and the hierarchy (L n Each neuron of ) performs an activation function (F), and the layer (L n The output of ) is the equation for input x It is made to be calculated using . In this case, the step (28142) of converting the neural network topology into an equivalent sparsely connected network of analog components comprises (i) the degree of possible input connections (N) according to one or more connection constraints. I > 1) and possible output connection extent (N O Step of deriving >1) (28144); and (ii) Based on the decision, a layer having K analog neurons performing an identity activation function ( LA p ), performing the identity activation function A layer (LA) having 2 analog neurons h A layer (LA) having L analog neurons that perform ), and an activation function (F). o Includes ) but hierarchy (LA p Each analog neuron of ) is N O Having outputs, layer (LA h Each analog neuron of ) is N I Number of inputs and N O It has fewer than 1 output, and the hierarchy (LA o Each analog neuron of ) N I The method includes the step of performing a trapezoidal transformation, which includes the step (28146) of constructing a 3-layer analog network having inputs. In some cases, the step (28148) of calculating a weight matrix for an equivalent sparsely connected network comprises layers (LA o The total output of ) is the equation To be calculated using, of the variables Matrix equations containing k equations Solving for sparse weight matrices (W o and W h Includes the step (2850) of generating a sparse weight matrix (W o ∈ R KΥM ) is a hierarchy (LA p ) and hierarchy (LA h Represents the connections between ), and the sparse weight matrix (W h ∈R MΥL ) is a hierarchy (LA h ) and hierarchy (LA o Represents the connections between ).

[0456] Next, referring to FIG. 28m, in some implementations, the step of performing a trapezoidal transformation is, In accordance with the decision, (i) A sublayer (L) having as many as K' neurons p1 ) and a sublayer (L) having (K-K') neurons p2 To obtain ) hierarchy (L p (ii) a step of dividing ) (28154); (ii) a sublayer having K' neurons (L p1 Regarding ), a step of performing a configuring step and a generating step (28156); and (iii) a sublayer having K-K' neurons (L p2 For ), it further includes a step (28158) of repeatedly performing a step of dividing, a step of configuring, and a step of generating.

[0457] Next, referring to FIG. 28n, the method includes the step of converting a multilayer perceptron into a trapezoid-based network. In some implementations, the neural network topology includes a multilayer perceptron network (28160), and the method further includes the step (28162) of iteratively performing a trapezoid transformation for each consecutive pair of layers of the multilayer perceptron network and calculating a weight matrix for an equivalent sparsely connected network.

[0458] Next, referring to FIG. 28o, the method includes the step of converting a recurrent neural network into a trapezoid-based network. In some implementations, the neural network topology includes a recurrent neural network (RNN) comprising (i) linear combination computation for two fully connected layers, (ii) element-wise addition, and (iii) non-linear function computation (28164). In such cases, the method further includes the step (28166) of performing a trapezoidal conversion for (i) two fully connected layers and (ii) non-linear function computation and computing a weight matrix for an equivalent sparsely connected network. Element-wise addition is a common operation that can be implemented in networks of any structure, and an example is provided above. Non-linear function computation is N o and N i It is a per-neuron operation independent of restrictions, and is generally calculated individually in each neuron as a 'sigmoid' or 'tanh' block.

[0459] Next, referring to FIG. 28p, the neural network topology comprises a Long Short-Term Memory (LSTM) network or a Gated Recurrence Unit (GRU) network comprising (i) a linear combination of multiple fully connected layers, (ii) element-wise addition, (iii) an Hadamard product, and (iv) multiple non-linear function calculations (sigmoid and hyperbolic tanject operations) (28168). In this case, the method further comprises the step (28170) of performing a trapezoidal transformation on (i) two fully connected layers and (ii) multiple non-linear function calculations and calculating a weight matrix for an equivalent sparsely connected network. Element-wise addition and the Hadamard product are common operations that can be implemented in networks of any structure described above.

[0460] Next, referring to FIG. 28q, the neural network topology comprises a convolutional neural network (CNN) comprising (i) a plurality of partially connected layers (e.g., a sequence of convolutional and pooling layers; each pooling layer is assumed to be a convolutional layer with a stride greater than 1) and (ii) one or more fully connected layers (the sequence ends at the fully connected layers) (28172). In this case, the method comprises (i) transforming the plurality of partially connected layers into equivalent fully connected layers by inserting missing connections with a weight of 1 (28174); and for each consecutive pair of equivalent fully connected layers and one or more fully connected layers, performing a ladder transformation and calculating a weight matrix for an equivalent sparsely connected network (28176).

[0461] Next, referring to FIG. 28r, the neural network topology has K inputs, L output neurons, and a weight matrix (U ∈ R LΥK It includes (28178), where R is a set of real numbers, and each output neuron performs an activation function (F). In this case, the step (28180) of converting the neural network topology into an equivalent sparsely connected network of analog components is such that, depending on one or more connection constraints, the possible degree of input connections (N I > 1) and possible output connection extent (N O Step of deriving >1) (28182); (ii) set {0, 1, ..., Step of selecting a parameter (p) from} (28184); (iii) depending on the decision that p > 0, the pyramidal neural network in its output layer The method comprises the step of constructing a pyramidal neural network (28186) that forms a first p layer of an equivalent sparsely connected network to have N neurons, wherein each neuron of the pyramidal neural network performs an identity function; and (iv) the step of performing an approximation transformation including the step of constructing a ladder-shaped neural network (28188) having Np inputs and L outputs. Each neuron of the last layer of the ladder-shaped neural network performs an activation function (F) and all other neurons perform an identity function. Additionally, in this case, the step of calculating a weight matrix for the equivalent sparsely connected network (28190) comprises (i) the step of generating weights for the pyramidal neural network (28192), wherein (i) the following rule (a) As, here C is a non-zero constant, and In, the above rule, and (b) k i For all weights (j) of the neuron excluding, The generating step comprising: (ii) setting the weights of all neurons (i) of the first layer of the pyramidal neural network according to the rule; and (ii) setting all other weights of the pyramidal neural network to 1; and (ii) generating weights for the trapezoidal neural network (28194), wherein (i) equation The generating step includes: setting weights of each neuron (i) of the first layer of the trapezoidal neural network (which is the (p+1)th layer when considering the entire network) according to; and (ii) setting other weights of the trapezoidal neural network to 1.

[0462] Next, referring to FIG. 28s, in some implementations, the neural network topology has K inputs at the i-th layer, S layers, and weight matrix for computational neurons and the i-th layer ( Includes )(28196), where L o=K. In this case, the step (28198) of converting the neural network topology into an equivalent sparsely connected network of analog components is, for each layer (j) (28200) of the S layers of the multilayer perceptron, L j-1 inputs, L j output neurons and weight matrix (U j By performing an approximation transformation on each single-layer perceptron composed of ), each pyramid-trapezoidal network (PTNNX j (ii) a step of constructing a pyramid-trapezoid network (28202); and (ii) a step of constructing an equivalent sparsely connected network by stacking each pyramid-trapezoid network (e.g., the output of the pyramid-trapezoid network (PTNNXj-1) is set as an input to PTNNXj) (28204).

[0463] Referring again to FIG. 28a, in some implementations, the method further includes the step (28016) of generating a schematic model for implementing an equivalent sparsely connected network using a weight matrix.

[0464] An exemplary method for calculating resistance values ​​for an analog hardware implementation of a trained neural network

[0465] FIGS. 29a through 29f illustrate a flowchart of a method (2900) for a hardware implementation (2902) of a neural network according to some implementations and hardware design constraints. The method is performed (2904) in a computing device (200) having one or more processors (202) and a memory (214) for storing one or more programs configured for execution by one or more processors (202) (e.g., using a weight quantization generation module (238)).

[0466] The method includes the step (2906) of obtaining a neural network topology (e.g., topology (224)) and weights (e.g., weights (222)) of a trained neural network (e.g., networks (220)). In some implementations, weight quantization is performed during training. In some implementations, the trained network is trained (2908) such that each layer of the neural network topology has quantized weights (e.g., specific values ​​from a list of individual values; e.g., each layer has only three weight values ​​of +1, 0, and -1).

[0467] The method also includes the step (2910) of converting the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (e.g., using a neural network conversion module (226)). Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.

[0468] The method also includes the step (2912) of calculating a weight matrix for an equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.

[0469] The method also includes the step (2914) of generating a resistance matrix for a weight matrix. Each element of the resistance matrix corresponds to each weight of the weight matrix and represents a resistance value.

[0470] Next, referring to FIG. 29B, in some implementations, the step of generating a resistance matrix for a weight matrix includes a simplified gradient descent-based iterative method for finding a set of resistors. In some implementations, the step of generating a resistance matrix for a weight matrix comprises (i) possible resistance values ​​{R min , R max Obtain a predetermined range of} and within the predetermined range, the initial base resistance value (R base(ii) a step of selecting (2916), wherein, for example, the range and base resistance are values ​​of elements of a weighting matrix; the values ​​are determined by a manufacturing process; ranges that can actually be manufactured—resistors; large resistors are undesirable; selected according to the quantization of those that can actually be manufactured, and in some implementations, the predetermined range of possible resistance values ​​includes resistors according to the nominal series E24 of the range 100 KΩ to 1 MΩ (2918); the step of acquiring and selecting; (ii) within the predetermined range, within a limited set of lengths of resistance values ​​{R i , R j For all combinations of}, the range [-R base , R base Possible weights within ] A step (2920) of selecting a limited set of lengths of resistance values ​​that provides the most uniform distribution of, wherein in some implementations, weight values ​​are outside this range but the mean squared distance between weights within this range is minimal; (iii) R+ = R- To make the resistor set value closest to , the maximum weight (w) of the input connections and bias of each neuron or layer of the equivalent analog network max Based on ), for each analog neuron or each layer of the equivalent analog network, a step (2922) of selecting a resistance value R+ = R- from a limited set of resistance values, wherein in some implementations, R+ and R- are independently selected for each layer of the equivalent analog network (2924), and in some implementations, R+ and R- are independently selected for each analog neuron of the equivalent analog network (2926); and (iv) for each element of the weight matrix, an equation for all possible values ​​of R1 and R2 within a predetermined range of possible resistance values A step (2928) of selecting each first resistance value (R1) and each second resistance value (R2) that minimizes the error according to, wherein w is each element of the weight matrix and r err It includes the step of selecting, which is a predetermined relative tolerance value for possible resistance values.

[0471] Next, referring to FIG. 29c, some implementations perform weight subtraction. In some implementations, one or more first weights and one or more first inputs of a weight matrix represent one or more connections to a first operational amplifier of an equivalent analog network (2930). The method further comprises, before generating a resistance matrix (2932), (i) modifying one or more first weights by a first value (e.g., dividing one or more first weights by the first value to reduce the weight range, or multiplying one or more first weights by the first value to increase the weight range) (2934); and (ii) configuring a first operational amplifier (2936) to multiply the first value by a linear combination of one or more first weights and one or more first inputs before performing an activation function. Some implementations perform weight subtraction to change the multiplication factor of one or more operational amplifiers. In some implementations, the set resistance values ​​generate weights in some ranges, and in some of these ranges, the error will be larger than in others. Assuming there are only two nominal values ​​(e.g., 1Ω and 4Ω), these resistors can generate weights [-3; -0.75; 0; 0.75; 3]. Assuming the weights of the first layer of the neural network are {0, 9} and the weights of the second layer are {0, 1}, some implementations reduce the overall error by dividing the weights of the first layer by 3 and multiplying the weights of the second layer by 3. Some implementations consider limiting weight values ​​during training by adjusting the loss function (e.g., using an l1 or l2 regularizer) so that the resulting network does not have weights that are too large relative to the set of resistors.

[0472] Next, referring to FIG. 29d, the method further includes the step of limiting the weights to intervals. For example, the method further includes the step of obtaining a predetermined range of weights (2938), and the step (2940) of updating a weight matrix according to the predetermined weight range so that an equivalent analog network produces an output similar to that of a neural network trained on the same input.

[0473] Next, referring to FIG. 29e, the method further includes a step of reducing the weight sensitivity of the network. For example, the method further includes a step (2942) of retraining the trained neural network to reduce sensitivity to errors in weights or resistance values ​​that cause the equivalent analog network to produce a different output compared to the trained neural network. In other words, some implementations include additional training on the already trained neural network to reduce sensitivity to small random distribution weight errors. Quantization and resistance manufacturing generate small weight errors. Some implementations transform the network so that the resulting network is less sensitive to each specific weight value. In some implementations, this is done by adding a small relative random value to each signal of at least some of the layers during training (e.g., similar to a dropout layer).

[0474] Next, referring to FIG. 29f, some implementations include a step of reducing the range of the weight distribution. Some implementations include a step (2944) of retraining the trained neural network to minimize any weight of any layer that is greater than a predetermined threshold than the average absolute weight for that layer. Some implementations perform this step through retraining. An example of a penalty function includes a sum for all layers (e.g., A * max(abs(w)) / mean(abs(w)), where the maximum and mean are calculated for the layers. Another example includes a magnitude greater or greater. In some implementations, this function affects weight quantization and network weight sensitivity. For example, relatively small changes in weights due to quantization can cause high output errors. Exemplary techniques include introducing some penalty function that penalizes when such weight outcasts occur during training.

[0475] Exemplary optimization method for analog hardware implementation of a trained neural network

[0476] FIGS. 30a through 30m illustrate a flowchart of a method (3000) for a hardware implementation (3002) of a neural network according to some implementations and hardware design constraints. The method is performed (3004) in a computing device (200) having one or more processors (202) and a memory (214) for storing one or more programs configured for execution by one or more processors (202) (e.g., using an analog neural network optimization module (246)).

[0477] The method includes the step (3006) of obtaining a neural network topology (e.g., topology (224)) and weights (e.g., weights (222)) of a trained neural network (e.g., networks (220)).

[0478] The method also includes the step (3008) of converting the neural network topology into an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (e.g., using a neural network conversion module (226)). Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.

[0479] Next, referring to FIG. 30l, in some implementations, the method further includes the step of pruning the trained neural network. In some implementations, the method further includes the step (3052) of pruning the trained neural network to update the neural network topology and the weights of the trained neural network before transforming the neural network topology, using pruning techniques for the neural network so that the equivalent analog network contains fewer than a predetermined number of analog components. In some implementations, pruning is performed iteratively (3054) considering the level of agreement or accuracy of the output between the trained neural network and the equivalent analog network.

[0480] Next, referring to FIG. 30m, in some implementations, the method further includes a step (3056) of performing network knowledge extraction before converting the neural network topology into an equivalent analog network. Knowledge extraction differs from probabilistic / learning methods such as pruning, but is more deterministic than pruning. In some implementations, knowledge extraction is performed independently of the pruning step. In some implementations, before converting the neural network topology into an equivalent analog network, connection weights are adjusted according to a predetermined optimality criterion (e.g., preference for zero weights or a specific range of weights over other weights) through knowledge extraction methods by deriving a causal relationship between the inputs and outputs of hidden neurons. Conceptually, in a single neuron or set of neurons, in a specific data set, there may be a causal relationship between the inputs and outputs that causes the weights to be readjusted in such a way that (1) a new set of weights produces the same network output, and (2) the new set of weights is easier to implement as resistors (e.g., more uniformly distributed values, more zero values, or no connections). For example, if the output of some neurons is always 1 for some datasets, some implementations remove the output connection of this neuron (and the entire neuron) and instead adjust the bias weights of the neurons following it. In this way, pruning requires retraining after removing neurons, and since learning is probabilistic but knowledge extraction is deterministic, the knowledge extraction step differs from pruning.

[0481] Referring again to FIG. 30a, the method also includes the step (3010) of calculating a weight matrix for an equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.

[0482] Next, referring to FIG. 30j, in some implementations, the method further includes the step of removing or transforming neurons based on bias values. In some implementations, the method further includes, for each analog neuron of an equivalent analog network, (i) the step of calculating a bias value for each analog neuron based on the weights of the trained neural network while calculating the weight matrix (3044); (ii) the step of removing each analog neuron from the equivalent analog network based on the determination that each bias value exceeds a predetermined maximum bias threshold (3046); and (iii) the step of replacing each analog neuron with a linear junction of the equivalent analog network based on the determination that each bias value is less than a predetermined minimum bias threshold (3048).

[0483] Next, referring to FIG. 30k, in some implementations, the method further includes the step of minimizing the number of neurons or compressing the network. In some implementations, the method further includes the step (3050) of reducing the number of neurons in an equivalent analog network by increasing the number of connections (inputs and outputs) from one or more analog neurons of the equivalent analog network before generating a weight matrix.

[0484] Referring again to FIG. 30a, the method also includes the step (3012) of generating a resistance matrix for a weight matrix. Each element of the resistance matrix corresponds to each weight of the weight matrix.

[0485] The method also includes the step (3014) of pruning an equivalent analog network to reduce the number of multiple operational amplifiers or multiple resistors based on the resistance matrix in order to obtain an optimized analog network of analog components.

[0486] Next, referring to FIG. 30b, in some implementations, the method includes the step of replacing minor resistors with conductors. In some implementations, the step of pruning an equivalent analog network includes the step (3016) of replacing resistors corresponding to one or more elements of a resistance matrix whose resistance values ​​are less than a predetermined minimum threshold resistance value with conductors.

[0487] Next, referring to FIG. 30c, in some implementations, the method further includes the step of removing connections having very high resistance. In some implementations, the step of pruning the equivalent analog network includes the step (3018) of removing one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that exceed a predetermined maximum threshold resistance value.

[0488] Next, referring to FIG. 30d, in some implementations, the step of pruning an equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of a weight matrix that are approximately zero (3020). In some implementations, the step of pruning an equivalent analog network further includes removing one or more analog neurons of the equivalent analog network that have no input connections (3022).

[0489] Next, referring to FIG. 30e, in some implementations, the method includes the step of removing unimportant neurons. In some implementations, the step of pruning an equivalent analog network includes (i) the step of ranking the analog neurons of the equivalent analog network based on detecting the use of the analog neurons when performing computations on one or more data sets (3024). For example, a training data set used to train the trained neural network; a typical data set; a data set developed for the pruning procedure. Some implementations perform ranking the neurons for pruning based on the frequency of use of a given neuron or neuron block when subjected to the training data set. For example, (a) when using the test data set, if there is no signal in a given neuron—meaning that this neuron or neuron block is not used and is pruned; (b) if the frequency of use of the neuron is very low, the neuron is pruned without a significant loss of accuracy; and (c) if the neuron is always used, the neuron may be pruned; (ii) a step of selecting one or more analog neurons of an equivalent analog network based on ranking (3026); and (iii) a step of removing one or more analog neurons from an equivalent analog network (3028).

[0490] Next, referring to FIG. 30f, in some implementations, the step of detecting the use of analog neurons includes (i) building a model of an equivalent analog network using modeling software (e.g., SPICe or similar software) (3030); and (ii) generating calculations for one or more data sets by measuring the propagation of analog signals (currents) using the model (removing blocks where the signal does not propagate when using special training sets) (3032).

[0491] Next, referring to FIG. 30g, in some implementations, the step of detecting the use of analog neurons includes (i) building a model of an equivalent analog network using modeling software (e.g., SPICe or similar software) (3034); and (ii) measuring the output signals of the model (currents or voltages) (e.g., signals of the outputs of some blocks of the SPICe model or actual circuits or amplifiers, and removing the region where the output signal for the training set is always 0 volts) using the model to generate calculations for one or more data sets (3036).

[0492] Next, referring to FIG. 30h, in some implementations, the step of detecting the use of analog neurons includes (i) building a model of an equivalent analog network using modeling software (e.g., SPICe or similar software) (3038); and (ii) measuring the power consumed by the analog neurons (e.g., power consumed by specific neurons or neuron blocks represented by the SPICe model or the operational amplifier of the actual circuit, and removing neurons or neuron blocks for which no power was consumed) using a model for generating calculations for one or more data sets (3040).

[0493] Next, referring to FIG. 30i, in some implementations, the method further includes the step (3042) of recalculating a weight matrix for an equivalent analog network and updating a resistance matrix based on the recalculated weight matrix after the step of pruning an equivalent analog network and before generating one or more lithographic masks for manufacturing a circuit implementing the equivalent analog network.

[0494] Exemplary analog neuromorphic integrated circuit and manufacturing method

[0495] An exemplary method for manufacturing an analog integrated circuit for a neural network

[0496] FIGS. 31a through 31q illustrate a flowchart of a method (3100) for manufacturing an integrated circuit (3102) comprising an analog network of analog components, according to some implementations. The method is performed (e.g., using an IC manufacturing module (258)) in a computing device (200) having one or more processors (202) and a memory (214) for storing one or more programs configured for execution by one or more processors (202). The method includes the step (3104) of obtaining a neural network topology and weights of a trained neural network.

[0497] The method also includes the step (3106) of converting a neural network topology into an equivalent analog network of analog components (e.g., using a neural network conversion module (226)) comprising a plurality of operational amplifiers and a plurality of resistors (in the case of a recurrent neural network, also using signal delay lines, multipliers, Tanh analog blocks, and sigmoid analog blocks). Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a first analog neuron and a second analog neuron.

[0498] The method also includes the step (3108) of calculating a weight matrix for an equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.

[0499] The method also includes the step (3110) of generating a resistance matrix for a weight matrix. Each element of the resistance matrix corresponds to each weight of the weight matrix.

[0500] The method also includes the step (3112) of generating one or more lithographic masks for manufacturing a circuit that implements an equivalent analog network of analog components based on a resistance matrix (e.g., generating masks (250 and / or 252) using a mask generation module (248)), and the step (3114) of manufacturing a circuit (e.g., ICs (262)) based on one or more lithographic masks using a lithographic process.

[0501] Next, referring to FIG. 31b, in some implementations, the integrated circuit further includes one or more digital-to-analog converters (3116) (e.g., DAC converters (260)) configured to generate an analog input to an equivalent analog network of analog components based on one or more digital signals (e.g., signals from one or more CCD / CMOS image sensors).

[0502] Next, referring to FIG. 31c, in some implementations, the integrated circuit further includes an analog signal sampling module (3118) configured to process one-dimensional or two-dimensional analog inputs at a sampling frequency based on the inference number of the integrated circuit (the inference number for the IC is determined by product specifications - we know the sampling rate of the neural network operation and the exact task the chip is to solve).

[0503] Next, referring to FIG. 31d, in some implementations, the integrated circuit further includes a voltage converter module (3120) that scales down or scales up analog signals to match the operating range of a plurality of operational amplifiers.

[0504] Next, referring to FIG. 31e, in some implementations, the integrated circuit further includes a tact signal processing module (3122) configured to process one or more frames acquired from a CCD camera.

[0505] Next, referring to FIG. 31f, in some implementations, the trained neural network is a long short-term memory (LSTM) network, and the integrated circuit further includes one or more clock modules that synchronize signal takts and allow time series processing.

[0506] Next, referring to FIG. 31g, in some implementations, the integrated circuit further includes one or more analog-to-digital converters (3126) (e.g., ADC converters (260)) configured to generate a digital signal based on the output of an equivalent analog network of analog components.

[0507] Next, referring to FIG. 31h, in some implementations, the integrated circuit includes one or more signal processing modules (3128) configured to process one-dimensional or two-dimensional analog signals obtained from edge applications.

[0508] Next, referring to FIG. 31i, the trained neural network is trained (3130) using training datasets containing signals from a gas sensor array (e.g., 2 to 25 sensors) for different gas mixtures to selectively detect different gases in a gas mixture containing a predetermined amount of gases to be detected (in other words, the operation of the trained chip is used to individually determine each of the gases known to the neural network in the gas mixture, even though there are other gases in the mixture). In some implementations, the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed to detect three binary gas components based on measurements by 16 gas sensors, and includes 16 sensor-specific 1D convolution blocks, 3 shared or common 1D convolution blocks, and 3 dense layers (3132). In some implementations, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) delay blocks for generating delays of any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections (3134).

[0509] Next, referring to FIG. 31j, a trained neural network is trained (3136) using training datasets containing thermal aging time series data for different MOSFETs (e.g., NASA MOSFET datasets containing thermal aging time series for 42 different MOSFETs; the data is sampled every 400 ms and is typically several hours' worth of data for each device) to predict the remaining effective life (RUL) of a MOSFET device. In some implementations, the neural network topology includes four LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively (3138). In some implementations, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) 3,000 to 3,200 analog neurons (e.g., 3,137 analog neurons), and (v) 123,000 to 124,000 connections (e.g., 123,200 connections) (3140).

[0510] Next, referring to FIG. 31k, a trained neural network is trained (3142) using training datasets containing time-series data including discharge and temperature data during continuous use of various commercially available lithium-ion batteries to monitor the state of health (SOH) and state of charge (SOC) of lithium-ion batteries to be used in a battery management system (BMS) (e.g., NASA battery usage dataset; this dataset represents continuous usage data of six commercially available lithium-ion batteries; network operation is based on analysis of the battery's discharge curve). In some implementations, the neural network topology includes an input layer, two LSTM layers with 64 neurons in each layer, and a subsequent output dense layer with two neurons to generate SOC and SOH values ​​(3144). The equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) 1,200 to 1,300 analog neurons (e.g., 1,271 analog neurons), and (v) 51,000 to 52,000 connections (e.g., 51,776 connections) (3146).

[0511] Next, referring to FIG. 31, a trained neural network is trained (3148) using training datasets (e.g., NASA battery usage dataset; this dataset represents continuous usage data of six commercially available lithium-ion batteries; network operation is based on analysis of the battery's discharge curve) to monitor the state of life (SOH) and state of charge (SOC) of lithium-ion batteries to be used in a battery management system (BMS). In some implementations, the neural network topology includes an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron (3150). In some implementations, the equivalent analog network comprises (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) 200 to 300 analog neurons (e.g., 201 analog neurons), and (v) 2200 to 2400 connections (e.g., 2300 connections) (3152).

[0512] Next, referring to FIG. 31m, the trained neural network is trained (3154) using training datasets containing speech commands (e.g., Google Speech Command dataset) to identify speech commands (e.g., "Yes," "No," "Up," "Down," "Left," "Right," "On," "Off," "Stop," "Move"). In some implementations, the neural network topology is a Depthwise Separable Convolutional Neural Network (DS-CNN) with one neuron (3156). In some implementations, the equivalent analog network includes (i) up to 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections (3158).

[0513] Next, referring to FIG. 31n, the trained neural network is trained (3160) using training datasets including photoplethysmography (PPG) data, accelerometer data, temperature data and skin electrical response signal data and reference heart rate data obtained from an ECG sensor (e.g., PPG data (CHECK LICENSE) of the PPG-Dalia dataset) for various individuals performing various physical activities for a predetermined time. Data is collected for 15 individuals performing various physical activities for 1 to 4 hours each. Wrist sensor data includes PPG, a 3-axis accelerometer, temperature and skin electrical response signals sampled from 4 to 64 Hz, and reference heart rate data obtained from an ECG sensor sampled at approximately 2 Hz. The raw data was split into sequences of 1,000 time steps (approx. 15 seconds) while shifting by 500 time steps to obtain a total of 16,541 samples. The dataset was split into 13,233 training samples and 3,308 test samples to determine heart rate during physical exercise (e.g., jogging, fitness exercise, stair climbing) based on PPG sensor data and 3-axis accelerometer data. The neural network topology includes two Conv1D layers with 20 kernels and 16 filters, each performing time-series convolution, two LSTM layers with 16 neurons, and two dense layers with 16 neurons and 1 neuron (3162). In some implementations, the equivalent analog network includes (i) delay blocks for generating any number of time steps, (ii) up to 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) 700 to 800 analog neurons (e.g., 713 analog neurons), and (vi) 12,000 to 12,500 connections (e.g., 12,072 connections) (3164).

[0514] Next, referring to FIG. 31o, the trained neural network is trained (3166) to classify various objects (e.g., people, cars, cyclists, scooters) based on pulse Doppler radar signals (removing clutter and providing noise to the Doppler radar signals), and the neural network topology includes a multi-scale LSTM neural network (3168).

[0515] Next, referring to FIG. 31p, the trained neural network is trained to perform human activity type recognition (e.g., walking, running, sitting, climbing stairs, exercise, activity tracking) based on inertial sensor data (e.g., 3-axis accelerometer, magnetometer, or gyroscope data from a fitness tracker, smartwatch, or mobile phone; 3-axis accelerometer data as input, sampled at a frequency of up to 96 Hz) (3170). The network was trained on three publicly available datasets, which present these activities such as "opening and closing a dishwasher," "drinking while standing," "closing a door with the left hand," "jogging," "walking," and "climbing stairs." In some implementations, the neural network topology consists of a convolutional layer, each with 12 filters and a kernel dimension of 64, followed by a max pooling layer, each with 1024 neurons and N neurons, and two common dense layers (where, It includes three channel-wise convolution networks (3172) including N (where N is the number of classes). In some implementations, the equivalent analog network includes (i) delay blocks for generating any number of time steps, (ii) up to 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) a signal limit of 5, (v) 10 layers, (vi) 1,200 to 1,300 analog neurons (e.g., 1,296 analog neurons), and (vi) 20,000 to 21,000 connections (e.g., 20,022 connections) (3174).

[0516] Next, referring to FIG. 31q, the trained neural network is further trained (3176) to detect abnormal patterns of human activity based on accelerometer data merged with heart rate data using convolution operations (to detect pre-stroke or pre-heart attack states or signals in the case of sudden abnormal patterns due to injury or malfunction for medical reasons such as epilepsy).

[0517] Some implementations include components not integrated into the chip (i.e., these are external elements connected to the chip) selected from a group consisting of voice recognition, video signal processing, image detection, temperature detection, pressure detection, radar processing, LIDAR processing, battery management, MOSFET circuit current and voltage, accelerometer, gyroscope, magnetic sensor, heart rate sensor, gas sensor, volume sensor, liquid level sensor, GPS satellite signal, human body conductivity sensor, gas flow sensor, concentration sensor, pH meter, and IR vision sensor.

[0518] Examples of analog neuromorphic integrated circuits manufactured according to the processes described above are provided in the following section, depending on some implementations.

[0519] Exemplary analog pneumotropic IC for selective gas detection

[0520] In some embodiments, the neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC is based on a deep convolutional neural network trained to selectively detect various gases in a gas mixture containing an amount of gas to be detected. The deep convolutional neural network is trained using a training dataset containing signal arrays of gas sensor arrays (e.g., 2 to 25 sensors) that respond to various gas mixtures. The integrated circuit (or the chip manufactured according to the techniques described herein) can be used to determine one or more known gases in a gas mixture despite the presence of other gases in the mixture.

[0521] In some implementations, the trained neural network is a multi-label 1D-DCNN network used for mixed gas classification. In some implementations, the network is designed to detect three binary gas components based on measurements from 16 gas sensors. In some implementations, the 1D-DCNN includes sensor-specific 1D convolution blocks (16 of these blocks), three common 1D convolution blocks, and three dense layers. In some implementations, the performance of the 1D-DCNN network for this task is 96.3%.

[0522] In some implementations, the original network is transformed into T with the following parameters: maximum input and output connections per neuron = 100; delay blocks can generate delays of any number of time steps; and there is a signal limit of 5.

[0523] In some implementations, the resulting T-network has properties such as 15 layers, about 100,000 analog neurons, and about 4,900,000 connections.

[0524] Exemplary Analog Neuromorphic IC for MOSFET Failure Prediction

[0525] MOSFET on-resistance degradation due to thermal stress is a well-known and serious problem in the field of power electronics. In real-world applications, the temperature of MOSFET devices fluctuates over short periods. This temperature sweep leads to thermal degradation of the device, which can result in exponential degradation. This effect is typically studied through power cycling, which generates temperature gradients that cause MOSFET performance degradation.

[0526] In some implementations, neuromorphic ICs are manufactured according to the processes described above. Neuromorphic ICs are based on the network discussed in the article titled "Real-time Deep Learning at the Edge for Scalable Reliability Modeling of SI-MOSFET Power Electronics Converters" to predict the remaining useful life (RUL) of MOSFET devices. The neural network can be used to determine the remaining useful life (RUL) of the device with an accuracy of over 80%.

[0527] In some implementations, the network is trained on the NASA MOSFET dataset, which contains thermal aging time series for 42 different MOSFETs. The data is sampled every 400 ms and typically contains several hours of data for each device. The network includes four LSTM layers, each consisting of 64 neurons, followed by two dense layers consisting of 64 neurons and 1 neuron.

[0528] In some implementations, the network is T-transformed with the following parameters: the maximum input and output connections per neuron are 100; the signal limit is 5, and the resulting T-network had the following properties: 18 layers, approximately 3,000 neurons (e.g., 137 neurons), and approximately 120,000 connections (e.g., 123,200 connections).

[0529] Exemplary Analog Neuromorphic IC for Lithium-Ion Battery Life and SoC Monitoring

[0530] In some implementations, neuromorphic ICs are manufactured according to the processes described above. Neuromorphic ICs can be used for predictive analysis of lithium-ion batteries for use in battery management systems (BMS). BMS devices typically provide these functions, such as overcharge and over-discharge protection, State of Health (SOH) and State of Charge (SOC) monitoring, and load balancing for multiple cells. SOH and SOC monitoring generally requires digital data processors, which add device cost and consume power. In some implementations, integrated circuits are used to obtain accurate SOC and SOH data without implementing digital data processors in the device. In some implementations, integrated circuits determine SOC with over 99% accuracy and SOH with over 98% accuracy.

[0531] In some implementations, network operation is based on battery discharge curve and temperature analysis, and / or data is represented as a time series. Some implementations use data from the NASA battery usage dataset. The dataset represents continuous usage data of six commercially available lithium-ion batteries. In some implementations, the network includes an input layer, two LSTM layers each consisting of 64 neurons, and an output dense layer consisting of two neurons (SOC and SOH values).

[0532] In some implementations, the network is T-transformed with the following parameters: maximum input and output connections per neuron = 100, and a signal limit of 5. In some implementations, the resulting T-network includes properties such as 9 layers, approximately 1,200 neurons (e.g., 1,271 neurons), and approximately 50,000 connections (e.g., 51,776 connections). In some implementations, network operation is based on battery discharge curve and temperature analysis. The network is trained using IndRnn, a network disclosed in the paper titled "State-of-Health Estimation of Li-ion Batteries in Electric Vehicle Using IndRNN under VariableLoad Condition," which was designed to process data from the NASA battery usage dataset. The dataset represents continuous usage data for six commercially available lithium-ion batteries. The IndRnn network includes an input layer with 18 neurons, a simple recurrent layer consisting of 100 neurons, and a dense layer consisting of 1 neuron.

[0533] In some implementations, the IndRnn network is transformed into T with the following parameters: maximum input and output connections per neuron = 100, and the signal limit is 5. In some implementations, the resulting T-network has properties such as 4 layers, about 200 neurons (e.g., 201 neurons), and about 2,000 connections (e.g., 2,300 connections). Some implementations output only the SOH with an estimation error of 1.3%. In some implementations, the SOC is obtained in a manner similar to how the SOH is obtained.

[0534] Exemplary Analog Neumorphic IC for Keyword Spotting

[0535] In some implementations, neuromorphic ICs are manufactured according to the processes described above. Neuromorphic ICs can be used for keyword spotting.

[0536] The input network is a neural network with 2D convolution and 2D depth-wise convolution layers with an input audio mel-spectrogram of size 49 x 10. In some implementations, the network includes 5 convolution layers, 4 depth-wise convolution layers, an average pooling layer, and a final dense layer.

[0537] In some implementations, networks are pre-trained to recognize 10 short speech keywords ("yes", "no", "up", "down", "left", "right", "on", "off", "stop", "move") from the Google Speech Command dataset with 94.4% recognition accuracy.

[0538] In some implementations, the integrated circuit is manufactured based on a Depthwise Separable Convolutional Neural Network (DS-CNN) for speech command identification. In some implementations, the original DS-CNN network is T-transformed with the following parameters: maximum input and output connections per neuron = 100, and a signal limit of 5. In some implementations, the resulting T-network has properties such as 13 layers, approximately 72,000 neurons, and approximately 2.6 million connections.

[0539] Exemplary DS-CNN Keyword Spotting Network

[0540] In one example, the keyword spotting network is converted into a T-network depending on the implementation. The network is a neural network composed of 2D convolutional and 2D depth-wise convolutional layers with a 49 x 10 input audio spectrogram. The network consists of 5 convolutional layers, 4 depth-wise convolutional layers, an average pooling layer, and a final dense layer. The network is pre-trained to recognize 10 short speech keywords ("yes", "no", "up", "down", "left", "right", "on", "off", "stop", "move") from the Google Speech Commands dataset https: / / ai.googleblog.com / 2017 / 08 / launching-speech-commands-dataset.html. There are 2 additional classes corresponding to 'silence' and 'unknown'. The network output is a softmax of length 12.

[0541] The trained neural network (input for the transformation) has a recognition accuracy of 94.4% depending on some implementations. In the neural network topology, each convolutional layer is followed by a BatchNorm layer and a ReLU layer, with no limit on ReLU activation, and involves approximately 2.5 million additional multiplication operations.

[0542] After transformation, the transformed analog network was tested with a test set of 1,000 samples (100 speech commands each). All test samples are also used as test samples from the original dataset. The original DS-CNN network provided a recognition error of nearly 5.7% on this test set. The network was transformed into a T-network of trivial neurons. Since the BatchNormalization layers in 'Test' mode produce a simple linear signal transformation, they can be interpreted as weight multipliers plus some additional bias. Convolution, mean pooling, and dense layers are T-transformed very simply. The softmax activation function was not implemented in the T-network and was applied separately to the T-network output.

[0543] The resulting T-network has 12 layers, including an input layer, about 72,000 neurons, and about 2.5 million connections.

[0544] FIGS. 26a through 26k illustrate exemplary histograms (2600) of absolute weights for each of layers 1 through 11 according to some implementations. A weight distribution histogram (for absolute weights) was calculated for each layer. The dashed lines in the chart correspond to the average absolute weight values ​​for each layer. After the transformation (i.e., T transformation), the average output absolute error (calculated for the test set) of the transformed network versus the original is calculated to be 4.1e-9.

[0545] Depending on some implementations, various examples for setting network limits for the transformed network are described herein. In the case of signal limits, the ReLU activation used in the network is unlimited, and some implementations use signal limits at each layer. This may potentially affect mathematical equivalence. To this end, some implementations use a signal limit of 5 at all layers corresponding to a power supply voltage of 5 with respect to the input signal range.

[0546] To quantize the weights, some implementations use a nominal set of 30 resistors [0.001, 0.003, 0.01, 0.03, 0.1, 0.324, 0.353, 0.436, 0.508, 0.542, 0.544, 0.596, 0.73, 0.767, 0.914, 0.985, 0.989, 1.043, 1.101, 1.149, 1.157, 1.253, 1.329, 1.432, 1.501, 1.597, 1.896, 2.233, 2.582, 2.844].

[0547] Some implementations select R- and R+ values ​​individually for each layer (see description above). For each layer, some implementations select the value that delivers the most accurate weight. In some implementations, all weights (including biases) of the T-network are subsequently quantized (e.g., set to the nearest value achievable with the input or selected resistors).

[0548] Some implementations transform the output layer as follows. The output layer is a dense layer without ReLU activation. The layer has a softmax activation that is not implemented in the T-transform and is left as a digital part depending on some implementations. Some implementations do not perform additional transformations.

[0549] Exemplary analog pneumotropic IC for acquiring heart rate

[0550] PPG is an optically acquired volumetric pulse wave that can be used to detect changes in blood volume within the microvascular layer of tissues. PPG is often acquired using a pulse oximeter, which illuminates the skin and measures changes in light absorption. PPG is frequently processed to determine heart rate in devices such as fitness trackers. Deriving the heart rate (HR) from the PPG signal is an essential task in edge device computing. PPG data obtained from a wrist-mounted device generally yields a reliable heart rate only when the device is stable. When a person engages in physical exercise, acquiring the heart rate from PPG data produces poor results unless combined with inertial sensor data.

[0551] In some implementations, an integrated circuit based on a combination of convolutional neural networks and LSTM layers can be used to accurately determine the heart rate based on data from a photoplethysmography (PPG) sensor and a 3-axis accelerometer. The integrated circuit can suppress motion artifacts in the PPG data and be used to determine the heart rate with an accuracy exceeding 90% during physical exercises such as jogging, fitness exercises, and stair climbing.

[0552] In some implementations, the input network is trained with PPG data from the PPG-Dalia dataset. Data is collected on 15 individuals performing various physical activities for predetermined periods (e.g., 1 to 4 hours each). The training data, which includes wrist sensor data, comprises PPG, a 3-axis accelerometer, temperature and skin electrical response signals sampled from 4 to 64 Hz, and reference heart rate data obtained from an ECG sensor sampled at approximately 2 Hz. The raw data was split into sequences of 1,000 time steps (approximately 15 seconds) while shifting by 500 time steps to obtain a total of 16,541 samples. The dataset was divided into 13,233 training samples and 3,308 test samples.

[0553] In some implementations, the input network includes two Conv1D layers with 16 filters performing time-series convolution, two LSTM layers each consisting of 16 neurons, and two dense layers consisting of 16 and 1 neuron. In some implementations, the network generates an MSE error of less than 6 bits per minute on the test set.

[0554] In some implementations, the network is T-transformed with the following parameters: delay blocks can generate delays of any number of time steps, a maximum of 100 input and output connections per neuron, and a signal limit of 5. In some implementations, the resulting T-network has properties such as 15 layers, about 700 neurons (e.g., 713 neurons), and about 12,000 connections (e.g., 12,072 connections).

[0555] Exemplary PPG Data Processing by T-Transformed LSTM Network

[0556] As described above, for recurrent neurons, some implementations use signal delay blocks added to each recurrent connection of GRU and LSTM neurons. In some implementations, the delay block has an external cycle timer (e.g., a digital timer) that activates the delay block for a constant time dt. This activation produces an output x(t-dt), where x(t) is the input signal of the delay block. This activation frequency may correspond, for example, to the network input signal frequency (e.g., the output frequency of analog sensors processed by the T-transform network). Generally, all delay blocks are activated simultaneously with the same activation signal. Some blocks may be activated simultaneously at one frequency, while others may be activated at different frequencies. In some implementations, these frequencies have a common multiplier, and the signals are synchronized. In some implementations, multiple delay blocks are used for a single signal to create additional time shifts. Examples of delay blocks are described above with reference to Fig. 13b, which illustrates two exemplary delay blocks according to some implementations.

[0557] Networks for processing PPG data use one or more LSTM neurons depending on some implementations. Examples of LSTM neuron implementations are described above with reference to FIG. 13a depending on some implementations.

[0558] The network also uses Conv1D, which is a convolution performed over time coordinates. Examples of Conv1D implementations are described above with reference to Figures 15a and 15b, depending on some implementations.

[0559] According to some implementations, details of PPG data are described herein. PPG is an optically acquired volumetric pulse wave that can be used to detect changes in blood volume in the microvascular layer of tissue. PPG is often acquired using a pulse oximeter that illuminates the skin and measures changes in light absorption. PPG is often processed to determine heart rate in devices such as fitness trackers. Deriving heart rate (HR) from a PPG signal is an essential task in edge device computing.

[0560] Some implementations use PPG data from the Capnobase PPG dataset. The data includes raw PPG signals for 42 individuals, each 8 minutes long, sampling 300 samples per second, and reference heart rate data obtained from an ECG sensor sampling about 1 sample per second. For training and evaluation, some implementations split the original data into 6,000 time step sequences with 1,000 time step shifts to obtain a total of 5,838 sample sets.

[0561] In some implementations, input-trained neural network (NN)-based methods allow for an accuracy of 1 to 3% when obtaining heart rate (HR) from PPG data.

[0562] This section describes a relatively simple neural network to demonstrate how T-transforms and analog processing can handle this task. This description is provided as an example, depending on some implementations.

[0563] In some implementations, the dataset is split into 4,670 training samples and 1,168 test samples. The network includes one Conv1D layer with 16 filters and 20 kernels, two LSTM layers with 24 neurons each, and two dense layers (containing 24 and 1 neuron, respectively). In some implementations, after training this network for 200 epochs, the test accuracy was found to be 2.1%.

[0564] In some implementations, the input network was T-transformed with the following parameters: delay blocks with durations of 1, 5, and 10 time steps, and the following properties: 17 layers, 15,448 connections, and 329 neurons (OP3 neurons and multiplier blocks, delay blocks are not counted).

[0565] Exemplary analog neuromorphic integrated circuit for object recognition based on pulsed Doppler radar signals

[0566] In some implementations, integrated circuits are manufactured based on multi-scale LSTM neural networks that can be used to classify objects based on pulsed Doppler radar signals. The IC can be used to classify various objects such as people, cars, cyclists, and scooters based on Doppler radar signals, remove clutter, and introduce noise into the Doppler radar signals. In some implementations, the object classification accuracy using multi-scale LSTM networks has exceeded 90%.

[0567] Exemplary analog neuromorphic IC for human activity type recognition based on inertial sensor data

[0568] In some implementations, a neuromorphic integrated circuit is manufactured and can be used for human activity type recognition based on a multi-channel convolutional neural network with input signals from a 3-axis accelerometer and input signals from a gyroscope and / or magnetometer of a fitness tracker, smartwatch, or mobile phone. The multi-channel convolutional neural network can be used to distinguish various types of human activities such as walking, running, sitting, climbing stairs, and exercising, and can be used for activity tracking. The IC can be used to detect abnormal patterns of human activity based on heart rate data and convolutional merged accelerometer data. Depending on some implementations, this IC can detect or signal a pre-stroke or pre-cardiac attack state in the case of sudden abnormal patterns caused by injury or malfunction due to medical reasons such as epilepsy.

[0569] In some implementations, the IC is based on the channel-wise 1D convolutional network discussed in the article "Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors". In some implementations, the network accepts 3-axis accelerometer data sampled at a frequency of up to 96 Hz as input. In some implementations, the network is trained on three publicly available datasets, which are three different publicly available datasets presenting activities such as "opening and closing a dishwasher", "drinking while standing", "closing a door with the left hand", "jogging", "walking", and "climbing stairs". In some implementations, the network includes a convolutional layer consisting of 12 filters and 64 kernels, each followed by a max pooling layer (4) and two common dense layers of 1024 each, and N neurons, where N is the number of classes. In some implementations, activity classification was performed with a low error rate (e.g., 3.12% error).

[0570] In some implementations, the network is T-transformed with the following parameters: delay blocks can generate delays of an arbitrary number of time steps, a maximum of 100 input and output connections per neuron, 10 output layers of neurons, and a signal limit of 5. In some implementations, the resulting T-network has properties such as 10 layers, approximately 1,200 neurons (e.g., 1,296 neurons), and approximately 20,000 connections (e.g., 20,022 connections).

[0571] Exemplary transformation of a modular network structure for library creation

[0572] According to some implementations, a modular structure of a transformed neural network is described herein. Each module of the modular neural network is obtained after transforming one or more trained neural networks (in whole or in part). In some implementations, one or more trained neural networks are subdivided into parts and then transformed into equivalent analog networks. The modular structure is typical for some of the neural networks currently in use, and the modular subdivision of neural networks corresponds to the trend in neural network development. Each module may have any number of inputs or connections of input neurons to output neurons of connected modules, and any number of outputs connected to the input layers of subsequent modules. In some implementations, a preliminary library of transformed modules (or a seed list thereof) is developed, including lithography masks for manufacturing each module. The final chip design is obtained from a combination (or connection) of the preliminaryly developed modules. Some implementations perform communication between modules. In some implementations, neurons and connections within the modules are transformed into a chip design using ready-made module design templates. This significantly simplifies chip manufacturing, which is completed simply by connecting the modules.

[0573] Some implementations generate libraries of off-the-shelf T-transformed neural networks and / or T-transformed modules. For example, layers of a CNN network are modular building blocks, and LSTM chains are another building block, and so on. Large neural networks (NNs) also have modular structures (e.g., LSTM modules and CNN modules). In some implementations, neural network libraries are more than byproducts of exemplary processes and can be sold independently. For example, a third party can manufacture a neura...

Claims

Claim 1 A method for an analog hardware implementation of a neural network for speech intelligibility, performed by one or more processors, comprising: a step of obtaining a neural network topology and weights of a trained neural network — said neural network is trained for speech intelligibility enhancement using a dataset of noisy speech signals and corresponding clean speech signals —; a step of converting said neural network topology into an equivalent analog network of analog components, including selecting a plurality of individual electronic analog components — said analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of said equivalent analog network, and each resistor represents a connection between two analog neurons —; a step of calculating a weight matrix for said equivalent analog network based on said weights of said neural network — said element of said weight matrix represents a respective connection between a pair of individual electronic analog components of said equivalent analog network —; A method comprising the step of generating a schematic model for implementing the equivalent analog network as an electronic circuit for improving speech intelligibility based on the weight matrix, wherein the step of generating the schematic model comprises: generating a resistance matrix from the weight matrix ― each element of the resistance matrix represents (i) a respective fixed resistance value and (ii) corresponds to a respective weight of the weight matrix ―; and selecting component values ​​for the analog components, including performing gradient descent to identify possible fixed resistance values ​​for the plurality of resistors. Claim 2 A method according to claim 1, wherein the neural network topology comprises a Fourier transform layer configured to convert an input analog signal into input features and an inverse Fourier transform layer configured to generate an analog output signal having improved speech intelligibility compared with the input analog signal, and the schematic model comprises one or more first electronic analog components configured to modify signals according to the Fourier transform layer and one or more second electronic analog components configured to modify signals according to the inverse Fourier transform layer. Claim 3 The method according to claim 1, wherein the neural network topology comprises one or more of a convolution layer, a max pooling layer, and a densely connected layer. Claim 4 A method according to claim 1, wherein the neural network topology includes a convolution layer, and the step of transforming the neural network topology comprises: a step of defining a dependency relationship between each output of the convolution layer and a related input subset, wherein the related input subset is defined by the filter, kernel, padding, and stride parameters of the convolution layer; a step of defining each weight subset according to the dependency relationship of each output; and a step of configuring an analog neuron layer such that each analog neuron corresponds to each output of the convolution layer, (ii) each analog neuron is connected to a related input subset of a previous analog neuron layer of the equivalent analog network, and (iii) the input connection to each analog neuron is weighted according to each weight subset of the corresponding output of the convolution layer. Claim 5 A method according to claim 1, wherein the neural network topology includes a maximum pooling layer, and the step of transforming the neural network topology includes the step of generating a multilayer network of analog neurons for the maximum pooling layer. Claim 6 delete Claim 7 delete Claim 8 In claim 5, the step of generating a multilayer network of analog neurons comprises generating a 4-input scheme comprising four single-neuron models (SNMs) arranged in three layers, wherein each SNM is a schematic model having analog components representing a specific type of mathematical neuron in a schematic form. Claim 9 A method according to claim 8, further comprising the step of converting the maximum pooling layer into a computation tree, wherein each node of the computation tree is selected from a group comprising: a 2-input schema comprising two SNMs arranged in two layers, wherein the SNM of the last layer has a maximum of two inputs; a 3-input schema comprising three SNMs arranged in three layers, wherein the SNM of the last layer has a maximum of three inputs; and a 4-input schema comprising four SNMs arranged in three layers, wherein the SNM of the last layer has a maximum of four inputs. Claim 10 A method according to claim 9, further comprising the step of minimizing the number of layers of the calculation tree. Claim 11 A method according to claim 9, further comprising the step of prioritizing the use of the 4-input schema over the use of the 3-input schema and the 2-input schema. Claim 12 A method according to claim 9, further comprising: (i) defining an analog neuron of the last layer of the multilayer network to perform an activation function; and (ii) applying ReLU to all other neurons of the multilayer network. Claim 13 A method according to claim 1, wherein each layer of the trained neural network calculates each output based on each mathematical function, and the step of converting the neural network topology into the equivalent analog network of analog components comprises: a step of identifying one or more function blocks based on each mathematical function for each layer of the trained neural network, wherein each function block has a schematic implementation having a block output corresponding to the output of each mathematical function; and a step of generating a multilayer network of each analog neuron based on arranging the one or more function blocks, wherein each analog neuron implements each function of the one or more function blocks, and each analog neuron of the first layer of the multilayer network is connected to one or more analog neurons of the second layer of the multilayer network. Claim 14 In Clause 13, the above one or more function blocks are block outputs It includes a weighted summation block having, where ReLU is a rectified linear unit (ReLU) activation function or a ReLU with a threshold, and V i represents the i-th input, and w i A method in which represents the weight corresponding to the i-th input, bias represents the bias value, and Σ is the summation operator. Claim 15 In Clause 13, the above one or more function blocks are block outputs It includes a weighted summation block having, wherein ReLU_X is a rectified linear unit (ReLU) activation function or a similar activation function that limits the output signal to a positive X, and V i represents the i-th input, and w i A method in which represents the weight corresponding to the i-th input, bias represents the bias value, and Σ is the summation operator. Claim 16 In paragraph 3, the neural network topology comprises a convolutional layer having K inputs and L outputs; and the step of converting the neural network topology into the equivalent analog network comprises, according to one or more connection constraints based on analog integrated circuit (IC) design constraints, possible input connection degree (N i ) and output connection degree (N o The step of deriving ) includes; the step of transforming the convolutional layer comprises transforming the convolutional layer into K inputs, L outputs, and a maximum input connection degree P. i , and maximum output connection degree P o (Here, P i ≤N i and P o ≤N o A method comprising the step of decomposing into a single sparsely connected layer having ). Claim 17 delete Claim 18 A method according to claim 1, further comprising: generating an equivalent digital network of digital components for one or more output layers of the neural network topology; and connecting the outputs of one or more layers of the equivalent analog network to the equivalent digital network of digital components. Claim 19 A system for a hardware implementation of a neural network comprises: one or more processors; and memory, wherein the memory stores one or more programs configured to be executed by the one or more processors, and the one or more programs include instructions for obtaining the neural network topology and weights of a trained neural network; converting the neural network topology into an equivalent analog network of analog components — wherein the analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons —; calculating a weight matrix for the equivalent analog network based on the weights of the trained neural network — wherein each element of the weight matrix represents one or more connections between the analog components of the equivalent analog network — and generating a schematic model for implementing the equivalent analog network based on the weight matrix, wherein generating the schematic model includes: generating a resistor matrix from the weight matrix — wherein each element of the resistor matrix represents (i) a respective fixed resistor value and (ii) corresponds to each weight of the weight matrix —; A system comprising selecting component values ​​of analog components, including performing gradient descent to identify possible fixed resistance values ​​for the plurality of resistors. Claim 20 A voice transmission device comprises an integrated circuit for voice clarification, wherein the integrated circuit comprises: a step of obtaining a neural network topology and weights of a trained neural network — said neural network is trained for voice intelligibility enhancement using a dataset of noisy voice signals and corresponding clean voice signals —; a step of converting said neural network topology into an equivalent analog network of analog components, including selecting a plurality of individual electronic analog components — said analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of said equivalent analog network, and each resistor represents a connection between two analog neurons —; a step of calculating a weight matrix for said equivalent analog network based on said weights of said neural network — said element of said weight matrix represents a respective connection between a pair of individual electronic analog components of said equivalent analog network —; A step of generating a schematic model for implementing the equivalent analog network as an electronic circuit for improving speech intelligibility based on the above weight matrix — the step of generating the schematic model comprises: a step of generating a resistance matrix from the above weight matrix — each element of the resistance matrix represents (i) a respective fixed resistance value and (ii) corresponds to each weight of the above weight matrix —;A voice transmission device comprising an analog network of analog components fabricated by a method comprising: a step of selecting component values ​​for the analog components, including performing gradient descent to identify possible fixed resistance values ​​for the plurality of resistors; a step of fabricating the integrated circuit for speech intelligibility enhancement according to the schematic model using a lithographic process; wherein the integrated circuit is configured to receive a signal from the voice transmission device, enhance the speech intelligibility of the signal to generate an output signal, and provide the output signal to the voice transmission device. Claim 21 A voice transmission device according to claim 20, wherein the step of generating the schematic model comprises: a step of generating a resistance matrix for the weight matrix, wherein each element of the resistance matrix corresponds to a respective weight of the weight matrix; and a step of generating one or more lithographic masks for fabricating the circuit that implements the equivalent analog network of analog components based on the resistance matrix. Claim 22 In paragraph 20, the voice transmission device is a voice transmission device integrated into a mobile phone. Claim 23 A voice transmission device according to claim 22, wherein an input from the microphone of the mobile phone is input to the integrated circuit. Claim 24 A voice transmission device according to claim 22, wherein the output from the integrated circuit is input to the speaker of the mobile phone, and the integrated circuit is configured to improve the voice clarity of a signal received by the mobile phone before providing input to the speaker of the mobile phone. Claim 25 In paragraph 20, the integrated circuit is coupled to one or more other noise canceling devices, and the integrated circuit is configured to: receive signals including voice-containing signals and noise signals; and to enhance the voice-containing signals and suppress the noise signals, a voice transmission device. Claim 26 In paragraph 20, the voice transmission device, wherein the integrated circuit is coupled to one or more noise reduction software programs executed in the voice transmission device. Claim 27 A voice transmission device according to claim 20, wherein the integrated circuit is configured to perform voice intelligibility enhancement according to one or more of the number of voices detected in a signal received from the voice transmission device, the estimated proximity between a voice source and a signal receiver, the estimated relative distance between voice sources and a signal receiver, and the volume of voices detected in the signal. Claim 28 A voice transmission device according to claim 20, wherein the integrated circuit is configured to improve speech clarity for signals generated from one or more sources. Claim 29 delete

Images