A neural network optimization circuit

Abstract

The application relates to the technical field of circuits, and discloses a neural network optimization circuit. The application has the capabilities of fast reasoning training, low expansion cost, high parallelism, storage-computation integration and on-chip learning while improving the optimization speed of a neural network training device. Moreover, the application is completely realized by using analog circuits, does not need a computer to participate in calculating weight error values for multiple times, has no non-continuous signals such as switch signals and clock signals in the whole training process, and realizes fast convergence under the guarantee of strict mathematical theories.

Description

Technical Field

[0001] This invention relates to the field of circuit technology, and more specifically to a neural network optimization circuit. Background Technology

[0002] In recent years, with the development of AI technology, the demand for computing power has risen rapidly, and improving optimization speed has become an important development direction. As a result, neural networks are being used more and more widely.

[0003] In related technologies, the theory of Random Weight Change (RWC) is introduced into neural networks to fabricate analog circuit neural network chips. For example... Figure 1 The diagram shown is a circuit diagram illustrating the application of a random generator in a neural network. Figure 1 In this approach, a shift register generates random values ​​±δ. A random pulse +Vr or -Vr is applied to one side of the shift register, and the pulse propagates to the other side, changing the random weights by ±δ. This change is transmitted to each weight circuit through switched capacitors, which are equivalent to resistors. For a neural network using the RWC random generation mechanism, if the neural network error does not decrease, a weight shift direction is given (by inputting a random value to the shift register and controlling the shift with a clock signal from the computer). If the neural network error decreases, the computer does not provide a clock signal to the shift register, and the shift register maintains its original weight shift direction.

[0004] Therefore, the judgment process of this RWC method requires computer participation each time, and it also requires sequential AD conversion, error comparison, summation, and judgment of error changes to give a clock signal. This results in a complex process, slow speed, and inability to perform pure on-chip training. Furthermore, the neural network requires multiple computer interventions when training a single data point, which disrupts the continuity of the system. The clock-driven method also results in slow convergence speed. Summary of the Invention

[0005] In view of this, the present invention provides a neural network optimization circuit to solve the problems of complex calculation process, slow speed, poor system continuity and poor convergence speed of neural network optimization circuits using the RWC method.

[0006] According to a first aspect, embodiments of the present invention provide a neural network optimization circuit applied to a target neural network, comprising:

[0007] The first conversion unit is used to convert the target training data into the target DC level signal corresponding to the target training data;

[0008] The second conversion unit is used to convert the label data corresponding to the target training data into a labeling result level signal;

[0009] The first multiplier is used to multiply the target DC level signal corresponding to the target training data and the adder output signal to output the prediction result signal corresponding to the target training data.

[0010] The error calculation unit is used to calculate the MSE error signal between the predicted result signal and the labeled result signal;

[0011] The first differentiator is used to receive the MSE error signal and output the first differentiator output signal.

[0012] The second differentiator is used to receive multiple weight change signals from multiple weight change data, and to output the second differentiator output signal.

[0013] The second multiplier is used to receive the output signal of the first differentiator and the output signal of the second differentiator, and to output the output signal of the second multiplier.

[0014] The integrator is used to receive the output signal of the second multiplier and output the integrator's output signal.

[0015] The adder is used to receive multiple weight change signals of multiple weight change data, the integrator output signal, and the output signal of the adder;

[0016] Each weight change signal corresponds to a different weight value, and each weight change data represents the amount of change in the weight value corresponding to the weight change data over time.

[0017] In one alternative implementation, the target neural network includes multiple different subnetworks, each subnetwork establishing a control chip, where each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron.

[0018] In one alternative implementation, both the input neuron and the output neuron are DC level signals.

[0019] In one alternative implementation, in two adjacent subnets, the output neurons of the front subnet are connected to the input neurons of the rear subnet.

[0020] In one alternative implementation, when training target training data in the target neural network, the MSE error signal is fed back to all subnetworks of the target neural network via wires.

[0021] In one alternative implementation, the input side of the target neural network is connected to the first conversion unit, and the output side of the target neural network is connected to the error calculation unit.

[0022] In an optional implementation, the neural network optimization circuit in this embodiment of the invention further includes:

[0023] The computer is used to send digital signals corresponding to the target training data to a first conversion unit, send digital signals corresponding to the label data of the target training data to a second conversion unit, and receive prediction result signals corresponding to the target training data.

[0024] In one alternative implementation, it further includes:

[0025] The third conversion unit is used to convert the prediction result signal corresponding to the target training data into a digital signal and then transmit it to the computer.

[0026] In one alternative implementation, the target neural network modifies the weight values ​​corresponding to each weight change data while training the target training data.

[0027] In one alternative implementation, the target neural network includes: a Transformer neural network, a CNN convolutional neural network, an RNN recurrent neural network, or a fully connected neural network.

[0028] The technical solution of this invention has the following advantages:

[0029] This invention discloses a neural network optimization circuit applied to a target neural network, comprising: a first conversion unit for converting target training data into a target DC level signal corresponding to the target training data; a second conversion unit for converting label data corresponding to the target training data into a labeled result level signal; a first multiplier for multiplying the target DC level signal corresponding to the target training data and the output signal of the adder, and outputting a prediction result signal corresponding to the target training data; an error calculation unit for calculating the MSE error signal between the prediction result signal and the labeled result signal; and a first differentiator for receiving the MSE error signal and outputting a first differential. The invention comprises the following components: a first differentiator (for receiving multiple weight change signals from multiple weight change data and outputting the second differentiator output signal); a second multiplier (for receiving the output signals from the first differentiator and the second differentiator, and outputting the second multiplier output signal); an integrator (for receiving the output signal from the second multiplier and outputting the integrator output signal); and an adder (for receiving multiple weight change signals from multiple weight change data, the integrator output signal, and outputting the adder output signal). Each weight change data corresponds to a different weight value, and each weight change data represents the change in the corresponding weight value over time. This invention improves the optimization speed of neural network training equipment while simultaneously possessing the capabilities of rapid inference training, low expansion cost, high parallelism, in-memory computing, and on-chip learning. Furthermore, this invention is implemented entirely using analog circuits, eliminating the need for multiple computer calculations of weight error values. There are no discontinuous signals such as switching signals or clock signals during the entire training process, achieving rapid convergence under the guarantee of rigorous mathematical theory. Attached Figure Description

[0030] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the structure of a random number generator according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the RWC learning process according to an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram comparing the RWC learning and backpropagation algorithms according to an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of a circuit network structure applying the RWC principle according to an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the neural network optimization circuit training process according to an embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram of the reasoning process and structural framework of a neural network optimization circuit training process according to an embodiment of the present invention.

[0037] Figure 7 This is a schematic diagram of the reasoning process and structural framework according to an embodiment of the present invention, and a schematic diagram of the modular layout of the internal modules of the analog circuit neural network.

[0038] Figure 8 This is a schematic diagram illustrating a three-layer 2*2*2 neural network with activation functions built in Simulink according to an embodiment of the present invention;

[0039] Figure 9 This is a schematic diagram of the loss curves for training two datasets according to an embodiment of the present invention;

[0040] Figure 10 This is a schematic diagram illustrating the changes in the solution during the training process according to an embodiment of the present invention;

[0041] Figure 11 This is a schematic diagram illustrating how the circuit according to an embodiment of the present invention completes the XOR task within 10 ns. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In recent years, with the development of AI technology, the demand for computing power has risen rapidly, and improving optimization speed is an important development direction. Therefore, the application of neural networks is becoming more and more widespread.

[0044] In related technologies, the theory of Random Weight Change (RWC) has been introduced into neural networks to fabricate analog circuit neural network chips. For example, in 1993, Kenichi Hirotsu and Martin A. Brooke of Georgia Institute of Technology refined the RWC theory. The chip weights could be modified in 100 ns at a time, completing the XOR task within 200 µs. This work inspired subsequent researchers to develop fast neural network optimization chips based on analog circuits, achieving significant influence. However, due to the second downturn in neural networks after 1993, development in this field was slow. Furthermore, subsequent work has not escaped the limitations of clock signals or the need for digital circuits.

[0045] RWC learning algorithm such as Figure 2 As shown, the weights change randomly from the initial state, with increments of +δ or -δ. If the sum of the expected output errors decreases with the weight changes, the process iterates using the same weight changes until the error increases. If the sum of errors increases, the weights are updated randomly again. The sum of the output errors is calculated by acquiring the output from the chip using a digital computer. Figure 3 The diagram illustrates a comparison between the RWC learning algorithm and the backpropagation algorithm. In backpropagation, the current solution descends along the steepest slope of the error surface. For RWC, the current solution fluctuates up and down on the error surface instead of descending linearly along the steepest slope, but statistically it does descend, eventually leading to the correct answer. This learning algorithm is less efficient than backpropagation in digital computer simulations; however, it is well-suited for simulation implementation because it does not require a backpropagation circuit that is exactly symmetrical to forward propagation and can update weights using very simple circuitry. It only involves weight changes in two states: +δ or -δ. Furthermore, the RWC learning algorithm does not require any specific network structure. It can be applied to networks other than feedforward networks.

[0046] exist Figure 1In this approach, a shift register generates a random value ±δ. A random pulse +Vr or -Vr is applied to one side of the shift register, and the pulse propagates to the other side. The random weight change ±δ is transmitted to each weight circuit through switched capacitors. Switched capacitors are equivalent to resistors and are a lower-cost implementation in digital circuits.

[0047] Based on the RWC algorithm principle and random number generation mechanism, the circuit operation rules can be summarized as follows: If the error does not decrease, a weight shift direction is given (by inputting a random value into the shift register and the computer providing a clock signal to control the shift). If the error decreases, no clock signal is given to the shift register, and the shift register maintains its original weight shift direction. This judgment process requires the participation of a digital computer each time, involving AD conversion, error comparison, summation, judgment of error changes, and the provision of a clock signal.

[0048] like Figure 4 The diagram shows a circuit network structure applying the RWC principle. The weight chip and neuron chip are manufactured separately and connected according to the diagram. Combining them can realize any network structure. Through cascading, various types of chips and larger networks can be obtained. Experimentally, a 2*2*1 three-layer neural network was built, achieving XOR training within 200µs.

[0049] The specific shortcomings of neural networks using the RWC learning algorithm are analyzed as follows:

[0050] First, the neural network in the RWC learning algorithm is clock-driven, essentially a discrete system. Within a control cycle, the weights can pass through and exceed their optimal values. Since the circuit has no constraints, the rate of change of the weights must be kept relatively small. This means it is limited by the control cycle limit, resulting in slow convergence. Furthermore, the direction of random perturbations is not guaranteed to be the direction of loss descent; therefore, there are certain control cycles where the loss increases.

[0051] Second, in the neural network of the RWC learning algorithm, if the sum of errors increases, the weights will be randomly updated again. Each random update of the sum of output errors requires the digital computer to collect the output from the chip and then calculate it. Therefore, the feedback loop requires the digital computer to participate multiple times, making pure on-chip training impossible.

[0052] Third, in the neural network of the RWC learning algorithm, the digital computer's involvement includes A / D conversion of the chip neuron output, error comparison, summation, judgment of error changes, and D / A conversion of clock signals, resulting in a complex process and slow speed.

[0053] Fourth, since the perturbation value of each weight in the circuit is discrete ±1, the directions of the weight perturbation are all orthogonal in the parameter space, making it impossible to modify the weight in any direction.

[0054] Fifth, training a single data point requires multiple computer interventions, disrupting the continuity of the system and failing to fully utilize the speed advantage of analog circuit continuity.

[0055] In view of this, according to embodiments of the present invention, such as Figure 5 The diagram illustrates a neural network optimization circuit applied to a target neural network. This circuit includes: a first conversion unit 51, a second conversion unit 52, a first multiplier 53, a first differentiator 54, a second differentiator 55, a second multiplier 56, an adder 57, an integrator 58, and an error calculation unit 59. Figure 5 The neural network optimization circuit shown is the basic training unit in this embodiment of the present disclosure, namely the weights of the neural network. Figure 5 In the diagram, the brown dashed line represents any target neural network, which can be a Transformer neural network, a CNN convolutional neural network, an RNN recurrent neural network, or a fully connected neural network.

[0056] In practical design, depending on the type of disturbance and the way the disturbance signal is introduced, the basic units of analog circuit neural networks can be divided into several different basic structures. The neural network optimization circuit in this embodiment is suitable for scenarios involving the processing of batch training data, and has the widest applicability, allowing for the expansion of networks sharing weights, such as batch processing and attention mechanisms. Figure 5 The neural network optimization circuit shown is suitable for scenarios where the target neural network is a Transformer neural network, a CNN convolutional neural network, or an RNN recurrent neural network, and each weight participates in the calculation of multiple data points simultaneously.

[0057] In this embodiment, a two-layer 2*2 fully connected neural network is used as an example. The two circles on the left are input neurons, with DC level as the input signal and DC level as the output signal. The input and output neurons are fully connected by four weights. (Solid lines connect the left and right neurons). Figure 5 The analog circuitry on the lower middle side illustrates the internal structure of the weights.

[0058] Among them, Figure 5 In the first conversion unit 51, the target training data is converted into the target DC level signal corresponding to the target training data.

[0059] Specifically, such as Figure 6As shown, the first conversion unit 51 in this embodiment can be a D / A digital-to-analog converter, which converts the target training data into a target DC level signal corresponding to the target training data. The target training data can be different types of data sample sets, and each data point in the data sample set is converted into a target DC level signal. For example, the target training data is represented by M, where M = {a1, a2, a3, ... a...}. n The method converts each of the n data points in the data sample set into a target DC level signal. In this embodiment, the target training data is also the input training data. Converting the target training data into the corresponding target DC level signal and using it as the input signal for the neural network helps ensure the continuity of the analog system signal, thereby enabling the circuit to train the target neural network at high speed.

[0060] Among them, Figure 5 In the middle, the second conversion unit 52 is used to convert the label data corresponding to the target training data into the label result level signal.

[0061] Specifically, such as Figure 6 As shown, the second conversion unit 52 in this embodiment can be a D / A digital-to-analog converter, which converts the label data corresponding to the target training data into a labeling result level signal. The labeling result signal of the target training is a label data level signal. Figure 5 In this context, both the input data level signal x and the label data level signal y are provided. For example, the target training data is represented by M, where M = {a1, a2, a3, ... a...}. n}, then all n data points in the data sample set will be converted into target data level signal x and tag data level signal y.

[0062] Among them, Figure 5 In the first multiplier 53, the multiplier is used to multiply the target DC level signal corresponding to the target training data and the adder output signal to output the prediction result signal corresponding to the target training data.

[0063] Specifically, in Figure 5 In the process of reasoning the target neural network, the weight value level signal at the output of adder 57 is multiplied to obtain the output value. Inference can then be completed. For example, the target DC level signal x corresponding to the target training data is multiplied by the adder output signal (weight value level signal) to output the prediction result signal corresponding to the target training data.

[0064] Among them, Figure 5 In the middle, the error calculation unit 59 is used to calculate the MSE error signal between the prediction result signal and the labeled result signal of the target training data.

[0065] Specifically, in Figure 5 In this circuit, the error calculation unit 59 and the first differentiator 54 can be MSE differentiating units. These MSE differentiating units receive the DC level of the tag data output from the PC via a D / A converter. Internally, the MSE differentiating unit calculates the negative of the MSE loss differential, outputs it as a wire, and then feeds it back to the analog circuit neural network. This forms a closed loop, and the weights of the analog circuit neural network converge in the direction of minimizing the error, thus completing the training.

[0066] Among them, Figure 5 In the middle, the first differentiator 54 is used to receive the MSE error signal and output the first differentiator output signal.

[0067] Specifically, in Figure 5 In the training of the target neural network, both the input data level signal x and the label data level signal y are given simultaneously, and the circuit's output signal... The MSE loss is calculated with y and inverted. Then, after passing through the first differentiator 54, it is connected to the second multiplier 56 to represent the negative number of the change in MSE loss.

[0068] Among them, Figure 5 In the second differentiator 55, multiple weight change signals of multiple weight change data are received, and the output signal of the second differentiator is output. Each weight change signal corresponds to a different weight value, and each weight change data represents the amount of change of the weight value corresponding to the weight change data over time.

[0069] Specifically, in the embodiments of this disclosure, the multiple weight change signals of multiple weight change data represent weight perturbation noise signals. For example, when the target neural network is training with batch data, there are N noise signals, which are described as multiple weight change signals. If the target neural network has N weight change data, it means that there are N corresponding noise signals, and each noise signal represents the amount of change of the weight value over time.

[0070] Among them, Figure 5 In this circuit, the second multiplier 56 receives the output signal of the first differentiator 54 and the output signal of the second differentiator 55, and outputs the output signal of the second multiplier 56. The integrator 58 receives the output signal of the second multiplier 56 and outputs the integrator's output signal. The adder 57 receives multiple weight change signals of multiple weight change data, the output signal of the integrator 58, and outputs the adder's output signal.

[0071] Specifically, in Figure 5 In the process, during inference, the target neural network takes a given input data level signal x and multiplies it with the weight value level signal at the output of adder 57 to obtain the output value. Inference can then be completed. During training, both the input data level signal x and the tag data level signal y are given simultaneously. The circuit's output signal... The MSE loss is calculated by inverting the value of y, and then passed through the first differentiator 54 and connected to the second multiplier 56, representing the negative of the change in MSE loss. The other end of the second multiplier 56 is connected to the noise via the second differentiator 55. Simultaneously, the noise is connected to the input of the adder 57, whose other input is the DC level signal of the weights. Therefore, the output of the adder 57 is the updated weight value (W+ΔW) after perturbation. The DC level signal of the weights is also the output of the integrator 58, meaning the change in the weight value is determined by the product of the two inputs of the bottom-most second multiplier 56. One quantity, the negative of the differential of the MSE loss, represents the trend of the loss change under the current perturbation; only its sign matters, indicating whether the error has decreased or increased. The other quantity represents the change in weights. The product of the two is the weight change that reduces the error. For example, when the weights change, if the negative of the differential of the MSE loss is positive, it indicates that the error is decreasing, and the weights should shift in that direction. In other words, the change in weights should be a positive number multiplied by the original change in weights. If the negative of the differential value of the MSE loss is negative, it indicates that the error is increasing, and the weights should be shifted in the opposite direction of the change in weights. That is, the change in weights should be a negative number multiplied by the original change in weights. In this way, the circuit can ensure that the weight values ​​always shift in the direction that reduces the error, thus completing the training of the neural network.

[0072] The neural network optimization circuit in this embodiment improves the optimization speed of neural network training equipment while simultaneously possessing the capabilities of fast inference training, low expansion cost, high parallelism, in-memory computing, and on-chip learning. Furthermore, the neural network optimization circuit in this embodiment is entirely implemented using analog circuitry, eliminating the need for multiple computer calculations of weight error values. It also eliminates discontinuous signals such as switching signals and clock signals during the entire training process, achieving rapid convergence under strict mathematical guarantees. The convergence speed is faster than existing optimizers (Simulink's simulation training speed is 147,900,000 times that of GPUs).

[0073] In one optional implementation, the target neural network comprises multiple subnetworks, each subnetwork establishing a control chip. Each pin on the input side of this control chip corresponds to an input neuron, and each pin on the output side corresponds to an output neuron. Both input and output neurons are DC level signals. In adjacent subnetworks, the output neuron of the preceding subnetwork is connected to the input neuron of the following subnetwork. When training the target neural network with target training data, the MSE error signal is fed back to all subnetworks of the target neural network via wires.

[0074] Specifically, such as Figure 7Part B shown illustrates the structural framework of the circuit (Analog neural network) for inference in this embodiment. The left side of the circuit represents the input terminal, with each pin corresponding to an input neuron, using a DC level as the input signal. The right side of the circuit represents the output terminal, with each pin corresponding to an output neuron, using a DC level as the output signal. The input signal is provided by a computer-controlled D / A digital-to-analog converter. The output signal is converted into a digital signal by an A / D analog-to-digital converter and transmitted to the computer.

[0075] exist Figure 7 Part D of the diagram illustrates the modular layout within an analog circuit neural network. When the network is large-scale or heterogeneous, it can be divided into different subnetworks, each with its own dedicated chip. In the forward propagation section, the output neurons of the preceding subnetworks are connected to the input neurons of the following subnetworks. The training section then feeds back the final differential error to all subnetworks to perform the training.

[0076] Therefore, the neural network optimization circuit in this embodiment trains all target training data simultaneously and performs error optimization on all target training at the same time. It is highly parallel, has a fast computing speed, and has on-chip learning capabilities. The training process does not have discontinuous signals such as switching signals and clock signals. It achieves fast convergence under the guarantee of strict mathematical theory. It uses all analog circuits, so the expansion cost is low.

[0077] In this embodiment, both the input and output neurons are DC level signals. Circuit storage and computation are implemented using weight levels, and since they share the same DC level signal, it belongs to an in-memory computing architecture. Furthermore, the DC level signal helps ensure the continuity of the analog system signal, thereby enabling the circuit to train the target neural network at high speed.

[0078] In one alternative implementation, in Figure 5 In this configuration, the input side of the target neural network is connected to the first conversion unit 51, and the output side of the target neural network is connected to the error calculation unit 59.

[0079] In one alternative implementation, such as Figure 6 and Figure 7 As shown, it also includes: a computer 50 and a third conversion unit 530, used to send digital signals corresponding to the target training data to the first conversion unit 51, send digital signals corresponding to the label data of the target training data to the second conversion unit 52, and receive prediction result signals corresponding to the target training data. The third conversion unit 530 is used to convert the prediction result signals corresponding to the target training data into digital signals and transmit them to the computer 50.

[0080] Specifically, the first conversion unit 51 and the second conversion unit 52 mentioned above are both D / A digital-to-analog converters, and the third conversion unit 530 is an A / D analog-to-digital converter. Figure 6 The computer 50 in this embodiment simultaneously provides input data and label data. The input data is provided to the first conversion unit 51, and the label data is provided to the second conversion unit 52. In this embodiment, the computer only provides data and does not participate in the weight error calculation. During the weight error calculation process, the data is processed by... Figure 5 The circuit shown is implemented using analog circuits. Therefore, the neural network optimization circuit in this embodiment is implemented entirely using analog circuits, without the participation of computers or other digital circuits, which can take advantage of the high speed of analog computing.

[0081] In one optional implementation, the target neural network simultaneously modifies the weight values ​​corresponding to each weight change data while training the target training data. Therefore, the neural network optimization circuit in this embodiment has a high degree of parallelism, and all weights are modified simultaneously during neural network training, resulting in low time complexity, which is only 1. This time complexity of only 1 means that all weight units are optimized simultaneously, and for each weight, the calculation process completes one optimization of the neural network in only one operation.

[0082] In a specific example, the neural network optimization circuit in this disclosure embodiment can achieve fast inference training. Fast inference refers to how long it takes for the neural network to obtain a stable output after being given an input. Since the weights do not change during the inference phase, it is equivalent to each weight unit only performing multiplication calculations. Under ideal device conditions, the entire neural network inference time is only constrained by the speed of light; under non-ideal device conditions, it is affected by device latency. It can generally be completed in nanoseconds or sub-nanosecond time. The theoretical guarantee of fast training is that the weight training effect is equivalent to backpropagation (BP), and the direction of weight modification is the unbiased estimate of BP. When external input data and label data are given, the weight levels of the neural network will converge, and the convergence direction is the negative direction of the gradient of the weights relative to the loss function, that is, the weight modification direction of BP (backpropagation). Therefore, as long as a neural network can be trained using BP, it can theoretically be trained using the circuit proposed in this disclosure embodiment, and the training results are consistent. If the problem being trained is a convex problem, the circuit will definitely converge to the optimal solution. The unbiased theoretical proof process is as follows:

[0083] Based on the weighted perturbation algorithm, Figure 5 The transfer function of the circuit is given by the following formula:

[0084]

[0085] Where W∈R N×1 Let L be the weight parameters, L ∈ R be the loss function, and ξ ∈ R.N×1 This is a disturbance.

[0086] ξ=∫δdt,δ~N(0,I N ).

[0087] Using the chain rule, we can obtain the following formula:

[0088]

[0089] in,

[0090] When ξ is sufficiently small, that is, the condition is satisfied. When, the following formula is given:

[0091]

[0092] at this time,

[0093]

[0094] n represents the number of gradient estimations. In analog circuits, the system is a continuous system. Therefore, within an infinitely short time step, since the electric field is established at the speed of light, n can be considered infinitely large if the circuit size is sufficiently small. Thus, at any given time, the change in weights satisfies the above equation (equivalent to backpropagation).

[0095] like Figure 8 The image shows a three-layer 2x2x2 neural network with activation functions built in Simulink. Figure 9 The diagram shows the loss curves for training two datasets. The experimental results show that training on both datasets was completed within 1 ns. The data was switched 4-9 times. Under the same data and network, GPU training required an average of 12 steps and took an average of 13.6 ms. The circuit involved in this embodiment is 13,600,000 times faster than the digital circuit GPU. To ensure experimental rigor, the GPU time only includes forward and backward propagation times, excluding GPU initialization, data initialization, and optimizer initialization times. Furthermore, the Simulink simulation uses ideal devices and does not consider latency, error, or noise. Moreover, GPUs are versatile and can be applied to networks with different structures, so GPUs may sacrifice some speed for versatility. Dedicated digital circuits may be faster. However, even dedicated digital circuits will not be this many orders of magnitude faster. Therefore, this experimental result proves that analog circuits are much faster than GPUs, as shown in Table 1 below.

[0096] Table 1

[0097]

[0098] To further ensure rigor, the experiment compared different CUDA versions, different GPUs, and different operating systems, and obtained similar experimental results.

[0099] like Figure 10 The figure shows the changes in the solution during training. It can be seen from the graph that the solution slides to the lowest point along the steepest descent direction. This is consistent with the proof of unbiased theory. Figure 11 As shown, the circuit completed the training of the XOR task within 10 ns. This also proves that the circuit has a fast training speed. The XOR task is a classic task in neural networks, which requires multiple layers of nonlinear networks to learn. The circuit involved in this embodiment also learned the XOR task within 10 ns. This shows that the circuit has the ability to be extended to other neural network structures, such as convolutional neural networks, ResNet, Transformer, GPT, etc.

[0100] Low expansion cost primarily considers the complexity of the optimization unit and the impact of networking on efficiency. Regarding the complexity of the optimization unit, the circuits involved in the embodiments of this disclosure are all... Figure 5 The circuit structure shown is an infinite repetition, and its structure is simple. In contrast, the complexity of a GPU is mainly reflected in the following aspects: 1. Complex programming and control logic. GPUs contain complex control units for managing task scheduling, memory management, error correction, and other system-level functions. 2. Complex data paths: GPUs are designed with complex data paths capable of handling large amounts of data movement, including data caching, prefetching, and efficient data transfer mechanisms. 3. Complex thermal and power management: High-performance GPUs require complex thermal and power management systems to cope with heat and power consumption under high loads.

[0101] Regarding networking efficiency, the circuits in the embodiments of this disclosure can use... Figure 7The structure shown is extended. The output neurons of the front subnet are connected to the input neurons of the back subnet during forward propagation, and the final error differential wire is fed back to all subnets. In contrast, GPU networking is much more complex. The main issues involved are: 1. Data transfer latency. Cross-GPU communication: In model-parallel settings, different parts of the model are assigned to different GPUs. This means that data generated by each GPU during computation (such as activation values ​​and gradients) needs to be passed to other GPUs. Data transfer can cause significant latency when GPUs are connected via PCIe bus or network (in the case of cross-nodes). Bandwidth limitations: Data transfer speed is limited by hardware bandwidth. Even high-speed connections such as InfiniBand or NVLink can become bottlenecks when the data volume is large. 2. Synchronization overhead. Barrier synchronization: Parallel processing requires ensuring that all processing units have completed their current step before proceeding to the next computation. This synchronization typically requires a barrier synchronization mechanism, which ensures that all GPUs reach the same state before starting the next computation cycle. Each synchronization causes all GPUs to wait for the slowest one, introducing latency. Consistency and State Management: Maintaining data consistency is essential during parallel processing, which may involve additional communication and processing overhead to ensure that the data state of all processing units is consistent. 3. Communication to Computation Ratio. Computation / Communication Ratio: This is an important metric for measuring parallel efficiency, describing the ratio of the time required to perform computation to the time required for communication. Ideally, we want this ratio to be as high as possible, meaning that most of the time is spent on actual computation rather than data transfer. However, in many model parallel scenarios, especially when the model is large or distributed across multiple computing nodes, this ratio may be too low.

[0102] High parallelism refers to the proportion of parameters being optimized simultaneously out of all parameters. A higher proportion indicates a higher degree of parallelism in the computing hardware. Higher parallelism leads to higher computational efficiency. According to Amdahl's Law, if the proportion of tasks that can be executed in parallel is 1, and the number of computing resources is the weight number N, then the Amdahl speedup is N. The main advantage of high parallelism is reflected in time complexity, as shown in Table 2.

[0103] Table 2 Time Complexity

[0104] Digital circuits ACO Forward propagation <![CDATA[O(LN 2 )]]> O(1) Backpropagation <![CDATA[O(N 2 )]]> O(1)

[0105] L is the number of layers in the neural network.

[0106] N is the number of neurons in the hidden layer.

[0107] In this embodiment, all weights of the analog circuit are optimized simultaneously, therefore the time complexity is not affected by the network size. The optimization speed of the digital circuit slows down as the network size increases.

[0108] Processing-in-Memory (PIM) computing is a technology that performs computational tasks directly within storage devices. Since the computation and storage of each weight are the same physical quantity, determined by the output level of the integrator, the circuit architecture of this invention belongs to the PIM architecture. This architecture breaks through the boundary between the processor and storage in traditional architectures, allowing data processing during the storage stage and reducing the need for data transfer between the processor and memory. This invention belongs to the PIM architecture and has its advantages. PIM has several significant advantages, especially suitable for data-intensive applications and large-scale data processing scenarios. 1. Reduced data transfer requirements: In traditional computing architectures, the CPU needs to frequently read and write data from memory, which leads to significant energy consumption and latency. PIM design significantly reduces data movement by processing data directly in memory, thereby reducing energy consumption and increasing data processing speed. 2. Reduced energy consumption: Data transfer is one of the main sources of energy consumption in modern computing systems. PIM technology effectively reduces the overall system energy consumption by reducing data transfer on the system bus and other interfaces. 3. Improved Performance: Since data processing speed is limited by data transfer speed, CPUs in traditional architectures often remain idle while waiting for data. In-memory computing reduces processor wait time, allowing data processing tasks to be completed faster, thus improving overall computing performance. 4. Scalability and Flexibility: In-memory computing technology offers greater flexibility, allowing for customized configuration of memory and computing resources based on application needs. This enables optimization for specific applications, such as big data analytics and machine learning. 5. Simplified System Design: By moving computational tasks to memory, the overall hardware design of the system can be simplified, reducing reliance on high-speed data buses and other high-cost hardware, and potentially reducing the physical size of the system. 6. Improved Parallel Processing Capabilities: In-memory computing architecture allows computational tasks to be executed in parallel across multiple memory modules, significantly enhancing parallel processing capabilities. This is particularly important for applications requiring high parallelism. 7. Enhanced Data Processing Security: By processing data directly in memory, data transfer within the system is reduced, thereby lowering the risk of potential data leaks and enhancing data security.

[0109] On-chip learning is a concept involving implementing learning and inference tasks directly on a microchip (typically a dedicated integrated circuit or processor). This invention, due to its rapid training capabilities, enables real-time training on edge devices, achieving on-chip learning. This approach is particularly important in the field of artificial intelligence, focusing on integrating the training and execution of machine learning models onto the hardware device itself, rather than relying on external large-scale computing systems or cloud infrastructure. On-chip learning is a crucial component of edge computing and smart hardware design, significantly improving the speed and efficiency of data processing while reducing data transmission requirements and latency.

[0110] In summary, the neural network optimization circuit in the embodiments of this disclosure has the following advantages:

[0111] 1. Fast inference and training: The circuit uses analog circuits for inference and training of neural networks while preserving the continuity of analog system signals, enabling the circuit to train neural networks at high speed.

[0112] 2. Low expansion cost: Neural networks can be expanded through direct connections, eliminating the need for signal conversion (such as NVLink), encoding / decoding, and other operations between circuits. Furthermore, analog circuit neural networks integrate in-memory computation and have a simple weight structure, resulting in low cost (compared to the in-memory computation separation structure of GPUs).

[0113] 3. Highly parallel: During neural network training, all weights are modified simultaneously, resulting in low time complexity. The time complexity is only 1.

[0114] 4. In-memory computing: Both circuit storage and computation are implemented through weighted levels and are the same DC level signal, thus belonging to the in-memory computing architecture.

[0115] 5. On-chip learning: Currently, edge computing devices capable of on-chip learning are scarce. Current on-chip learning devices generally suffer from low training speeds and can only train a subset of weights. The embodiments disclosed in this disclosure can achieve on-chip learning of all weights.

[0116] 6. Fast convergence speed, no clock-driven circuit required.

[0117] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A neural network optimization circuit, characterized in that, Applied to target neural networks, including: The first conversion unit is used to convert the target training data into the target DC level signal corresponding to the target training data; The second conversion unit is used to convert the label data corresponding to the target training data into a labeling result level signal; The first multiplier is used to multiply the target DC level signal corresponding to the target training data and the adder output signal to output the prediction result signal corresponding to the target training data. An error calculation unit is used to calculate the MSE error signal between the predicted result signal and the labeled result level signal; The first differentiator is used to receive the MSE error signal and output the first differentiator output signal; The second differentiator is used to receive multiple weight change signals from multiple weight change data, and to output the second differentiator output signal. The second multiplier is used to receive the output signal of the first differentiator and the output signal of the second differentiator, and to output the output signal of the second multiplier. An integrator is used to receive the output signal of the second multiplier and output the integrator output signal; An adder is used to receive multiple weight change signals of the multiple weight change data, the integrator output signal, and the output signal of the adder; Each weight change signal corresponds to a different weight value, and each weight change data represents the amount of change in the weight value corresponding to the weight change data over time.

2. The neural network optimization circuit according to claim 1, characterized in that, The target neural network includes multiple different subnetworks, each of which establishes a control chip. Each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron.

3. The neural network optimization circuit according to claim 2, characterized in that, Both the input neuron and the output neuron are DC level signals.

4. The neural network optimization circuit according to claim 2, characterized in that, In two adjacent subnets, the output neurons of the anterior subnet are connected to the input neurons of the posterior subnet.

5. The neural network optimization circuit according to claim 2, characterized in that, When training the target training data in the target neural network, the MSE error signal is fed back to all subnetworks of the target neural network through wires.

6. The neural network optimization circuit according to claim 1, characterized in that, The input side of the target neural network is connected to the first conversion unit, and the output side of the target neural network is connected to the error calculation unit.

7. The neural network optimization circuit according to claim 1, characterized in that, Also includes: The computer is configured to send digital signals corresponding to the target training data to the first conversion unit, send digital signals corresponding to the label data of the target training data to the second conversion unit, and receive prediction result signals corresponding to the target training data.

8. The neural network optimization circuit according to claim 7, characterized in that, Also includes: The third conversion unit is used to convert the prediction result signal corresponding to the target training data into a digital signal and then transmit it to the computer.

9. The neural network optimization circuit according to claim 1, characterized in that, When training the target training data, the target neural network simultaneously modifies the weight values ​​corresponding to each weight change data.

10. The neural network optimization circuit according to any one of claims 1 to 9, characterized in that, The target neural network includes: Transformer neural network, CNN convolutional neural network, RNN recurrent neural network, or fully connected neural network.

Images