Neural network optimization circuit
The neural network optimization circuit designed using analog circuits solves the problems of complex computation process, slow speed and poor system continuity in the existing technology. It realizes fast training and inference, has high parallelism and low expansion cost, and is suitable for neural networks such as Transformer, CNN and RNN.
Patent Information
- Application Number
- CN202411117422.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-14
AI Technical Summary
Existing neural network optimization circuits suffer from problems such as complex computation processes, slow speed, poor system continuity, and slow convergence speed. In particular, when applying the RWC method, the computer needs to participate multiple times, which disrupts the continuity of the system and training efficiency.
A neural network optimization circuit was designed, which uses analog circuits to implement the training process. It includes a first conversion unit, a second conversion unit, a first multiplier, an error calculation unit, a differentiator, a multiplier, an integrator, and an adder. The weights are optimized through analog signal processing, avoiding multiple computer interventions and the use of clock signals, thus forming a closed-loop system.
It achieves rapid training and inference of neural networks, possesses high parallelism, low expansion cost, in-memory computing and on-chip learning capabilities, converges faster than existing technologies, and achieves rapid convergence under the guarantee of rigorous mathematical theory.
Smart Images

Figure CN119026652A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of circuit technology, and in particular to a neural network optimization circuit. Background Art
[0002] In recent years, with the development of AI technology, the demand for computing power has increased rapidly, and improving optimization speed is an important development direction. Therefore, the application of neural networks is becoming more and more extensive.
[0003] In the related art, the random weight change (RWC) theory is introduced into the neural network to produce an analog circuit neural network chip. Figure 1 As shown in the figure, it is a circuit diagram of applying random generator in neural network. Figure 1 In the process, a shift register is used to generate random values ±δ. A random pulse +Vr or -Vr is applied to one side of the shift register, and the pulse propagates to the other side. The random weight changes ±δ and is transmitted to each weight circuit through the switch capacitor. The switch capacitor is equivalent to a resistor. For a neural network that uses the RWC random generation mechanism, if the neural network error does not decrease, a weight movement direction is given (a random value is input through the shift register, and the computer gives a clock signal to control the shift). If the neural network error decreases, the computer does not give the shift register a clock signal, and the shift register maintains the original weight movement direction.
[0004] Therefore, the judgment process of the RWC method requires the participation of a computer every time, and it is also necessary to perform AD conversion, error comparison, summation, and judgment of error changes in sequence to give a clock signal, which leads to a complicated process and slow speed, and it is impossible to perform pure on-chip training. In addition, the neural network requires multiple participation of the computer when training a piece of data, which destroys the continuity of the system. The clock-driven method has a slow convergence speed. Summary of the invention
[0005] In view of this, the present invention provides a neural network optimization circuit to solve the problems of complex calculation process, slow speed, poor system continuity and poor convergence speed of the neural network optimization circuit using the RWC method.
[0006] According to a first aspect, an embodiment of the present invention provides a neural network optimization circuit, which is applied to a target neural network, comprising:
[0007] A first conversion unit, configured to convert the target training data into a target DC level signal corresponding to the target training data;
[0008] A second conversion unit, used to convert the label data corresponding to the target training data into a labeling result level signal;
[0009] A first multiplier, configured to multiply a target DC level signal corresponding to the target training data by the adder output signal, and output a prediction result signal corresponding to the target training data;
[0010] An error calculation unit, used to calculate the MSE error signal between the prediction result signal and the labeled result signal;
[0011] A first differentiator, configured to receive the MSE error signal and output a first differentiator output signal;
[0012] a second differentiator, configured to receive a plurality of weight change signals of a plurality of weight change data, and output a second differentiator output signal;
[0013] a second multiplier, configured to receive the first differentiator output signal, the second differentiator output signal, and output the second multiplier output signal;
[0014] an integrator, configured to receive the second multiplier output signal and output an integrator output signal;
[0015] an adder, configured to receive a plurality of weight change signals of a plurality of weight change data, an integrator output signal, and output an adder output signal;
[0016] Each weight change signal corresponds to a different weight value, and each weight change data represents the change in time of the weight value corresponding to the weight change data.
[0017] In an optional embodiment, the target neural network includes a plurality of different subnetworks, each subnetwork establishes a control chip, each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron.
[0018] In an optional implementation, both the input neuron and the output neuron are DC level signals.
[0019] In an optional implementation, in two adjacent sub-networks, the output neurons of the front sub-network are connected to the input neurons of the rear sub-network.
[0020] In an optional implementation, when the target training data is trained in the target neural network, the MSE error signal is fed back to all sub-networks of the target neural network through a wire.
[0021] In an optional embodiment, the input side of the target neural network is connected to the first conversion unit, and the output side of the target neural network is connected to the error calculation unit.
[0022] In an optional implementation manner, the neural network optimization circuit in the embodiment of the present invention further includes:
[0023] A computer is used to send a digital signal corresponding to target training data to a first conversion unit, send a digital signal corresponding to label data of the target training data to a second conversion unit, and receive a prediction result signal corresponding to the target training data.
[0024] In an optional implementation, it also includes:
[0025] The third conversion unit is used to convert the prediction result signal corresponding to the target training data into a digital signal and transmit it to the computer.
[0026] In an optional implementation, when training the target training data, the target neural network simultaneously modifies each weight change data to correspond to a different weight value.
[0027] In an optional embodiment, the target neural network includes: a Transformer neural network, a CNN convolutional neural network, a RNN recurrent neural network, or a fully connected neural network.
[0028] The technical solution of the present invention has the following advantages:
[0029] The present invention discloses a neural network optimization circuit, which is applied to a target neural network, and includes: a first conversion unit, which is used to convert target training data into a target DC level signal corresponding to the target training data; a second conversion unit, which is used to convert label data corresponding to the target training data into a labeling result level signal; a first multiplier, which is used to multiply the target DC level signal corresponding to the target training data and the adder output signal, and output a prediction result signal corresponding to the target training data; an error calculation unit, which is used to calculate the MSE error signal between the prediction result signal and the labeling result signal; a first differentiator, which is used to receive the MSE error signal and output a first differential signal; The invention comprises a first differentiator for receiving a plurality of weight change signals of a plurality of weight change data, and outputting a second differentiator output signal; a second multiplier for receiving a first differentiator output signal, a second differentiator output signal, and outputting a second multiplier output signal; an integrator for receiving a second multiplier output signal and outputting an integrator output signal; an adder for receiving a plurality of weight change signals of a plurality of weight change data, an integrator output signal, and outputting an adder output signal; wherein each weight change data corresponds to a different weight value, and each weight change data represents the amount of change in time of the weight value corresponding to the weight change data. The invention improves the optimization speed of the neural network training device while having the capabilities of fast reasoning training, low expansion cost, high parallelism, integrated storage and calculation, and on-chip learning. Moreover, the invention is implemented entirely using analog circuits, without the need for a computer to participate in calculating the weight error value multiple times, and without non-continuous signals such as switch signals and clock signals during the entire training process, and achieves rapid convergence under the guarantee of strict mathematical theory. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0031] Figure 1 is a schematic diagram of the structure of a random number generator according to an embodiment of the present invention;
[0032] Figure 2 is a schematic diagram of an RWC learning process according to an embodiment of the present invention;
[0033] Figure 3 is a comparative schematic diagram of RWC learning and back propagation algorithms according to an embodiment of the present invention;
[0034] Figure 4 is a schematic diagram of a circuit network structure using the RWC principle according to an embodiment of the present invention;
[0035] Figure 5 is a schematic diagram of a neural network optimization circuit training process according to an embodiment of the present invention;
[0036] Figure 6 is a schematic diagram of the reasoning process and structural framework of the neural network optimization circuit training process schematic diagram according to an embodiment of the present invention;
[0037] Figure 7 It is a schematic diagram of the reasoning process and structural framework according to an embodiment of the present invention and a schematic diagram of the layout of the internal modularization of the analog circuit neural network;
[0038] Figure 8 It is a schematic diagram showing a three-layer 2*2*2 neural network with an activation function built in Simulink according to an embodiment of the present invention;
[0039] Fig. 9 is a schematic diagram of a loss curve for training two data according to an embodiment of the present invention;
[0040] Fig.10 is a schematic diagram of the change of the solution during the training process according to an embodiment of the present invention;
[0041] Fig.11 It is a training schematic diagram of a circuit according to an embodiment of the present invention completing an XOR task within 10 ns. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0043] In recent years, with the development of AI technology, the demand for computing power has increased rapidly. Improving the optimization speed is an important development direction. Therefore, the application of neural networks has become more and more extensive.
[0044] In the related technology, the Random Weight Change (RWC) theory is introduced into the neural network to produce analog circuit neural network chips. For example, in 1993, Kenichi Hirotsu and Martin A. Brooke of Georgia Institute of Technology perfected the RWC theory. The single modification time of the chip weight is 100ns, and the XOR task is completed within 200us. This work inspired later generations to develop fast neural network optimization chips based on analog circuits, which has achieved great influence. However, since the neural network experienced a second trough after 1993, this field has developed slowly. In addition, subsequent work has not gotten rid of the limitations of clock signals or the need for digital circuit participation.
[0045] RWC learning algorithm Figure 2 As shown, the weights are randomly changed from the initial state by increments of +δ or -δ. If the sum of the output errors of the expected value decreases as the weights change, the same weight changes are repeated until the errors increase. If the sum of the errors increases, the weights are randomly updated again. The sum of the output errors is then calculated using a digital computer that collects the output from the chip. Figure 3 The figure shows a comparison diagram of the RWC learning algorithm and the back-propagation algorithm. During the back-propagation learning process, the current solution descends along the steepest slope of the error surface. For RWC, the current solution fluctuates up and down on the error surface, rather than descending in a straight line along the steepest slope, but it statistically descends and eventually gets the correct answer. This learning algorithm is less efficient than back-propagation in digital computer simulations, however, it is well suited for analog implementation because it does not require a back-propagation circuit that is precisely symmetrical with the forward propagation, and the weights can be updated using very simple circuits. It only has two states of weight changes +δ or -δ. In addition, the RWC learning algorithm does not require any specific network structure. It can be applied to networks other than feedforward networks.
[0046] exist Figure 1In the example, a shift register is used to generate random values ±δ, and a random pulse +Vr or -Vr is applied to one side of the shift register, and the pulse propagates to the other side. The random weight change ±δ is transmitted to each weight circuit through the switched capacitor. The switched capacitor is equivalent to a resistor and is a lower-cost implementation form in digital circuits.
[0047] Combining the principle of RWC algorithm and the random number generation mechanism, the circuit operation rules can be summarized as follows: If the error does not decrease, a weight movement direction is given (a random value is input through the shift register, and the computer gives a clock signal control to achieve the shift). If the error decreases, the shift register clock signal is not given, and the shift register maintains the original weight movement direction. This judgment process requires the participation of a digital computer every time, and there must be AD conversion, error comparison, summation, judgment of error changes, and giving a clock signal.
[0048] like Figure 4 As shown in the figure, it is a schematic diagram of the circuit network structure using the RWC principle. The weight chip and the neuron chip are produced separately and connected according to the structure shown in the figure. The combination of the two can realize any network structure. Through cascading, various types of chips and larger networks can be obtained. In terms of experiments, a 2*2*1 three-layer neural network was built, and the XOR task was completed within 200us.
[0049] For the neural network using the RWC learning algorithm, the specific defects are analyzed as follows:
[0050] First, the neural network of the RWC learning algorithm is driven by a clock and is essentially a discrete system. In a control cycle, the weight can pass and exceed the optimal value. The circuit has no constraints, so the weight change rate of the circuit must be kept at a small level. That is, it is limited by the control cycle limit and the convergence speed is slow. At the same time, the direction of random perturbation cannot be guaranteed to be the direction of loss reduction. Therefore, there are certain control cycles that cause loss to increase.
[0051] Second, in the neural network of the RWC learning algorithm, if the sum of the errors increases, the weights will be randomly updated again. Each random update of the sum of the output errors requires the digital computer to collect the output from the chip and then calculate it. Therefore, the feedback loop requires the digital computer to participate multiple times, and pure on-chip training cannot be performed.
[0052] Third, in the neural network of the RWC learning algorithm, the digital computer's participation includes A / D conversion of the chip neuron output, error comparison, summation, judgment of error changes, and D / A conversion of the clock signal, resulting in many processes and slow speed.
[0053] Fourth, since the perturbation value of each weight in the circuit is discrete ±1, the directions of the weight perturbations are all orthogonal in the parameter space, and weight modification in any direction cannot be achieved.
[0054] Fifth, training a piece of data requires the participation of computers multiple times, the continuity of the system is broken, and the speed advantage of the continuity of analog circuits is not fully utilized.
[0055] In view of this, according to an embodiment of the present invention, Figure 5 As shown, a neural network optimization circuit is provided, which is applied to a target neural network. The neural network optimization circuit includes: a first conversion unit 51, a second conversion unit 52, a first multiplier 53, a first differentiator 54, a second differentiator 55, a second multiplier 56, an adder 57, an integrator 58, and an error calculation unit 59. Figure 5 The neural network optimization circuit shown is the basic training unit in the embodiment of the present disclosure, that is, the weight of the neural network. Figure 5 In the figure, the brown dotted line represents an arbitrary target neural network, which can be a Transformer neural network, a CNN convolutional neural network, a RNN recurrent neural network, or a fully connected neural network.
[0056] In actual design, the basic unit of the analog circuit neural network can be divided into several different basic structures according to different disturbance types and the introduction method of disturbance signals. The neural network optimization circuit in the embodiment of the present disclosure is suitable for the scenario of processing batch training data. It has the widest applicability and can be extended to networks with shared weights such as batch processing and attention mechanism. Figure 5 The neural network optimization circuit shown is suitable for scenarios where the target neural network is a Transformer neural network, a CNN convolutional neural network, or an RNN recurrent neural network, and each weight participates in the calculation of multiple data at the same time.
[0057] In the disclosed embodiment, a two-layer 2*2 fully connected neural network is taken as an example. The two circles on the left are input neurons, with DC level as input signal, and the output signal is also DC level. The input and output neurons are fully connected by 4 weights. (The connection between the left and right neurons is a solid line). Figure 5 The simulated circuit in the lower center shows the internal structure of the weights.
[0058] Among them, Figure 5 In the embodiment, the first conversion unit 51 is used to convert the target training data into a target DC level signal corresponding to the target training data.
[0059] Specifically, Figure 6As shown, the first conversion unit 51 in the embodiment of the present disclosure may be a D / A digital-to-analog converter, through which the target training data is converted into a target DC level signal corresponding to the target training data, wherein the target training data may be a data sample set of different types, and each data in the data sample set is converted into a target DC level signal, for example, the target training data is represented by M, M = {a1, a2, a3, ...a n}, converting the n data in the data sample set into target DC level signals. In the disclosed embodiment, the target training data is also the input training data. In the disclosed embodiment, converting the target training data into the target DC level signal corresponding to the target training data as the input signal of the neural network is conducive to ensuring the continuity of the analog system signal, thereby enabling the circuit to train the target neural network at high speed.
[0060] Among them, Figure 5 In the embodiment, the second conversion unit 52 is used to convert the label data corresponding to the target training data into a labeling result level signal.
[0061] Specifically, Figure 6 As shown, the second conversion unit 52 in the embodiment of the present disclosure may be a D / A digital-to-analog converter, through which the label data corresponding to the target training data is converted into a labeling result level signal. The labeling result signal of the target training is a label data level signal. Figure 5 In the example, the input data level signal x and the label data level signal y are given at the same time. For example, the target training data is represented by M, M = {a1, a2, a3, ...a n}, then the n data in the data sample set are correspondingly converted into the target data level signal x and the label data level signal y.
[0062] Among them, Figure 5 In the example, the first multiplier 53 is used to perform multiplication based on the target DC level signal corresponding to the target training data and the adder output signal, and output a prediction result signal corresponding to the target training data.
[0063] Specifically, in Figure 5 When the target neural network is inferred, it is multiplied by the weight value level signal at the output of adder 57 to obtain the output value For example, the target DC level signal x corresponding to the target training data is multiplied by the adder output signal (weight value level signal), and the prediction result signal corresponding to the target training data is output.
[0064] Among them, Figure 5 In the example, the error calculation unit 59 is used to calculate the MSE error signal between the prediction result signal and the labeled result signal of the target training data.
[0065] Specifically, in Figure 5 In the example, the error calculation unit 59 and the first differentiator 54 can be an MSE differential unit, which receives the DC level of the label data output by the PC computer via the D / A converter. The MSE differential unit calculates the opposite number of the MSE loss differential, outputs it as a wire, and then feeds it back to the analog circuit neural network. From then on, the circuit forms a closed loop, and the weights of the analog circuit neural network converge in the direction of minimizing the error, thereby completing the training.
[0066] Among them, Figure 5 In the embodiment, the first differentiator 54 is used to receive the MSE error signal and output a first differentiator output signal.
[0067] Specifically, in Figure 5 In the training of the target neural network, the input data level signal x and the label data level signal y are given at the same time, and the output signal of the circuit is The MSE loss is calculated with y and inverted, and then connected to the second multiplier 56 after passing through the first differentiator 54, which represents the inverse of the change in the MSE loss.
[0068] Among them, Figure 5 In the embodiment, the second differentiator 55 is used to receive a plurality of weight change signals of a plurality of weight change data, and output a second differentiator output signal, wherein each weight change signal corresponds to a different weight value, and each weight change data represents the change in time of the weight value corresponding to the weight change data.
[0069] Specifically, in the embodiments of the present disclosure, the multiple weight change signals of the multiple weight change data represent weight disturbance noise signals. For example, when the target neural network is training batch data, there are N noise signals, which are described as multiple weight change signals. If the target neural network has N weight change data, it means that there are N corresponding noise signals, and each noise signal represents the change in the weight value over time.
[0070] Among them, Figure 5 In the embodiment, the second multiplier 56 is used to receive the output signal of the first differentiator 54, the output signal of the second differentiator 55, and output the output signal of the second multiplier 56. The integrator 58 is used to receive the output signal of the second multiplier 56 and output the integrator output signal. The adder 57 is used to receive multiple weight change signals of multiple weight change data, the output signal of the integrator 58, and output the adder output signal.
[0071] Specifically, in Figure 5 In the target neural network, when inferring, the given input data level signal x is multiplied by the weight value level signal at the output end of adder 57 to obtain the output value The reasoning is completed. During training, the input data level signal x and the label data level signal y are given at the same time. The output signal of the circuit is The MSE loss is calculated with y and inverted, and then connected to the second multiplier 56 after passing through the first differentiator 54, which represents the opposite number of the change in the MSE loss. The other end of the second multiplier 56 is connected to the noise through the second differentiator 55. At the same time, the noise is connected to the input end of the adder 57, and the other input end of the adder 57 is the DC level signal of the weight. Therefore, the output of the adder 57 is the weight value (W+ΔW) after the disturbance and update. The weight DC level signal is also the output of the integrator 58, that is, the change in the weight value is determined by the product of the two input quantities of the second multiplier 56 at the bottom. The opposite number of the differential of the MSE loss in one of the quantities represents the change trend of the loss under the current disturbance, and only pays attention to the positive and negative, indicating whether the error has become smaller or larger. The other quantity represents the weight change. The product of the two is the weight change that reduces the error. For example, when the weight changes, if the opposite number of the differential of the MSE loss is positive, it means that the error is getting smaller, and the weight should move in this direction. That is, the weight change should be a positive number multiplied by the original weight change. If the opposite of the differential value of the MSE loss is negative, it means that the error is increasing, and the weight should move in the opposite direction of the weight change. That is, the weight change should be a negative number multiplied by the original weight change. In this way, the circuit can ensure that the weight value always moves in the direction that reduces the error and completes the training of the neural network.
[0072] The neural network optimization circuit in the disclosed embodiment improves the optimization speed of the neural network training device, and at the same time has the capabilities of fast reasoning training, low expansion cost, high parallelism, integrated storage and computing, and on-chip learning. In addition, the neural network optimization circuit in the disclosed embodiment is all implemented using analog circuits, without the need for computers to participate in calculating weight error values multiple times. There are no non-continuous signals such as switch signals and clock signals during the entire training process, and rapid convergence is achieved under the guarantee of strict mathematical theory. The convergence speed is faster than that of existing optimizers (the simulink simulation training speed is 147,900,000 times that of GPU).
[0073] In an optional embodiment, the target neural network includes a plurality of different subnets, each subnet establishes a control chip, each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron. Both the input neuron and the output neuron are DC level signals. In two adjacent subnets, the output neuron of the front subnet is connected to the input neuron of the rear subnet. When the target training data is trained in the target neural network, the MSE error signal is fed back to all subnets of the target neural network through wires.
[0074] Specifically, Figure 7Part B shown in the figure shows the structural framework of the circuit (Analog neural network) in the embodiment of the present disclosure for reasoning. The left side of the circuit is the input end, each pin corresponds to an input neuron, and the DC level is the input signal. The right side of the circuit is the output end, each pin corresponds to an output neuron, and the DC level is the output signal. The input signal is given by a computer-controlled D / A digital-to-analog converter. The output signal is converted into a digital signal by the A / D analog-to-digital converter and transmitted to the computer.
[0075] exist Figure 7 Part D in the figure shows the modular layout of the analog circuit neural network. When the network is large-scale or heterogeneous, the neural network can be divided into different subnets, and each subnet has a dedicated chip. In the forward propagation part, the output neurons of the front subnet are connected to the input neurons of the rear subnet. In the training part, the final error differential wire is fed back to all subnets for training.
[0076] Therefore, the neural network optimization circuit in the embodiment of the present invention trains all target training data at the same time, and performs error optimization on all target training at the same time. It is highly parallel, has a fast calculation speed, has on-chip learning capabilities, and the training process has no non-continuous signals such as switching signals and clock signals. It achieves rapid convergence under the guarantee of strict mathematical theory, uses all analog circuits, and has low expansion costs.
[0077] In the disclosed embodiment, both the input neuron and the output neuron are DC level signals. Circuit storage and calculation are both realized through weight levels, and are DC level signals at the same place, so it belongs to a storage and calculation integrated architecture. At the same time, the DC level signal is conducive to ensuring the continuity of the analog system signal, so that the circuit can train the target neural network at high speed.
[0078] In an optional embodiment, Figure 5 In the embodiment, the input side of the target neural network is connected to the first conversion unit 51, and the output side of the target neural network is connected to the error calculation unit 59.
[0079] In an optional embodiment, if Figure 6 and Figure 7 As shown, it also includes: a computer 50 and a third conversion unit 530, which is used to send a digital signal corresponding to the target training data to the first conversion unit 51, send a digital signal corresponding to the label data of the target training data to the second conversion unit 52, and receive a prediction result signal corresponding to the target training data. The third conversion unit 530 is used to convert the prediction result signal corresponding to the target training data into a digital signal and transmit it to the computer 50.
[0080] Specifically, the first conversion unit 51 and the second conversion unit 52 mentioned above are both D / A digital-to-analog converters, and the third conversion unit 530 is an A / D analog-to-digital converter. Figure 6 The computer 50 in the embodiment provides input data and label data at the same time. The input data is provided to the first conversion unit 51, and the label data is provided to the second conversion unit 52. The computer in the embodiment of the present disclosure only has the function of providing data and does not participate in the weight error calculation. In the weight error calculation process, Figure 5 The circuit implementation shown, therefore, the neural network optimization circuit in the embodiment of the present disclosure is all implemented using analog circuits, without the participation of computers or other digital circuits, and can take advantage of the fast analog computing speed.
[0081] In an optional implementation, when the target neural network is training the target training data, each weight change data is modified to correspond to a different weight value. Therefore, the neural network optimization circuit in the embodiment of the present disclosure has a high degree of parallelism. When the neural network is trained, all weights are modified at the same time, and the time complexity is low, and the time complexity is only 1. The time complexity of only 1 means that all weight units are optimized at the same time. For each weight, the calculation process is only calculated once to complete the optimization of the neural network.
[0082] In a specific example, the neural network optimization circuit in the embodiment of the present disclosure can realize fast reasoning training. Fast reasoning refers to how long it takes for a neural network to obtain a stable output after a given input. Since the weight does not change during the reasoning stage, it is equivalent to each weight unit only performing multiplier calculations. Under ideal device conditions, the entire neural network reasoning time is only constrained by the speed of light, and under non-ideal device conditions, it is affected by device delays. It can generally be completed in ns or sub-ns time. The theoretical guarantee of fast training is that the weight training effect is equivalent to BP, and the direction of weight modification is an unbiased estimate of BP. When the input data and label data are given externally, the weight level of the neural network will converge, and the convergence direction is the negative direction of the gradient of the weight relative to the loss function, that is, the weight modification direction of BP (back propagation). Therefore, as long as the neural network is trainable using BP, it can theoretically be trained using the circuit proposed in the embodiment of the present disclosure, and the training results are consistent. If the problem being trained is a convex problem, the circuit can definitely converge to the optimal solution. The unbiased theoretical proof process is as follows:
[0083] Based on the weight perturbation algorithm, Figure 5 The transfer function of the circuit is as follows:
[0084]
[0085] where W∈R N×1 is the weight parameter, L∈R is the loss function, ξ∈RN×1 For disturbance.
[0086] ξ=∫δdt,δ~N(0,I N ).
[0087] Using the chain rule, we can get the following formula:
[0088]
[0089] in,
[0090] When ξ is small enough, it satisfies When , there is the following formula:
[0091]
[0092] at this time,
[0093]
[0094] n is the number of times the gradient is estimated. In analog circuits, the system is a continuous system. Therefore, in an infinitely short time step, since the electric field is established at the speed of light, if the circuit scale is small enough, n can be considered infinite. Therefore, at any time, the change in weight satisfies the above formula (equivalent to BP).
[0095] like Figure 8 As shown in the figure, a three-layer 2*2*2 neural network with activation function built in simulink is shown. Fig. 9 As shown, it is the loss curve for training two data. In the experimental results, the training of the two data was completed within 1ns. The data was switched 4-9 times. Under the same data and the same network, using GPU training: an average of 12 steps are required, and the average time is 13.6ms. The circuit involved in the embodiment of the present disclosure is 13,600,000 times faster than the digital circuit GPU. In order to ensure the rigor of the experiment, the GPU time only counts the time for forward propagation and backward propagation, and does not include GPU initialization, data initialization, and optimizer initialization time. In addition, the devices involved in the simulink simulation are ideal devices, and delay, error, and noise factors are not considered. Moreover, the GPU is versatile and can be applied to networks of different structures, so the GPU may sacrifice some speed for versatility. Dedicated digital circuits may be faster. But even dedicated digital circuits will not be so many orders of magnitude faster, so this experimental result can prove that analog circuits are much faster than GPUs, as shown in Table 1 below.
[0096] Table 1
[0097]
[0098] To further ensure rigor, the experiment compared different CUDA versions, different GPUs, and different operating systems, and obtained similar experimental results.
[0099] like Fig.10 As shown in the figure, it shows the change of the solution during the training process. It can be seen from the figure that the solution slides to the lowest point along the direction of the steepest descent. This is consistent with the conclusion of the unbiased theory. Fig.11 As shown, it means that the circuit completed the training of the XOR task within 10ns. It also proves that the circuit has a faster training speed. The XOR task is also a very classic task in a neural network, which requires a multi-layer, nonlinear network to learn. The circuit involved in the embodiment of the present disclosure also learned the XOR task within 10ns. This shows that the circuit has the ability to be expanded to other neural network structures. For example, convolutional neural networks, ResNet, Transformer, GPT, etc.
[0100] The low expansion cost mainly considers the complexity of the optimization unit and the impact of networking on efficiency. In terms of the complexity of the optimization unit, the circuits involved in the embodiments of the present disclosure are all in the actual chip design process. Figure 5 The circuit structure shown is infinitely repeated, and the circuit structure is simple. In contrast, the complexity of GPU is mainly reflected in the following aspects: 1. Complex programming and control logic. The GPU contains a complex control unit to manage task scheduling, memory management, error correction, and other system-level functions. 2. Complex data path: The GPU is designed with a complex data path that can handle the movement of large amounts of data, including data caching, prefetching, and efficient data transmission mechanisms. 3. Complex thermal management and power management: High-performance GPUs require complex thermal management systems and power management strategies to cope with heat and power consumption under high loads.
[0101] In terms of networking efficiency, the circuit in the embodiment of the present disclosure can be used Figure 7The structure shown in the figure is expanded. The output neurons of the front subnet of the forward propagation are connected to the input neurons of the rear subnet, and the final error differential wire is fed back to all subnets. In contrast, GPUs are much more complicated when networking and expanding. The main issues involved are: 1. Data transmission delay. Cross-GPU communication: In the model parallel setting, different parts of the model are assigned to different GPUs. This means that the data generated by each GPU during the calculation process (such as activation values and gradients) needs to be passed to other GPUs. When GPUs are connected through the PCIe bus or through the network (in the case of cross-node), data transmission will incur significant delays. Bandwidth limitation: The data transmission speed is limited by the hardware bandwidth. Even high-speed connections such as InfiniBand or NVLink may become a bottleneck when the amount of data is large. 2. Synchronization overhead. Barrier synchronization: In parallel processing, it is necessary to ensure that all processing units have completed the current step before proceeding to the next step of calculation. This synchronization usually requires the use of a barrier synchronization mechanism, which ensures that all GPUs reach the same state before starting the next calculation cycle. Each synchronization causes all GPUs to wait for the slowest one, thereby introducing delays. Consistency and state management: During parallel processing, it is necessary to maintain data consistency, which may involve additional communication and processing overhead to ensure that the data state of all processing units is consistent. 3. The ratio of communication to computation. Computation / communication ratio: This is an important indicator for measuring parallel efficiency, which describes the ratio of the time required to perform calculations to the time required for communication. Ideally, we want this ratio to be as high as possible, meaning that most of the time is spent on actual calculations rather than data transmission. However, in many model parallel scenarios, especially when the model is large or distributed across multiple computing nodes, this ratio may be low.
[0102] High parallelism refers to the proportion of parameters optimized at the same time to all parameters. The higher the proportion, the higher the degree of parallelism of the computing hardware. The higher the degree of parallelism, the higher the computing efficiency. According to Amdahl's law, the proportion of tasks that can be executed in parallel is 1, the number of computing resources is the weight number N, and the Amdahl speedup ratio is N. The advantage of high parallelism is mainly reflected in the time complexity. As shown in Table 2.
[0103] Table 2 Time complexity
[0104] Digital Circuit ACO Forward Propagation <![CDATA[O(LN 2 )]]> O(1) Back Propagation <![CDATA[O(N 2 )]]> O(1)
[0105] L is the number of neural network layers
[0106] N is the number of neurons in the hidden layer
[0107] In the embodiment of the present disclosure, all weights of the analog circuit are optimized simultaneously, so the time complexity is not affected by the network scale. The optimization speed of the digital circuit will slow down as the network scale increases.
[0108] Processing-In-Memory (PIM) is a technology that performs computing tasks directly in storage devices. In the present invention, since the calculation and storage of each weight are the same physical quantity, which is determined by the output level of the integrator, the circuit architecture of the present invention belongs to the processing-in-memory architecture. This architecture breaks through the boundary between the processor and storage in the traditional architecture, allowing data to be processed in the storage stage, reducing the need for data transmission between the processor and the memory. The present invention belongs to the processing-in-memory architecture and has its advantages. Processing-in-memory has several significant advantages and is particularly suitable for data-intensive applications and large-scale data processing scenarios. 1. Reduce data transmission requirements: In traditional computing architectures, the CPU needs to frequently read and write data from the memory, and this data movement will cause significant energy consumption and delay. The processing-in-memory design significantly reduces the movement of data by directly processing data in the memory, thereby reducing energy consumption and increasing data processing speed. 2. Reduce energy consumption: Data transmission is one of the main sources of energy consumption in modern computing systems. The processing-in-memory technology can effectively reduce the energy consumption of the entire system by reducing the transmission of data on the system bus and other interfaces. 3. Improve performance: Since the speed of data processing is limited by the speed of data transmission, the CPU in the traditional architecture often becomes idle while waiting for data. Storage and computing can reduce the waiting time of the processor, allowing data processing tasks to be completed faster, thereby improving overall computing performance. 4. Scalability and flexibility: Storage and computing technology provides greater flexibility, and can customize the configuration of memory and computing resources according to application requirements. This allows it to be optimized for specific applications, such as big data analysis and machine learning. 5. Simplify system design: By moving computing tasks to memory, the overall hardware design of the system can be simplified, reducing dependence on high-speed data buses and other high-cost hardware, while potentially reducing the physical size of the system. 6. Improve parallel processing capabilities: The storage and computing architecture allows computing tasks to be performed in parallel in multiple memory modules, greatly enhancing the ability of parallel processing. This is especially important for applications that require high parallelism. 7. Improve data processing security: By processing data directly in memory, data transmission within the system can be reduced, thereby reducing the potential risk of data leakage and enhancing data security.
[0109] On-chip learning is a concept that involves implementing learning and reasoning tasks directly on a microchip (usually a specialized integrated circuit or processor). The present invention can be trained in real time on edge devices due to its fast training characteristics to achieve on-chip learning. This approach is particularly important in the field of artificial intelligence, focusing on integrating the training and execution of machine learning models into the hardware devices themselves, rather than relying on external large-scale computing systems or cloud infrastructure. On-chip learning is an important part of edge computing and intelligent hardware design. It can greatly improve the speed and efficiency of data processing while reducing the demand and latency of data transmission.
[0110] In summary, the neural network optimization circuit in the embodiment of the present disclosure has the following advantages:
[0111] 1. Fast reasoning and training: Use analog circuits to reason and train neural networks, while retaining the continuity of analog system signals, so that the circuit can train neural networks at high speed.
[0112] 2. Low expansion cost. Neural networks can be expanded through direct connection, and circuits do not need signal conversion (such as nvlink), encoding and decoding, etc. Secondly, the analog circuit neural network has integrated storage and computing, and the weight structure is simple, which is low cost (compared to the GPU storage and computing separation structure).
[0113] 3. Highly parallel, when the neural network is trained, all weights are modified at the same time, and the time complexity is low. The time complexity is only 1.
[0114] 4. Storage and computing are integrated. Circuit storage and calculation are both realized through weight levels, and are the same DC level signal, so it belongs to the storage and computing integrated architecture.
[0115] 5. On-chip learning: Currently, there is a lack of edge computing devices that can perform on-chip learning. Current on-chip learning devices generally have problems such as low training speed and only being able to train some weights. The disclosed embodiments can achieve on-chip learning of all weights.
[0116] 6. Fast convergence speed, no need for clock drive circuit.
[0117] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A neural network optimization circuit, characterized in that: Applied to target neural networks, including: A first conversion unit, configured to convert the target training data into a target DC level signal corresponding to the target training data; A second conversion unit, used to convert the label data corresponding to the target training data into a labeling result level signal; A first multiplier, configured to multiply a target DC level signal corresponding to the target training data by the adder output signal, and output a prediction result signal corresponding to the target training data; An error calculation unit, used to calculate an MSE error signal between the prediction result signal and the marked result level signal; A first differentiator, configured to receive the MSE error signal and output a first differentiator output signal; a second differentiator, configured to receive a plurality of weight change signals of a plurality of weight change data, and output a second differentiator output signal; a second multiplier, configured to receive the first differentiator output signal, the second differentiator output signal, and output a second multiplier output signal; an integrator, configured to receive the second multiplier output signal and output an integrator output signal; an adder, configured to receive a plurality of weight change signals of the plurality of weight change data, the integrator output signal, and output an adder output signal; Each weight change signal corresponds to a different weight value, and each weight change data represents the change in time of the weight value corresponding to the weight change data.
2. The neural network optimization circuit according to claim 1, characterized in that: The target neural network includes a plurality of different subnetworks, each subnetwork establishes a control chip, each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron.
3. The neural network optimization circuit according to claim 2, characterized in that: The input neuron and the output neuron are both DC level signals.
4. The neural network optimization circuit according to claim 2, characterized in that: In two adjacent subnetworks, the output neurons of the front subnetwork are connected to the input neurons of the rear subnetwork.
5. The neural network optimization circuit according to claim 2, characterized in that: When the target training data is trained in the target neural network, the MSE error signal is fed back to all sub-networks of the target neural network through wires.
6. The neural network optimization circuit according to claim 1, characterized in that: The target neural network input side is connected to the first conversion unit, and the target neural network output side is connected to the error calculation unit.
7. The neural network optimization circuit according to claim 1, characterized in that: Also includes: The computer is used to send a digital signal corresponding to the target training data to the first conversion unit, send a digital signal corresponding to the label data of the target training data to the second conversion unit, and receive a prediction result signal corresponding to the target training data.
8. The neural network optimization circuit according to claim 7, characterized in that: Also includes: The third conversion unit is used to convert the prediction result signal corresponding to the target training data into a digital signal and transmit it to the computer.
9. The neural network optimization circuit according to claim 1, characterized in that: When training the target training data, the target neural network simultaneously modifies different weight values corresponding to each weight change data.
10. The neural network optimization circuit according to any one of claims 1 to 9, characterized in that: The target neural network includes: a Transformer neural network, a CNN convolutional neural network, a RNN recurrent neural network, or a fully connected neural network.
Citation Information
Patent Citations
Hall strip synapse unsupervised learning circuit based on pulse amplitude
CN114676833A
VHF signal propagation path error prediction method and system
CN114742265A
Atomic air chamber temperature active disturbance rejection control method based on LSTM neural network
CN115525076A
System and method using neural networks for analog-to-information processors
US10970441B1
Cited By
Neural network optimization circuit
WO2026037447A1
