Neural network optimization circuit

By using the perturbation constraint mechanism of analog circuits and fully connected neural networks in neural network optimization circuits, the convergence efficiency and stability problems of neural networks in batch data processing are solved, and a fast and stable training process and low scaling cost are achieved.

CN119026651APending Publication Date: 2024-11-26INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411117006.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the process of processing batch data of neural networks, the convergence efficiency and stability of data cannot be guaranteed, especially due to the influence of weight perturbation signals.

Method used

A neural network optimization circuit is designed, including the first conversion unit, the second conversion unit, the multiplier, the differentiator, the integrator and the adder, which achieves rapid convergence through the analog circuit and connects each layer of the neural network through a fully connected neural network. The weight corresponding to the connection relationship between the same output neuron is added to the same perturbation, reducing the total perturbation amount and improving convergence efficiency and stability.

Benefits of technology

It realizes highly parallel computing in training single-object training data scenarios, quickly converges, has on-chip learning ability, no switching signals and clock signals during the training process, low expansion cost, faster convergence speed than existing optimizers, and improves the network stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026651A_ABST
    Figure CN119026651A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of circuits, and discloses a neural network optimization circuit, which is used for a target neural network, performs error optimization on single target training data in a scene of training the single target training data, is high in parallelism and high in calculation speed, has on-chip learning ability, does not have discontinuous signals such as switching signals and clock signals in a training process, and improves the training efficiency. And rapid convergence is realized under the guarantee of a strict mathematical theory, analog circuits are completely used, and the expansion cost is low. Moreover, a third multiplier is added in front of an adder, and the constraint of the network is considered by adding the same disturbance to the weight corresponding to the connection relationship between each layer of neural network and the same output neuron in the full-connection neural network, so that the total disturbance quantity is reduced, and the convergence efficiency is maximized. Besides, the second differentiator is arranged from the disturbance source to the output position of the adder, the original convergence direction of the circuit can be kept, the requirement for disturbance power is reduced, and therefore the stability of the network is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of circuit technology, and in particular to a neural network optimization circuit. Background Art

[0002] In recent years, with the development of AI technology, the demand for computing power has increased rapidly, and improving optimization speed is an important development direction. Therefore, the application of neural networks is becoming more and more extensive.

[0003] In the related art, when the neural network processes batch data, the convergence efficiency and stability of the neural network data cannot be guaranteed. Summary of the invention

[0004] In view of this, the present invention provides a neural network optimization circuit to solve the problem that the convergence efficiency and stability of the neural network data cannot be guaranteed during the process of the neural network processing batch data.

[0005] According to a first aspect, an embodiment of the present disclosure provides a neural network optimization circuit, which is applied to a target neural network, including:

[0006] A first conversion unit, configured to convert the target training data into a target DC level signal corresponding to the target training data;

[0007] A second conversion unit, used to convert the label data corresponding to the target training data into a labeling result level signal;

[0008] A first multiplier, configured to multiply a target DC level signal corresponding to the target training data by the adder output signal, and output a prediction result signal corresponding to the target training data;

[0009] An error calculation unit, used to calculate the MSE error signal between the prediction result signal and the labeled result signal of the target training data;

[0010] A first differentiator, configured to receive the MSE error signal and output a first differentiator output signal;

[0011] a second differentiator, configured to receive the adder output signal and output a second differentiator output signal;

[0012] a second multiplier, configured to receive the first differentiator output signal, the second differentiator output signal, and output the second multiplier output signal;

[0013] an integrator, configured to receive the second multiplier output signal and output an integrator output signal;

[0014] an adder, used for receiving the second adder output signal and the integrator output signal;

[0015] a third multiplier, configured to receive a target DC level signal corresponding to the target training data, a disturbance signal of each weight, and output a third multiplier output signal;

[0016] Among them, the weights corresponding to the connection relationship between each layer of the neural network in the target neural network and the same output neuron come from the same disturbance signal.

[0017] In an optional embodiment, the target neural network includes a plurality of different subnetworks, each subnetwork establishes a control chip, each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron.

[0018] In an optional implementation, both the input neuron and the output neuron are DC level signals.

[0019] In an optional implementation, in two adjacent sub-networks, the output neurons of the front sub-network are connected to the input neurons of the rear sub-network.

[0020] In an optional implementation, when the target training data is trained in the target neural network, the MSE error signal is fed back to all sub-networks of the target neural network through a wire.

[0021] In an optional embodiment, the input side of the target neural network is connected to the first conversion unit, and the output side of the target neural network is connected to the second conversion unit.

[0022] In an optional implementation, the neural network optimization circuit in the embodiment of the present disclosure further includes:

[0023] The computer is used to send a digital signal corresponding to the target training data to a first conversion unit, send a digital signal corresponding to the label data of the target training data to a second conversion unit, and receive a prediction result signal corresponding to the target training data.

[0024] In an optional implementation, it further includes: a third conversion unit, configured to convert the prediction result signal corresponding to the target training data into a digital signal and transmit the digital signal to the computer.

[0025] In an optional implementation, when training the target training data, the target neural network simultaneously modifies each weight change data to correspond to a different weight value.

[0026] In an optional implementation, the target neural network includes: a fully connected neural network.

[0027] The technical solution of the present invention has the following advantages:

[0028] The present invention discloses a neural network optimization circuit, which is applied to a target neural network, and includes: a first conversion unit, which is used to convert target training data into a target DC level signal corresponding to the target training data; a second conversion unit, which is used to convert label data corresponding to the target training data into a labeling result level signal; a first multiplier, which is used to multiply the target DC level signal corresponding to the target training data and the adder output signal, and output a prediction result signal corresponding to the target training data; an error calculation unit, which is used to calculate the MSE error signal between the prediction result signal and the labeling result signal of the target training data; a first differentiator, which is used to receive the MSE error signal and output the first differentiator. Output signal; a second differentiator for receiving the adder output signal and outputting the second differentiator output signal; a second multiplier for receiving the first differentiator output signal, the second differentiator output signal, and outputting the second multiplier output signal; an integrator for receiving the second multiplier output signal and outputting the integrator output signal; an adder for receiving the second adder output signal and the integrator output signal; a third multiplier for receiving the target DC level signal corresponding to the target training data, the disturbance signal of each weight, and outputting the third multiplier output signal; wherein the weights of each layer of the neural network connected to the same output neuron are a group, and the disturbance signal of each group of weights is the same. In the scenario of training a single target training data, the present invention performs error optimization on the single target training data, is highly parallel, has a fast calculation speed, has on-chip learning capabilities, has no non-continuous signals such as switch signals and clock signals in the training process, achieves rapid convergence under the guarantee of strict mathematical theory, uses all analog circuits, and has low expansion cost. In addition, by adding a third multiplier before the adder and adding the same disturbance to the weights corresponding to the connection relationship between each layer of the neural network and the same output neuron in the fully connected neural network, the network constraints are considered, the total disturbance is reduced, and the convergence efficiency is maximized. In addition, by post-placing the second differentiator from the disturbance source to the output of the adder, the original convergence direction of the circuit can be maintained, the requirement for disturbance power can be reduced, and the stability of the network can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 is a schematic diagram of a neural network optimization circuit training process according to an embodiment of the present invention;

[0031] Figure 2is a schematic diagram of the training process and structural framework of a neural network optimization circuit according to an embodiment of the present invention;

[0032] Figure 3 It is a schematic diagram of the reasoning process and structural framework of the neural network optimization circuit according to an embodiment of the present invention and a schematic diagram of the modular layout of the internal of the analog circuit neural network according to an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram showing a three-layer 2*2*2 neural network with an activation function built in Simulink according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of a loss curve for training two data according to an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of the change of the solution during the training process according to an embodiment of the present invention;

[0036] Figure 7 It is a training schematic diagram of a circuit according to an embodiment of the present invention completing an XOR task within 10 ns. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0038] In the related technology, although some neural networks can train and process batch training data, they are not affected by network constraints due to the disturbance signals of the weights and cannot guarantee the convergence efficiency of the neural network data; and the introduction of strong disturbances will also affect the stability of the network.

[0039] In view of this, according to an embodiment of the present invention, Figure 1 As shown, a neural network optimization circuit is provided, which is applied to a target neural network. The neural network optimization circuit includes: a first conversion unit 11, a second conversion unit 12, a first multiplier 13, a first differentiator 14, a second differentiator 15, a second multiplier 16, an adder 17, an integrator 18, an error calculation unit 19, and a third multiplier 10. Figure 1 The neural network optimization circuit shown is the basic training unit in the embodiment of the present disclosure, that is, the weight of the neural network. Figure 1In the figure, the brown dotted line represents an arbitrary target neural network, which can be a fully connected neural network.

[0040] In actual design, according to different disturbance types and the introduction methods of disturbance signals, the basic units of analog circuit neural networks can be divided into several different basic structures. The neural network optimization circuit in the disclosed embodiment is suitable for the scenario of single data, such as a fully connected neural network, which fully considers the constraints of the network and maximizes the convergence efficiency.

[0041] In the embodiment of the present disclosure, Figure 1 In the figure, a two-layer 2*2 fully connected neural network is taken as an example. The two circles on the left are input neurons, with DC level as input signal, and the output signal is also DC level. The input and output neurons are fully connected by 4 weights. (The connection between the left and right neurons is a solid line). Figure 1 The simulated circuit in the lower center shows the internal structure of the weights.

[0042] Among them, Figure 1 In the embodiment, the first conversion unit 11 is used to convert the target training data into a target DC level signal corresponding to the target training data.

[0043] Specifically, Figure 2 As shown, the first conversion unit 11 in the embodiment of the present disclosure may be a D / A digital-to-analog converter, through which the target training data is converted into a target DC level signal corresponding to the target training data, wherein the target training data may be a data sample set of different types, and each data in the data sample set is converted into a target DC level signal, for example, the target training data is represented by M, M = {a1, a2, a3, ... a n}, converting the n data in the data sample set into target DC level signals. In the disclosed embodiment, the target training data is also the input training data. In the disclosed embodiment, converting the target training data into the target DC level signal corresponding to the target training data as the input signal of the neural network is conducive to ensuring the continuity of the analog system signal, thereby enabling the circuit to train the target neural network at high speed.

[0044] The second conversion unit 12 is used to convert the label data corresponding to the target training data into a labeling result level signal.

[0045] Specifically, in Figure 2 In the embodiment of the present disclosure, the second conversion unit 12 may be a D / A digital-to-analog converter, through which the label data corresponding to the target training data is converted into a labeling result level signal. The labeling result signal of the target training is a label data level signal. Figure 1In the example, the input data level signal x and the label data level signal y are given at the same time. For example, the target training data is represented by M, M = {a1, a2, a3, ... a n}, then the n data in the data sample set are correspondingly converted into the target data level signal x and the label data level signal y.

[0046] Among them, Figure 1 In the example, the first multiplier 13 is used to perform multiplication based on the target DC level signal corresponding to the target training data and the adder output signal, and output a prediction result signal corresponding to the target training data.

[0047] Specifically, in Figure 1 When the target neural network is inferred, it is multiplied by the weight value level signal at the output of adder 17 to obtain the output value For example, the target DC level signal x corresponding to the target training data is multiplied by the adder output signal (weight value level signal), and the prediction result signal corresponding to the target training data is output.

[0048] Among them, Figure 1 In the example, the error calculation unit 19 is used to calculate the MSE error signal between the prediction result signal and the labeled result signal of the target training data.

[0049] Specifically, in Figure 1 In the example, the error calculation unit 19 and the first differentiator 14 can be an MSE differential unit, which simultaneously receives the DC level of the label data output by the PC computer through the D / A converter. The MSE differential unit calculates the opposite of the MSE loss differential quantity, outputs it as a wire, and then feeds it back to the analog circuit neural network. From then on, the circuit forms a closed loop, and the weights of the analog circuit neural network converge in the direction of minimizing the error, thereby completing the training.

[0050] Among them, Figure 1 In the embodiment, the first differentiator 14 is used to receive the MSE error signal and output a first differentiator output signal.

[0051] Specifically, in Figure 1 In the example, when the target neural network is being trained, the input data level signal x and the label data level signal y are given at the same time. The output signal ∧y of the circuit is used to calculate the MSE loss and invert it, and then passes through the first differentiator 14 and is connected to the second multiplier 16, which represents the inverse of the change in the MSE loss.

[0052] Among them, Figure 1In the embodiment, the second differentiator 15 is used to receive the adder output signal and output the second differentiator output signal. The second multiplier 16 is used to receive the first differentiator output signal, the second differentiator output signal, and output the second multiplier output signal. The integrator 18 is used to receive the second multiplier output signal and output the integrator output signal; the adder 17 is used to receive the second adder output signal and the integrator output signal; the third multiplier 10 is used to receive the target DC level signal corresponding to the target training data, the disturbance signal of each weight, and output the third multiplier output signal; wherein the weights corresponding to the connection relationship between each layer of the neural network in the target neural network and the same output neuron come from the same disturbance signal.

[0053] Specifically, in Figure 1 In the example, when the target neural network is inferring, the input data level signal x is given, and the weight value level signal at the output end of the adder 17 is multiplied to obtain the output value ∧y, and the inference can be completed. During training, the input data level signal x and the label data level signal y are given at the same time. The output signal ∧y of the circuit is calculated with y and the MSE loss is negated, and then connected to the second multiplier 16 after passing through the first differentiator 14, indicating the opposite number of the change in the MSE loss. The other end of the second multiplier 16 is connected to the output end of the second differentiator 15. At the same time, the noise is connected to the input end of the third multiplier 10, and the other input end of the third multiplier 10 is the target DC level signal corresponding to the target training data. The output end of the third multiplier 10 is connected to the adder 17, so the output of the adder 17 is the weight value (W+ΔW) after disturbance and update. The weight DC level signal is also the output of the integrator 18, that is, the change in the weight value is determined by the product of the two input quantities of the second multiplier 16 at the bottom. One of the quantities, the opposite of the differential of the MSE loss, represents the changing trend of the loss under the current disturbance, focusing only on the positive and negative, indicating whether the error has become smaller or larger. The other quantity represents the weight change. The product of the two is the weight change that reduces the error. For example, when the weight changes, if the opposite of the differential of the MSE loss is positive, indicating that the error is decreasing, the weight should move in this direction. That is, the weight change should be a positive number multiplied by the original weight change. If the opposite of the differential of the MSE loss is negative, indicating that the error is increasing, the weight should move in the opposite direction of the weight change. That is, the weight change should be a negative number multiplied by the original weight change. In this way, the circuit can ensure that the weight value always moves in the direction that reduces the error, completing the training of the neural network.

[0054] The neural network optimization circuit in the disclosed embodiment improves the optimization speed of the neural network training device, and at the same time has the capabilities of fast reasoning training, low expansion cost, high parallelism, integrated storage and computing, and on-chip learning. In addition, the neural network optimization circuit in the disclosed embodiment is all implemented using analog circuits, without the need for computers to participate in calculating weight error values ​​multiple times. There are no non-continuous signals such as switch signals and clock signals during the entire training process, and rapid convergence is achieved under the guarantee of strict mathematical theory. The convergence speed is faster than that of existing optimizers (the simulink simulation training speed is 147,900,000 times that of GPU).

[0055] In an optional embodiment, the target neural network in the disclosed embodiment includes a plurality of different subnets, each subnet establishes a control chip, each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron. Both the input neuron and the output neuron are DC level signals. In two adjacent subnets, the output neuron of the front subnet is connected to the input neuron of the rear subnet. When training the target training data in the target neural network, the MSE error signal is fed back to all subnets of the target neural network through wires.

[0056] Specifically, Figure 3 Part B shown in the figure shows the structural framework of the circuit (Analog neural network) in the embodiment of the present disclosure for reasoning. The left side of the circuit is the input end, each pin corresponds to an input neuron, and the DC level is the input signal. The right side of the circuit is the output end, each pin corresponds to an output neuron, and the DC level is the output signal. The input signal is given by a computer-controlled D / A digital-to-analog converter. The output signal is converted into a digital signal by the A / D analog-to-digital converter and transmitted to the computer.

[0057] exist Figure 3 Part D in the figure shows the modular layout of the analog circuit neural network. When the network is large-scale or heterogeneous, the neural network can be divided into different subnets, and each subnet has a dedicated chip. In the forward propagation part, the output neurons of the front subnet are connected to the input neurons of the rear subnet. In the training part, the final error differential wire is fed back to all subnets for training.

[0058] for Figure 1 For example, it is applied to a single data scenario. That is, the brown dotted box is a 2*2 neural network. At this time, the two circles on the left are input neurons, and each inputs a level signal as input data x. When x is only one piece of data, it is called a single data scenario. In this scenario, the network can perform reasoning and training on a single piece of data. For example, in a fully connected neural network, the weight W is defined as ijrepresents the weight connecting the i-th input neuron and the j-th output neuron, W 12 ,W 22 ,...W n2 The perturbation of the weight input is the same. Assuming that the number of neurons in the input layer and the output layer is n, the total perturbation introduced by the fully connected neural network in this layer is n.

[0059] Therefore, the neural network optimization circuit in the embodiment of the present disclosure performs error optimization on single target training in the scenario of training single target training data. It is highly parallel and has fast calculation speed. It has on-chip learning ability. There are no non-continuous signals such as switch signals and clock signals in the training process. It achieves rapid convergence under the guarantee of strict mathematical theory. All analog circuits are used, and the expansion cost is low. In addition, a third multiplier is added before the adder, and a single target training data is trained through a fully connected neural network, that is, the weight corresponding to the connection relationship between each layer of the neural network and the same output neuron in the fully connected neural network is added with the same disturbance mode to consider the constraints of the network, reduce the total disturbance amount, and maximize the convergence efficiency. In addition, the second differentiator is placed from the disturbance source to the output position of the adder, which can maintain the original convergence direction of the circuit, reduce the requirement for disturbance power, and thus improve the stability of the network.

[0060] In the disclosed embodiment, both the input neuron and the output neuron are DC level signals. Circuit storage and calculation are both realized through weight levels, and are DC level signals at the same place, so it belongs to a storage and calculation integrated architecture. At the same time, the DC level signal is conducive to ensuring the continuity of the analog system signal, so that the circuit can train the target neural network at high speed.

[0061] In an optional embodiment, Figure 1 In the embodiment, the input side of the target neural network is connected to the first conversion unit 11, and the output side of the target neural network is connected to the error calculation unit 19.

[0062] In an optional embodiment, if Figure 2 and Figure 3 As shown, it also includes: a computer 20 and a third conversion unit 23. The computer 20 is used to send a digital signal corresponding to the target training data to the first conversion unit 11, send a digital signal corresponding to the label data of the target training data to the second conversion unit 12, and receive a prediction result signal corresponding to the target training data. The third conversion unit 23 is used to convert the prediction result signal corresponding to the target training data into a digital signal and transmit it to the computer 20. .

[0063] Specifically, the first conversion unit 11 and the second conversion unit 12 mentioned above are both D / A digital-to-analog converters, and the third conversion unit 23 is an A / D analog-to-digital converter. Figure 3The computer 20 in the embodiment of the present disclosure provides input data and label data at the same time. The input data is provided to the first conversion unit 11, and the label data is provided to the second conversion unit 12. The computer in the embodiment of the present disclosure only has the function of providing data and does not participate in the weight error calculation. In the weight error calculation process, Figure 1 The circuit implementation shown, therefore, the neural network optimization circuit in the embodiment of the present disclosure is all implemented using analog circuits, without the participation of computers or other digital circuits, and can take advantage of the fast analog computing speed.

[0064] In an optional implementation, when the target neural network is training the target training data, each weight change data is modified to correspond to a different weight value. Therefore, the neural network optimization circuit in the embodiment of the present disclosure has a high degree of parallelism. When the neural network is trained, all weights are modified at the same time, and the time complexity is low, and the time complexity is only 1. The time complexity of only 1 means that all weight units are optimized at the same time. For each weight, the calculation process is only calculated once to complete the optimization of the neural network.

[0065] In a specific example, the neural network optimization circuit in the embodiment of the present disclosure can realize fast reasoning training. Fast reasoning refers to how long it takes for the neural network to obtain a stable output after a given input. Since the weight does not change during the reasoning stage, it is equivalent to each weight unit only performing multiplier calculations. Under ideal device conditions, the entire neural network reasoning time is only constrained by the speed of light, and under non-ideal device conditions, it is affected by device delays. It can generally be completed in ns or sub-ns time. The theoretical guarantee of fast training is that the weight training effect is equivalent to BP, and the direction of weight modification is an unbiased estimate of BP. When the input data and label data are given externally, the weight level of the neural network will converge, and the convergence direction is the negative direction of the gradient of the weight relative to the loss function, that is, the weight modification direction of BP (back propagation). Therefore, as long as the neural network is trainable using BP, it can theoretically be trained using the circuit proposed in the embodiment of the present disclosure, and the training results are consistent. If the problem being trained is a convex problem, the circuit can definitely converge to the optimal solution. The unbiased theoretical proof process is as follows:

[0066] In the neural network optimization circuit of the embodiment of the present disclosure, Figure 1 In the example, a third multiplier 10 is added before the adder 17, and the target DC level signal corresponding to the target training data and the disturbance signal of each weight variable are transmitted to the third multiplier 10 through the third multiplier 10. At the same time, the second differentiator is placed from the disturbance source to the position of the adder output. That is, it is proved by the following formula:

[0067]

[0068] Its convergence is proved as follows:

[0069]

[0070] make

[0071]

[0072] The significance of this operation is to maintain the original convergence direction of the circuit, reduce the requirement for disturbance power, and improve the stability of the network.

[0073] As for the fast training capability, since the system is a continuous system in the analog circuit, the learning rate can be considered infinitely small. According to the convergence theorem, the continuous convergence of the circuit can be guaranteed. And because the electric field is established at the speed of light, when the circuit scale is small enough, the iteration speed (the number of iterations per unit time) can be considered infinite. Therefore, under ideal device conditions, the circuit can theoretically achieve extremely fast training speed.

[0074] In the embodiments of the present disclosure, Figure 4 As shown in the figure, a three-layer 2*2*2 neural network with activation function is built in simulink. Figure 5 As shown, it is the loss curve for training two data. In the experimental results, the training of the two data was completed within 1ns. The data was switched 4-9 times. Under the same data and the same network, using GPU training: an average of 12 steps are required, and the average time is 13.6ms. The circuit involved in the embodiment of the present disclosure is 13,600,000 times faster than the digital circuit GPU. In order to ensure the rigor of the experiment, the GPU time only counts the time for forward propagation and backward propagation, and does not include GPU initialization, data initialization, and optimizer initialization time. In addition, the devices involved in the simulink simulation are ideal devices, and delay, error, and noise factors are not considered. Moreover, the GPU is universal, it can be applied to networks of different structures, so the GPU may sacrifice some speed for universality. Dedicated digital circuits may be faster. But even dedicated digital circuits will not be so many orders of magnitude faster, so this experimental result can prove that analog circuits are much faster than GPUs, as shown in Table 1 below.

[0075] Table 1

[0076]

[0077] To further ensure rigor, the experiment compared different CUDA versions, different GPUs, and different operating systems, and obtained similar experimental results.

[0078] like Figure 6As shown in the figure, it shows the change of the solution during the training process. It can be seen from the figure that the solution slides to the lowest point along the direction of the steepest descent. This is consistent with the conclusion of the unbiased theory. Figure 7 As shown, it means that the circuit completed the training of the XOR task within 10ns. It also proves that the circuit has a faster training speed. The XOR task is also a very classic task in a neural network, which requires a multi-layer, nonlinear network to learn. The circuit involved in the embodiment of the present disclosure also learned the XOR task within 10ns. This shows that the circuit has the ability to be expanded to other neural network structures. For example, convolutional neural networks, ResNet, Transformer, GPT, etc.

[0079] In terms of networking efficiency, the circuit in the embodiment of the present disclosure can be used Figure 3 The structure shown in the figure is expanded. The output neurons of the front subnet of the forward propagation are connected to the input neurons of the rear subnet, and the final error differential wire is fed back to all subnets. In contrast, GPUs are much more complicated when networking and expanding. The main issues involved are: 1. Data transmission delay. Cross-GPU communication: In the model parallel setting, different parts of the model are assigned to different GPUs. This means that the data generated by each GPU during the calculation process (such as activation values ​​and gradients) needs to be passed to other GPUs. When GPUs are connected through the PCIe bus or through the network (in the case of cross-node), data transmission will incur significant delays. Bandwidth limitation: The data transmission speed is limited by the hardware bandwidth. Even high-speed connections such as InfiniBand or NVLink may become a bottleneck when the amount of data is large. 2. Synchronization overhead. Barrier synchronization: In parallel processing, it is necessary to ensure that all processing units have completed the current step before proceeding to the next step of calculation. This synchronization usually requires the use of a barrier synchronization mechanism, which ensures that all GPUs reach the same state before starting the next calculation cycle. Each synchronization causes all GPUs to wait for the slowest one, thereby introducing delays. Consistency and state management: During parallel processing, it is necessary to maintain data consistency, which may involve additional communication and processing overhead to ensure that the data state of all processing units is consistent. 3. The ratio of communication to computation. Computation / communication ratio: This is an important indicator for measuring parallel efficiency, which describes the ratio of the time required to perform calculations to the time required for communication. Ideally, we want this ratio to be as high as possible, meaning that most of the time is spent on actual calculations rather than data transmission. However, in many model parallel scenarios, especially when the model is large or distributed across multiple computing nodes, this ratio may be low.

[0080] High parallelism refers to the proportion of parameters optimized at the same time to all parameters. The higher the proportion, the higher the degree of parallelism of the computing hardware. The higher the degree of parallelism, the higher the computing efficiency. According to Amdahl's law, the proportion of tasks that can be executed in parallel is 1, the number of computing resources is the weight number N, and the Amdahl speedup ratio is N. The advantage of high parallelism is mainly reflected in the time complexity. As shown in Table 2.

[0081] Table 2

[0082] Time Complexity

[0083] Digital Circuits ACO Forward Propagation <![CDATA[O(LN 2 )]]> O(1) Back Propagation <![CDATA[O(N 2 )]]> O(1)

[0084] L is the number of neural network layers

[0085] N is the number of neurons in the hidden layer

[0086] In the embodiment of the present disclosure, all weights of the analog circuit are optimized simultaneously, so the time complexity is not affected by the network scale. The optimization speed of the digital circuit will slow down as the network scale increases.

[0087] Processing-In-Memory (PIM) is a technology that performs computing tasks directly in storage devices. In the present invention, since the calculation and storage of each weight are the same physical quantity, which is determined by the output level of the integrator, the circuit architecture of the present invention belongs to the processing-in-memory architecture. This architecture breaks through the boundary between the processor and storage in the traditional architecture, allowing data to be processed in the storage stage, reducing the need for data transmission between the processor and the memory. The present invention belongs to the processing-in-memory architecture and has its advantages. Processing-in-memory has several significant advantages and is particularly suitable for data-intensive applications and large-scale data processing scenarios. 1. Reduce data transmission requirements: In traditional computing architectures, the CPU needs to frequently read and write data from the memory, and this data movement will cause significant energy consumption and delay. The processing-in-memory design significantly reduces the movement of data by directly processing data in the memory, thereby reducing energy consumption and increasing data processing speed. 2. Reduce energy consumption: Data transmission is one of the main sources of energy consumption in modern computing systems. The processing-in-memory technology can effectively reduce the energy consumption of the entire system by reducing the transmission of data on the system bus and other interfaces. 3. Improve performance: Since the speed of data processing is limited by the speed of data transmission, the CPU in the traditional architecture often becomes idle while waiting for data. Storage and computing can reduce the waiting time of the processor, allowing data processing tasks to be completed faster, thereby improving overall computing performance. 4. Scalability and flexibility: Storage and computing technology provides greater flexibility, and can customize the configuration of memory and computing resources according to application requirements. This allows it to be optimized for specific applications, such as big data analysis and machine learning. 5. Simplify system design: By moving computing tasks to memory, the overall hardware design of the system can be simplified, reducing dependence on high-speed data buses and other high-cost hardware, while potentially reducing the physical size of the system. 6. Improve parallel processing capabilities: The storage and computing architecture allows computing tasks to be performed in parallel in multiple memory modules, greatly enhancing the ability of parallel processing. This is especially important for applications that require high parallelism. 7. Improve data processing security: By processing data directly in memory, data transmission within the system can be reduced, thereby reducing the potential risk of data leakage and enhancing data security.

[0088] On-chip learning is a concept that involves implementing learning and reasoning tasks directly on a microchip (usually a specialized integrated circuit or processor). The present invention can be trained in real time on edge devices due to its fast training characteristics to achieve on-chip learning. This approach is particularly important in the field of artificial intelligence, focusing on integrating the training and execution of machine learning models into the hardware devices themselves, rather than relying on external large-scale computing systems or cloud infrastructure. On-chip learning is an important part of edge computing and intelligent hardware design. It can greatly improve the speed and efficiency of data processing while reducing the demand and latency of data transmission.

[0089] In summary, the neural network optimization circuit in the embodiment of the present disclosure has the following advantages:

[0090] 1. Fast reasoning and training: Use analog circuits to reason and train neural networks, while retaining the continuity of analog system signals, so that the circuit can train neural networks at high speed.

[0091] 2. Low expansion cost. Neural networks can be expanded through direct connection, and circuits do not need signal conversion (such as nvlink), encoding and decoding, etc. Secondly, the analog circuit neural network has integrated storage and computing, and the weight structure is simple, which is low cost (compared to the GPU storage and computing separation structure).

[0092] 3. Highly parallel, when the neural network is trained, all weights are modified at the same time, and the time complexity is low. The time complexity is only 1.

[0093] 4. Storage and computing are integrated. Circuit storage and calculation are both realized through weight levels, and are the same DC level signal, so it belongs to the storage and computing integrated architecture.

[0094] 5. On-chip learning: Currently, there is a lack of edge computing devices that can perform on-chip learning. Current on-chip learning devices generally have problems such as low training speed and only being able to train some weights. The disclosed embodiments can achieve on-chip learning of all weights.

[0095] 6. Fast convergence speed, no need for clock drive circuit.

[0096] 7. By adding the weights of each layer of the neural network connected to the same output neuron in the fully connected neural network, the constraints of the network are considered, the total disturbance is reduced, and the convergence efficiency is maximized.

[0097] 8. Placing the second differentiator from the disturbance source to the adder output position can maintain the original convergence direction of the circuit, reduce the requirement for disturbance power, and thus improve the stability of the network.

[0098] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A neural network optimization circuit, characterized in that: Applied to target neural networks, including: A first conversion unit, configured to convert the target training data into a target DC level signal corresponding to the target training data; A second conversion unit, used to convert the label data corresponding to the target training data into a labeling result level signal; A first multiplier, configured to multiply a target DC level signal corresponding to the target training data by the adder output signal, and output a prediction result signal corresponding to the target training data; An error calculation unit, used to calculate the MSE error signal between the prediction result signal and the labeling result signal of the target training data; A first differentiator, configured to receive the MSE error signal and output a first differentiator output signal; a second differentiator, configured to receive the adder output signal and output a second differentiator output signal; a second multiplier, configured to receive the first differentiator output signal, the second differentiator output signal, and output a second multiplier output signal; an integrator, configured to receive the second multiplier output signal and output an integrator output signal; an adder, used for receiving the second adder output signal and the integrator output signal; a third multiplier, configured to receive a target DC level signal corresponding to the target training data, a disturbance signal of each weight, and output a third multiplier output signal; Among them, the weights corresponding to the connection relationship between each layer of the neural network in the target neural network and the same output neuron come from the same disturbance signal.

2. The neural network optimization circuit according to claim 1, characterized in that: The target neural network includes a plurality of different subnetworks, each subnetwork establishes a control chip, each pin on the input side of the control chip corresponds to an input neuron, and each pin on the output side of the control chip corresponds to an output neuron.

3. The neural network optimization circuit according to claim 2, characterized in that: The input neuron and the output neuron are both DC level signals.

4. The neural network optimization circuit according to claim 2, characterized in that: In two adjacent subnetworks, the output neurons of the front subnetwork are connected to the input neurons of the rear subnetwork.

5. The neural network optimization circuit according to claim 2, characterized in that: When the target training data is trained in the target neural network, the MSE error signal is fed back to all sub-networks of the target neural network through wires.

6. The neural network optimization circuit according to claim 1, characterized in that: The target neural network input side is connected to the first conversion unit, and the target neural network output side is connected to the second conversion unit.

7. The neural network optimization circuit according to claim 1, characterized in that: Also includes: The computer is used to send a digital signal corresponding to the target training data to the first conversion unit, send a digital signal corresponding to the label data of the target training data to the second conversion unit, and receive a prediction result signal corresponding to the target training data.

8. The neural network optimization circuit according to claim 7, characterized in that: Also includes: The third conversion unit is used to convert the prediction result signal corresponding to the target training data into a digital signal and transmit it to the computer.

9. The neural network optimization circuit according to claim 1, characterized in that: When training the target training data, the target neural network simultaneously modifies different weight values ​​corresponding to each weight change data.

10. The neural network optimization circuit according to any one of claims 1 to 9, characterized in that: The target neural network includes: a fully connected neural network.

Citation Information

Cited By

  • Neural network optimization circuit

    WO2026037447A1