Method and device for improving calculation precision of neural network on analog arithmetic device

Abstract

The invention discloses a method and a device for improving the calculation precision of a neural network on an analog arithmetic device. The method comprises the following steps: acquiring an original neural network; inserting a noise reduction layer at a preset position corresponding to a neural network layer of the original neural network to form a target neural network; and calculating a neural network layer included in an original neural network in the target neural network based on the analog arithmetic device to obtain a first feature map, and calculating the first feature map in the noise reduction layer based on the digital arithmetic device to obtain a target feature map. According to the method, the reasoning precision of the neural network can be improved, and the calculated amount is low.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a method for improving the calculation accuracy of a neural network on an analog computing device, a computer-readable storage medium, an operator, and a device for improving the calculation accuracy of a neural network on an analog computing device. Background Art

[0002] The Computing In Memory (CIM) crossbar array (abbreviated as CIM) is an efficient analog computing device that can be used to implement large-scale matrix-vector multiplication and addition operations, and is usually widely used in artificial intelligence and neural network calculations. The basic computing unit of its circuit is usually a conductance or charge modulated circuit device, such as a memristor, phase change memory, magnetic memory, floating gate transistor, dynamic random access memory, static random access memory, etc. By gating the row and column switches, the current is accumulated and sampled in the analog domain to achieve efficient matrix multiplication operations.

[0003] like Figure 1 As shown, taking the memristor memory-computation integrated array for neural network reasoning (matrix multiplication) as an example, the weight information is usually stored in the memory-computation integrated device in the form of conductance (G), and the row and column switches are selected by using the word line (WL), and the DAC (Digital-to-Analog Converter) is used to apply different levels of voltage on the bit line (BL) to obtain the current (Itot), and the current is collected at the source line (SL) end and sampled by the ADC (Analog-to-Digital Converter), that is, the results of multiple sets of matrix multiplication and addition operations can be obtained in parallel. Since the analog circuit directly uses physical quantities for calculation, there are non-ideal noises at the device level, circuit level, and system level.

[0004] At present, the relevant technology adopts the strategy of weight replication to perform multiple sets of repeated calculations and take the average value to reduce the random noise in the calculation process, but this will take up additional array storage space. Alternatively, the weights are not copied, and the random noise is reduced by taking the average of multiple calculations, but this will increase the delay and power consumption of the calculation. In addition, there is also a calculation method that reduces the parallelism of array calculations. Each calculation only opens a small number of rows, such as 2 rows, 4 rows, 8 rows, etc., for multiplication and accumulation calculations. This can reduce the fluctuation range of random noise and improve the signal-to-noise ratio, but it will increase the delay of the calculation and reduce the computing power. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present invention is to propose a method for improving the calculation accuracy of a neural network on an analog computing device, obtain an original neural network, insert a denoising layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network, calculate the neural network layer contained in the original neural network in the target neural network based on the analog computing device to obtain a first feature map, and calculate the first feature map in the denoising layer based on the digital computing device to obtain a target feature map, thereby improving the reasoning accuracy of the neural network with low calculation amount.

[0006] A second object of the present invention is to provide a computer-readable storage medium.

[0007] The third objective of the present invention is to provide an operator.

[0008] A fourth object of the present invention is to provide a device for improving the computational accuracy of a neural network on an analog computing device.

[0009] To achieve the above-mentioned purpose, the first aspect of the embodiment of the present invention proposes a method for improving the calculation accuracy of a neural network on an analog computing device, the method comprising: obtaining an original neural network; inserting a noise reduction layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network; calculating the neural network layer contained in the original neural network in the target neural network based on the analog computing device to obtain a first feature map, and calculating the first feature map in the noise reduction layer based on a digital computing device to obtain a target feature map.

[0010] According to the method for improving the calculation accuracy of a neural network on an analog computing device according to an embodiment of the present invention, an original neural network is obtained, a denoising layer is inserted at a preset position corresponding to the neural network layer of the original neural network to form a target neural network, the neural network layer included in the original neural network in the target neural network is calculated based on the analog computing device to obtain a first feature map, and the first feature map in the denoising layer is calculated based on the digital computing device to obtain a target feature map. Thus, the method can improve the reasoning accuracy of the neural network with low calculation amount.

[0011] In addition, the method for improving the calculation accuracy of a neural network on an analog computing device according to the above embodiment of the present invention may also have the following additional technical features:

[0012] According to one embodiment of the present invention, inserting the noise reduction layer at a preset position of the original neural network includes: inserting the noise reduction layer at the neural network layers separated by a preset number of layers.

[0013] According to one embodiment of the present invention, inserting the noise reduction layer at a preset position of the original neural network includes: inserting the noise reduction layer between adjacent neural network layers.

[0014] According to one embodiment of the present invention, the method also includes: calculating each neural network layer of the original neural network based on the digital computing device to obtain a feature map of each neural network layer as a reference feature map of each neural network layer; calculating the neural network layer included in the original neural network based on the analog computing device, and based on the calculated result, using the digital computing device to calculate the noise reduction layer to obtain an actual feature map; training the target neural network based on the reference feature map and the actual feature map.

[0015] According to one embodiment of the present invention, the training of the target neural network based on the target feature map and the actual feature map includes: passing the reference feature map and the actual feature map into a loss function to obtain a loss value; and performing a gradient update on the weights of the denoising layer according to the loss value and the loss function based on a neural network training method based on gradient backpropagation.

[0016] According to one embodiment of the present invention, the method further includes: determining the weight of the denoising layer in the target neural network when the loss value is reduced to a preset loss threshold.

[0017] According to one embodiment of the present invention, the method further includes: determining the weight of the denoising layer in the target neural network when the number of iterations reaches a preset number of iterations.

[0018] To achieve the above-mentioned purpose, a second aspect of the present invention proposes a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-mentioned method for improving the calculation accuracy of a neural network on an analog computing device.

[0019] According to the computer-readable storage medium of an embodiment of the present invention, by implementing the above-mentioned method for improving the calculation accuracy of a neural network on an analog computing device during execution, the reasoning accuracy of the neural network can be improved with low calculation amount.

[0020] To achieve the above-mentioned purpose, an embodiment of the third aspect of the present invention proposes an operator, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned method for improving the calculation accuracy of the neural network on the analog computing device is implemented.

[0021] According to the computing unit of the embodiment of the present invention, by executing the above-mentioned method for improving the calculation accuracy of the neural network on the analog computing device, the reasoning accuracy of the neural network can be improved with low calculation amount.

[0022] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present invention proposes a device for improving the calculation accuracy of a neural network on an analog computing device, the device comprising: an acquisition module for acquiring an original neural network; an insertion module for inserting a noise reduction layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network; a calculation module for calculating the neural network layer contained in the original neural network in the target neural network based on the analog computing device to obtain a first feature map, and calculating the first feature map in the noise reduction layer based on a digital computing device to obtain a target feature map.

[0023] According to the device for improving the calculation accuracy of a neural network on an analog computing device according to an embodiment of the present invention, the acquisition module is used to acquire the original neural network, the insertion module is used to insert a denoising layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network, and the calculation module is used to calculate the neural network layer included in the original neural network in the target neural network based on the analog computing device to obtain a first feature map, and calculate the first feature map in the denoising layer based on the digital computing device to obtain a target feature map. Thus, the device can improve the reasoning accuracy of the neural network with low calculation amount.

[0024] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a method for improving the calculation accuracy of a neural network on an analog computing device according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of neural network reasoning for a storage-computation-in-one array in the prior art;

[0027] Figure 3 A schematic diagram of specific calculations for neural network reasoning performed by a storage-computation integrated array in the prior art;

[0028] Figure 4 A schematic diagram of online training of a denoising layer according to an embodiment of the present invention;

[0029] Figure 5 Schematic diagram of the loss effect of the feature map of the denoising layer after the input data is trained according to an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of a hardware system supporting hybrid training and reasoning according to an embodiment of the present invention;

[0031] Figure 7 A flowchart of a method for improving the calculation accuracy of a neural network on an analog computing device according to a specific example of the present invention;

[0032] Figure 8 is a block diagram of an arithmetic unit according to an embodiment of the present invention;

[0033] Fig. 9 Schematic diagram of a block diagram of a device for improving the computational accuracy of a neural network on an analog computing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0035] like Figure 2 As shown, taking the memristor memory-computation integrated array for neural network reasoning (matrix multiplication) as an example, the weight information is usually stored in the memory-computation integrated device in the form of conductance (G), and the row and column switches are selected by using the word line (WL), and the digital-to-analog converter device DAC is used to apply different levels of voltage on the bit line (BL) to obtain the current (Itot), and the current is collected at the source line (SL) end and sampled by the analog-to-digital converter device (ADC, Analog-to-Digital Converter), that is, the results of multiple sets of matrix multiplication and addition operations can be obtained in parallel. Specifically, as Figure 3 As shown in the figure, the Figure 2 A row in the storage-computation integrated array shown. The weight values ​​of the neural network are quantized to integers and stored in the memristor unit. WL controls all transistor switches in the row circuit. BL is the input voltage of each memristor, and the voltage value (V1-Vn) is adjusted to correspond to different input signals. The input signal is a neuron or feature map quantized to an integer. According to Kirchhoff's current law, the current through each memristor is V×G. All currents flowing through the memristor will converge on SL and be sampled as digital signals by the back-end ADC, completing a multiplication and addition operation in the analog domain. The current formula in the analog domain is as follows: I tot =V1×G1+V2×G2+...+V n ×G n Among them, I tot [A] is the total current, V i(i＝1,2,...,n)[V] is the voltage applied to each memristor, G i(i＝1,2,...,n) [S] is the conductance value of each memristor, and the corresponding dimension is in square brackets. When performing neural network inference, the input data X and weight value W need to be mapped to the input voltage V and the conductance value G respectively. Since analog circuits directly use physical quantities for calculations, there are non-ideal noises at the device level, circuit level, and system level. These noises will cause the calculation accuracy of analog computing devices to decrease. For example, when performing neural network inference, as the number of layers of the network increases, the noise will also accumulate layer by layer, thereby reducing the signal-to-noise ratio of the feature map, and ultimately causing a significant decrease in the inference accuracy of the neural network.

[0036] In addition, in the related art, each storage unit uses a lower information storage density, such as splitting a 4-bit weight according to high and low bits and using two 2T2R units to save it. Or each storage unit only stores 1 bit of information, and multiple storage units are used to represent a multi-bit weight. But this will take up additional array storage space. There is also a strategy of weight replication to perform multiple groups of repeated calculations and take the average value to reduce random noise in the calculation process, but this will take up additional array storage space. Alternatively, the weights are not copied, but the random noise is reduced by taking the average of multiple calculations, but this will increase the delay and power consumption of the calculation. Or a calculation method that reduces the parallelism of array calculations is used, and only a small number of rows, such as 2 rows, 4 rows, 8 rows, etc., are opened each time for multiplication and accumulation calculations, which can reduce the fluctuation range of random noise and improve the signal-to-noise ratio. But it will increase the delay of calculations and reduce computing power.

[0037] To this end, the present application proposes a method for improving the computational accuracy of neural networks on analog computing devices. When an analog computing device is needed to infer a neural network, a noise reduction layer can be inserted into the network, and the noise reduction layer can use any algorithm that can improve the accuracy of the network. The purpose of inserting these noise reduction layers is to suppress the noise in the feature map, improve the signal-to-noise ratio of the feature map, extract and strengthen the effective information in the feature map, so as to avoid the problem that the noise continues to accumulate with the depth of the neural network, and eventually causes the signal-to-noise ratio of the feature map to drop below an acceptable range. In this way, the reasoning accuracy of the neural network can be improved without occupying an additional storage and computing array, and the operating accuracy of the deep neural network on the analog computing device can be greatly improved while slightly increasing the amount of calculation.

[0038] The following describes, with reference to the accompanying drawings, a method for improving the computational accuracy of a neural network on an analog computing device, a computer-readable storage medium, an operator, and a device for improving the computational accuracy of a neural network on an analog computing device, which are proposed in embodiments of the present invention.

[0039] Figure 1The present invention is a flowchart of a method for improving the calculation accuracy of a neural network on an analog computing device according to an embodiment of the present invention.

[0040] like Figure 1 As shown, the method for improving the calculation accuracy of a neural network on an analog computing device according to an embodiment of the present invention may include the following steps:

[0041] S1, obtain the original neural network.

[0042] S2, inserting a noise reduction layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network, wherein the preset position can be determined according to actual conditions.

[0043] S3, based on the analog computing device, the neural network layer included in the original neural network in the target neural network is calculated to obtain a first feature map, and based on the digital computing device, the first feature map in the noise reduction layer is calculated to obtain a target feature map.

[0044] Specifically, obtaining the original neural network means obtaining or constructing an unmodified original neural network model, which may have been trained for a specific task or may be a predefined network architecture. In order to improve the robustness of the neural network to noise, especially when running on an analog computing device, a noise reduction layer may be inserted at a preset position corresponding to the neural network layer of the original neural network to form a target neural network, that is, the target neural network contains the neural network layer and the noise reduction layer of the original neural network. For example, it can be determined after which neural network layer to insert the noise reduction layer based on the analysis of the network architecture and noise characteristics. After inserting the noise reduction layer, the noise in the feature map can be suppressed, the signal-to-noise ratio of the feature map can be improved, and the effective information in the feature map can be extracted and strengthened, thereby avoiding the problem that the noise continues to accumulate with the depth of the neural network, and eventually the signal-to-noise ratio of the feature map is reduced to below the acceptable range.

[0045] After obtaining the target neural network, the neural network layer contained in the original neural network in the target neural network can be calculated based on the analog computing device, such as calculating the input data by simulating the storage and computing integrated device to simulate the noise and computing limitations in the actual hardware operation. This step generates the first feature map, that is, the data calculated by the analog device, which may contain noise and errors introduced by other hardware. After that, the first feature map calculated by the analog computing device is sent to the denoising layer in the digital computing device (such as CPU (Central Processing Unit, central processing unit) or GPU (Graphics Processing Unit, graphics processing unit)). In the digital computing device, the first feature map is calculated using the denoising layer to generate the target feature map, which is the result of the output of the denoising layer, aiming to simulate the output under ideal conditions, that is, to minimize the impact of noise. Among them, the digital operator only calculates the denoising layer, and the analog operator neural network layer calculates, that is, the analog computing device is usually more efficient than the digital computing device when performing operations such as matrix-vector multiplication, and can calculate a large amount of data in parallel. By assigning computationally intensive tasks to the analog operator, the burden of the digital operator can be reduced, thereby improving the overall computing efficiency. Digital operators generally have higher flexibility and programming convenience. By assigning the calculation task of the denoising layer to the digital operator, precise calculations and complex algorithm calculations can be performed more efficiently. By calculating the denoising layer on the digital operator, the noise can be more precisely controlled and reduced, thereby improving the robustness and accuracy of the network.

[0046] In addition, digital operators are easier to update and expand to accommodate new algorithms and models, and by keeping the denoising layer in the digital domain, it is easier to iterate and optimize the denoising layer. This can improve the inference accuracy of the neural network with low computational effort.

[0047] According to one embodiment of the present invention, inserting a noise reduction layer at a preset position of an original neural network includes: inserting a noise reduction layer between adjacent neural network layers.

[0048] Specifically, when inserting a denoising layer at a preset position of the original neural network, the denoising layer can be inserted between adjacent neural network layers. Inserting a denoising layer between adjacent layers helps to continuously suppress noise, so that the noise can be calculated in time before it has a chance to propagate and accumulate in the neural network. And by performing denoising calculations immediately after each layer, it can be ensured that the feature map passed to the next layer has a higher signal-to-noise ratio, thereby improving the feature learning and decision-making capabilities of subsequent neural network layers. On hardware platforms, such as analog storage and computing devices, noise can be a significant problem, and inserting denoising layers between adjacent layers can help the network better adapt to these hardware characteristics. Layer-by-layer denoising can significantly improve the reasoning accuracy of the neural network.

[0049] According to another embodiment of the present invention, inserting a noise reduction layer at a preset position of the original neural network includes: inserting a noise reduction layer at a neural network layer separated by a preset number of layers. The preset number of layers can be determined according to actual conditions.

[0050] Specifically, when inserting a noise reduction layer at a preset position of the original neural network, the noise reduction layer can be inserted at the neural network layer with a preset number of layers. For example, the preset number of layers can be 2 layers, that is, a noise reduction layer is inserted between every two neural network layers to improve the robustness and performance of the network. That is to say, in a neural network, especially a deep network, noise and errors may accumulate between layers, resulting in a decrease in output quality. By inserting a noise reduction layer in the interval layer, these noises can be regularly reduced to maintain the quality of the network output. Moreover, compared to inserting a noise reduction layer between adjacent neural network layers, inserting a noise reduction layer at intervals can reduce the computational burden, because not every layer contains additional noise reduction calculations, so that computing resources can be used more efficiently. In addition, noise reduction layers can be selectively inserted according to the importance of different neural network layers and their sensitivity to noise, providing greater flexibility for network design.

[0051] According to an embodiment of the present invention, the noise reduction layer is designed in a group convolution manner.

[0052] Specifically, the denoising layer can use any algorithm that can improve the accuracy of the network, such as a convolution layer with a convolution kernel size of 1*1 and a step size of 1, or, in one embodiment of the present invention, grouped convolution can be used to further reduce the amount of calculation. That is, in grouped convolution, the input channels are divided into multiple groups, and each group uses an independent convolution kernel for convolution operations, that is, the number of convolution kernels used in each group can be less than the number of input channels, which helps to reduce the complexity and computational burden of the model, and the use of grouped convolution in the denoising layer can better calculate complex image or signal structures, because different groups can capture different local features. Therefore, by using grouped convolution in the denoising layer, the impact of noise can be effectively reduced while maintaining or even improving the performance of the neural network. In addition, not only grouped convolution, any form of neural network layer can be used as the design of the denoising layer.

[0053] According to one embodiment of the present invention, the method for improving the calculation accuracy of a neural network on an analog computing device also includes: calculating each neural network layer of the original neural network based on a digital computing device to obtain a feature map of each neural network layer as a reference feature map of each neural network layer; calculating the neural network layers included in the original neural network based on the analog computing device, and based on the calculated results, using a digital computing device to calculate the noise reduction layer to obtain an actual feature map; and training the target neural network based on the reference feature map and the actual feature map.

[0054] Specifically, hardware-aware training is currently used in neural network training, that is, noise is added to weights or intermediate data streams during training to improve the robustness and resistance to noise of the network. However, this method has limited effect and is difficult to fit for deep neural networks during training; at the same time, the noise added during training is only simulated and approximated based on the statistical characteristics of the simulated calculation, which is difficult to reflect the actual noise characteristics. Alternatively, the network is deployed layer by layer on the analog computing device using online training and trained layer by layer. One or more layers are deployed each time, and then the subsequent weights are retrained so that the weights in the network can perceive the real noise introduced by the previous network on the analog computing device to improve the accuracy of the network running on the hardware. However, this method is very cumbersome to operate and requires corresponding operations layer by layer during training; at the same time, this solution cannot solve the problem of additional noise introduced into the feature map after each layer is calculated. When the number of neural network layers increases, the noise will still accumulate. Alternatively, there is also a method of inserting a noise reduction layer in the network, and using a low-noise or noise-free computing device to calculate the noise reduction layer to improve the signal-to-noise ratio of the feature map and enhance the effective information in the feature map. Hardware-aware training is used during training to introduce noise into the network layer that is expected to be calculated on the simulated computing device to improve the robustness of the network. However, this method still uses noise that is similar to the simulated computing hardware, which is difficult to reflect the actual noise characteristics.

[0055] Therefore, in one embodiment of the present invention, the weights of all layers can be updated simultaneously by performing online training of the neural network layer by layer or across multiple layers, without the need for cumbersome layer-by-layer training, thus greatly improving the training speed. Figure 4 As shown, the denoising layer is set in a way of inserting the neural network layer layer by layer, and each neural network layer of the original neural network is calculated based on the digital computing device to obtain the feature map of each neural network layer, that is, the digital computing device (such as CPU or GPU) usually provides high-precision and low-noise computing power, and the use of digital computing devices to calculate each layer of the original neural network means that the weights and input data of each layer are calculated in a digital environment and are not affected by analog hardware noise. The output of each layer after calculation (i.e., the reference feature map) is considered to be the output of the layer under ideal conditions, without interference from hardware noise. These reference feature maps are used as targets or references in the training process to evaluate and guide the training of the neural network.

[0056] After obtaining the reference feature map, the neural network layers included in the original neural network can be calculated based on the analog computing device, and based on the calculated results, the denoising layer can be calculated using the digital computing device to obtain the actual feature map. That is, analog computing devices (such as storage and computing integrated arrays) are usually used to perform fast matrix operations, but may introduce noise and other non-ideal characteristics. The input data and weights of each layer are calculated by the analog computing device to generate a preliminary feature map. The input of the denoising layer is the feature map calculated by the analog computing device, which is used to perform denoising operations through the digital computing device (digital computing device), such as applying a specific algorithm to reduce the impact of noise. The output after the denoising calculation is the actual feature map, which reflects the output of the neural network when it runs on analog hardware.

[0057] After determining the reference feature map and the actual feature map, the target neural network can be trained based on the reference feature map and the actual feature map. For example, by constructing a specific loss function, it is ensured that the network is as close to the reference feature map as possible when generating the actual feature map, or by using the transfer learning method, part or all of the weights of the pre-trained model can be transferred to the target neural network, and then the actual feature map is adjusted to approximate the reference feature map.

[0058] In addition, in one embodiment of the present invention, Figure 5 As shown, a block diagram of a hardware architecture that supports on-chip training and reasoning is shown. The hardware architecture includes a controller, a cache, a digital computing core, and an analog computing core. Among them, the controller is responsible for assigning tasks to the appropriate computing core (digital computing core or analog computing core) to ensure that the task is executed on the most suitable processing unit. The cache is a layer of fast storage located between the processor and the main memory, which is used to reduce the time required for the processor to access the main memory, thereby improving the overall computing efficiency. The digital computing core is responsible for the learning and reasoning of the denoising layer neural network, and can also be used to solve the loss function and perform computing tasks that require high precision and high reliability. The analog computing core is used for reasoning in other layers of the neural network. Perform fast matrix operations and data calculations.

[0059] According to one embodiment of the present invention, a target neural network is trained based on a target feature map and an actual feature map, including: passing a reference feature map and an actual feature map into a loss function to obtain a loss value; and performing a gradient update on the weights of a denoising layer according to the loss value and the loss function based on a neural network training method based on gradient backpropagation.

[0060] Furthermore, according to an embodiment of the present invention, the method for improving the computational accuracy of a neural network on an analog computing device further includes: determining the weight of a noise reduction layer in a target neural network when the loss value is reduced to a preset loss threshold, wherein the preset loss threshold may be determined according to actual conditions.

[0061] Further, according to another embodiment of the present invention, the method for improving the computational accuracy of a neural network on an analog computing device further includes: determining the weight of a noise reduction layer in a target neural network when the number of iterations reaches a preset number of iterations. The preset number of iterations may be determined according to actual conditions.

[0062] Specifically, if Figure 4 As shown, a noise-free computing device (such as a CPU or GPU) is used to infer the original neural network to obtain reference feature maps for each layer. These reference feature maps represent the output that the network should produce under ideal conditions, that is, when there is no noise interference. Next, a simulated computing device is used to calculate the neural network containing the denoising layer to obtain actual feature maps. These actual feature maps are the outputs obtained on the simulated storage-computing integrated computing device after taking into account the hardware noise. The reference feature map and the actual feature map are passed into the loss function to calculate the difference between the two. This difference is the loss value, which quantifies the deviation between the model output and the ideal output. Among them, the loss function has multiple definitions, such as the loss function can be mean square error, cross entropy loss, L1 norm loss, L2 norm loss (Euclidean distance). In the training of the denoising layer, the L2 norm loss can be used to calculate the difference between the reference feature map and the actual feature map. The L2 norm loss can quantify the Euclidean distance between the two and intuitively reflects the similarity between the feature maps. That is, during the training process, by minimizing the L2 loss, the model learns how to adjust the weights of the denoising layer so that the actual feature map is as close to the reference feature map as possible, thereby improving the prediction accuracy of the network.

[0063] After obtaining the loss value, the weights of the denoising layer can be updated based on the loss value and loss function based on the gradient back propagation neural network training method. That is, gradient back propagation is the core algorithm in neural network training, which is used to update the network weights based on the loss value and loss function to minimize the loss. Specifically, starting from the output layer, the gradient of the loss function with respect to the neural network weights is calculated. This gradient represents the sensitivity of the loss function to each weight under the current weight. These gradients can be back-propagated back to each layer of the network using the chain rule, which can calculate the derivatives of complex functions (such as neural network layers). Once the gradient of each layer's weight is calculated, the gradient descent algorithm can be used to update the weights. The update rule is: W = W-η*w, where W is the weight, η is the learning rate, and w is the gradient of the weight. The learning rate is a hyperparameter that controls the step size of the weight update. Repeat the above steps to update the weights of the denoising layer based on the loss value and loss function.

[0064] The size of the loss value is judged. When the loss value is reduced to the preset loss threshold, the weight of the noise reduction layer in the target neural network can be determined. In other words, the preset loss threshold is a performance indicator that can be set according to the needs of the actual application and the characteristics of the hardware. When the loss value is lower than this threshold, it can be considered that the neural network is accurate enough and the training can be stopped. That is, when the loss value is reduced to the preset loss threshold, the weight of the noise reduction layer is determined. These weights are optimized so that the neural network can have high computational accuracy and robustness when running on an analog computing device.

[0065] In addition, in another embodiment of the present invention, if the training time is too long, the neural network may learn the noise and outliers in the training data, resulting in poor performance of the model on new data, that is, overfitting. Therefore, by limiting the number of iterations, it is possible to avoid overfitting the model to the training data. In other words, the weight of the noise reduction layer in the target neural network can be determined when the number of iterations reaches a preset number of iterations. In addition, the neural network may have shown good performance after reaching a certain number of iterations, and continued training may not bring significant performance improvements. By setting a preset number of iterations, training can be stopped when the model performance is no longer significantly improved, saving time and resources. Therefore, at this number of iterations, the model has reached a balance point, that is, under the current training conditions, the performance of the model has been optimized as much as possible. As Figure 6 As shown in the figure, a test example obtained using the on-chip training method is shown. It can be seen that the feature map loss of the trained denoising layer enhanced network is much smaller than that of the untrained original neural network, which proves the effectiveness of this method.

[0066] Combine the following Figure 7 To describe the method of the present invention.

[0067] As a specific example, the method of improving the calculation accuracy of a neural network on an analog computing device of the present invention may include the following steps:

[0068] S101, obtaining the original neural network.

[0069] S102, calculating each neural network layer of the original neural network based on a digital computing device to obtain a feature map of each neural network layer as a reference feature map of each neural network layer.

[0070] S103, calculating the neural network layer included in the original neural network based on the analog computing device, and based on the calculated result, calculating the noise reduction layer using the digital computing device to obtain the actual feature map.

[0071] S104, passing the reference feature map and the actual feature map into the loss function to obtain the loss value.

[0072] S105, based on the gradient back propagation neural network training method, the weight of the denoising layer is gradient updated according to the loss value and the loss function.

[0073] S106, determine whether the loss value is reduced to a preset loss threshold, or whether the number of iterations reaches a preset number of iterations. If yes, execute step S107; if not, execute step S105.

[0074] S107, determining the weight of the denoising layer in the target neural network.

[0075] S108, inserting a noise reduction layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network.

[0076] S109, calculating the neural network layer included in the original neural network in the target neural network based on the analog computing device to obtain a first feature map, and calculating the first feature map in the denoising layer based on the digital computing device to obtain a target feature map.

[0077] In summary, according to the method for improving the calculation accuracy of a neural network on an analog computing device according to an embodiment of the present invention, an original neural network is obtained; a denoising layer is inserted at a preset position corresponding to the neural network layer of the original neural network to form a target neural network; the neural network layer included in the original neural network in the target neural network is calculated based on the analog computing device to obtain a first feature map, and the first feature map in the denoising layer is calculated based on the digital computing device to obtain a target feature map. Therefore, the method can improve the reasoning accuracy of the neural network with low calculation amount.

[0078] Corresponding to the above embodiment, the present invention also proposes a computer-readable storage medium.

[0079] The computer-readable storage medium of an embodiment of the present invention stores a program thereon, and when the program is executed by a processor, the method for improving the calculation accuracy of a neural network on an analog computing device is implemented as described above.

[0080] According to the computer-readable storage medium of an embodiment of the present invention, by executing the above-mentioned method for improving the calculation accuracy of a neural network on an analog computing device, the reasoning accuracy of the neural network can be improved with low calculation amount.

[0081] Corresponding to the above embodiment, the present invention also proposes an operator.

[0082] like Figure 8As shown, the operator 200 of the embodiment of the present invention may include: a memory 210, a processor 220, and a program stored in the memory 210 and executable on the processor 220. When the processor 220 executes the program, the above-mentioned method of improving the calculation accuracy of the neural network on the analog computing device is implemented.

[0083] According to the computing unit of the embodiment of the present invention, by executing the above-mentioned method for improving the calculation accuracy of the neural network on the analog computing device, the reasoning accuracy of the neural network can be improved with low calculation amount.

[0084] Corresponding to the above-mentioned embodiment, the present invention also proposes a device for improving the calculation accuracy of a neural network on an analog computing device.

[0085] like Fig. 9 As shown, the method and device 100 of the embodiment of the present invention for improving the calculation accuracy of a neural network on an analog computing device includes: an acquisition module 110, an insertion module 120 and a calculation module 130.

[0086] Among them, the acquisition module 100 is used to obtain the original neural network. The insertion module 110 is used to insert the noise reduction layer at the preset position corresponding to the neural network layer of the original neural network to form a target neural network. The calculation module 130 is used to calculate the neural network layer included in the original neural network in the target neural network based on the analog operation device to obtain the first feature map, and calculate the first feature map in the noise reduction layer based on the digital operation device to obtain the target feature map.

[0087] According to one embodiment of the present invention, the inserting module 120 inserts a noise reduction layer at a preset position of the original neural network, and is specifically used to: insert a noise reduction layer at a neural network layer separated by a preset number of layers.

[0088] According to one embodiment of the present invention, the inserting module 120 inserts a noise reduction layer at a preset position of the original neural network, and is specifically used to: insert a noise reduction layer between adjacent neural network layers.

[0089] According to one embodiment of the present invention, the computing module 130 is also used to: calculate each neural network layer of the original neural network based on a digital computing device to obtain a feature map of each neural network layer as a reference feature map of each neural network layer; calculate the neural network layers included in the original neural network based on an analog computing device, and based on the calculated results, use a digital computing device to calculate the noise reduction layer to obtain an actual feature map; train the target neural network based on the reference feature map and the actual feature map.

[0090] According to one embodiment of the present invention, the computing module 130 trains the target neural network based on the target feature map and the actual feature map, and is specifically used to: pass the reference feature map and the actual feature map into the loss function to obtain the loss value; based on the neural network training method of gradient backpropagation, the weights of the denoising layer are gradient updated according to the loss value and the loss function.

[0091] According to one embodiment of the present invention, the calculation module 130 is further used to determine the weight of the denoising layer in the target neural network when the loss value is reduced to a preset loss threshold.

[0092] According to one embodiment of the present invention, the calculation module 130 is further used to: determine the weight of the denoising layer in the target neural network when the number of iterations reaches a preset number of iterations.

[0093] It should be noted that for details not disclosed in the device for improving the computational accuracy of a neural network on an analog computing device in an embodiment of the present invention, please refer to the details disclosed in the method for improving the computational accuracy of a neural network on an analog computing device in an embodiment of the present invention, and the details will not be repeated here.

[0094] According to the device for improving the calculation accuracy of a neural network on an analog computing device according to an embodiment of the present invention, the acquisition module is used to acquire the original neural network, the insertion module is used to insert a denoising layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network, and the calculation module is used to calculate the neural network layer included in the original neural network in the target neural network based on the analog computing device to obtain a first feature map, and calculate the first feature map in the denoising layer based on the digital computing device to obtain a target feature map. Thus, the device can improve the reasoning accuracy of the neural network with low calculation amount.

[0095] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0096] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0097] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0098] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0099] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0100] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for improving the calculation accuracy of a neural network on an analog computing device, characterized in that: The method comprises: Get the original neural network; Inserting a noise reduction layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network; Based on the analog computing device, the neural network layer included in the original neural network in the target neural network is calculated to obtain a first feature map, and based on the digital computing device, the first feature map in the denoising layer is calculated to obtain a target feature map.

2. The method for improving the calculation accuracy of a neural network on an analog computing device according to claim 1, characterized in that: The inserting of a noise reduction layer at a preset position of the original neural network comprises: The denoising layer is inserted at the neural network layer separated by a preset number of layers.

3. The method for improving the calculation accuracy of a neural network on an analog computing device according to claim 1, characterized in that: The inserting of a noise reduction layer at a preset position of the original neural network comprises: The denoising layer is inserted between adjacent neural network layers.

4. The method for improving the calculation accuracy of a neural network on an analog computing device according to claim 1, characterized in that: The method further comprises: Calculating each neural network layer of the original neural network based on the digital operation device to obtain a feature map of each neural network layer as a reference feature map of each neural network layer; The neural network layer included in the original neural network is calculated based on the analog operation device, and based on the calculated result, the noise reduction layer is calculated using the digital operation device to obtain an actual feature map; The target neural network is trained based on the reference feature map and the actual feature map.

5. The method for improving the calculation accuracy of a neural network on an analog computing device according to claim 4, characterized in that: The training of the target neural network based on the target feature map and the actual feature map includes: Passing the reference feature map and the actual feature map into a loss function to obtain a loss value; Based on the gradient back propagation neural network training method, the weight of the denoising layer is gradient updated according to the loss value and the loss function.

6. The method for improving the calculation accuracy of a neural network on an analog computing device according to claim 5, characterized in that: The method further comprises: When the loss value is reduced to a preset loss threshold, the weight of the denoising layer in the target neural network is determined.

7. The method for improving the calculation accuracy of a neural network on an analog computing device according to claim 5, characterized in that: The method further comprises: When the number of iterations reaches a preset number of iterations, the weight of the denoising layer in the target neural network is determined.

8. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for improving the calculation accuracy of a neural network on an analog computing device according to any one of claims 1 to 7 is implemented.

9. A computing unit, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for improving the calculation accuracy of a neural network on an analog computing device according to any one of claims 1 to 7 is implemented.

10. A device for improving the calculation accuracy of a neural network on an analog computing device, characterized in that: The device comprises: An acquisition module is used to obtain the original neural network; An insertion module, configured to insert a noise reduction layer at a preset position corresponding to the neural network layer of the original neural network to form a target neural network; A calculation module is used to calculate the neural network layer included in the original neural network in the target neural network based on the analog operation device to obtain a first feature map, and to calculate the first feature map in the denoising layer based on the digital operation device to obtain a target feature map.

Images