Double-regularization network training method
By adopting the dual regularization method in the neural network, the weights are divided into two groups, and the L1 and L2 regularizations are used to perform rapid sparseness and smooth convergence respectively, which solves the problem of weight bit width limitation in the in-memory computing circuit and improves the model generalization ability and anti-overfit ability.
Patent Information
- Application Number
- CN202411179813.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-17
AI Technical Summary
On resource-constrained in-memory computing circuits, the weight bit width limitation and storage bottleneck of neural network models lead to high complexity of the model, making it difficult to achieve weight sparseness and smooth convergence without affecting performance, and a single regularization method is difficult to simultaneously Meet these needs.
The dual regularization network training method is adopted to divide the network weights into two groups. The first weight group quickly converges to zero through L1 regularization, and the second weight group smoothly converges through L2 regularization, combining forward propagation and backpropagation. Synergistically, controls the convergence speed and size of weights.
It effectively reduces the complexity of the model, improves the generalization ability and anti-overfit ability of the model, and is suitable for deploying efficient neural network models on memory circuits that limit the width of the weight to achieve fine control and stability of weights.
Smart Images
Figure CN119150943A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of network regularization, and in particular to a double regularization network training method. Background Art
[0002] In recent years, with the rapid development of artificial intelligence and machine learning technologies, deep neural networks (DNNs) have achieved remarkable results in many fields such as image recognition, natural language processing, and speech recognition. However, the efficient training and deployment of neural networks still face many challenges, one of which is how to reduce the complexity and storage requirements of the model without affecting the performance of the model. In particular, when deploying neural networks on resource-constrained in-memory computing circuits, the model's weight bit width limitation and storage bottleneck become a key issue.
[0003] Traditional neural network training methods usually use fixed-bit-width weight representations, which are susceptible to physical limitations in hardware implementation, resulting in insufficient weight accuracy or excessive hardware resource usage. At the same time, too many model parameters may also lead to model overfitting, causing a model that performs well on training data to perform poorly on test data. To overcome these problems, researchers have proposed various regularization techniques, such as L1 regularization and L2 regularization, to control the complexity of the model and improve its generalization ability.
[0004] L1 regularization imposes a penalty on the absolute value of the model weights, causing some weights to converge to zero, thereby achieving sparsification and feature selection; while L2 regularization imposes a penalty on the square of the weights, keeping the weights small but not zero, in order to smooth the model and prevent overfitting. However, in practical applications, especially when dealing with complex neural network models, the effect of a single regularization method is often limited, and it is difficult to simultaneously meet the requirements of weight sparsification and smooth convergence. Summary of the invention
[0005] The present invention provides a dual-regularization network training method, which is suitable for deploying an efficient neural network model on a storage and computing circuit with limited weight bit width, while improving the generalization ability and anti-overfitting ability of the model.
[0006] The embodiment of the present invention provides a dual regularization network training method, comprising the following steps:
[0007] The network weights are grouped by forward propagation and back propagation to obtain a first weight group and a second weight group, wherein the first weight group includes weights outside the convolution kernel weight width limit range, and the second weight group includes weights that meet the convolution kernel weight width limit range standard;
[0008] The weights in the first weight group are subjected to L1 regularization of the absolute value of the weights, the weights in the first weight group that meet the convolution kernel weight width limit range standard after L1 regularization are transferred to the second weight group, and the second weight group is subjected to L2 regularization based on the square of the weights, so that both the first weight group and the second weight group converge.
[0009] Optionally, in one embodiment of the present invention, the first weight group includes weights that are outside the range of the convolution kernel kernel[4n] and are not between 0 and 3.
[0010] Optionally, in one embodiment of the present invention, the forward propagation includes a weight input W, a position classification Possplit, a threshold classification Thresh split, the first weight group, the second weight group, a convolution layer, a double regularization module and a loss function, wherein the condition of the position classification Pos split is whether the weight is at the position of the convolution kernel kernel[4n], the condition of the threshold classification Thresh split is whether the weight value is in the range of 0 to 3, the first weight group represents the weight outside the weight bit width limit range, that is, the value is outside the range of 0 to 3, and the second weight group represents the weight that has met the standard, that is, the value is within the range of 0 to 3.
[0011] Optionally, in one embodiment of the present invention, the specific processing process of forward propagation is:
[0012] 1) The input weight W is initially processed by the convolutional layer;
[0013] 2) dividing the weights into the first weight group and the second weight group by using position classification Pos split and threshold classification Thresh split;
[0014] 3) The first weight group is processed by L1 regularization to make it converge quickly to 0;
[0015] 4) The second weight group is smoothly converged by L2 regularization;
[0016] 5) The first weight group and the second weight group that have been regularized are then output through a convolutional layer, the output values are compared with the target values, and a loss function is calculated.
[0017] Optionally, in one embodiment of the present invention, the back propagation includes a convolution layer, a double regularization module, the first weight group, the second weight group, a weight merging module and a weight output; the specific processing process of the back propagation is:
[0018] 1) The weights in the first weight group that have met the criteria in the forward propagation are moved to the second weight group through a merging operation;
[0019] 2) performing L1 regularization on the first weight group. L1 regularization penalizes the absolute value of the weight so that the weight value tends to zero, thereby promoting model sparsification;
[0020] 3) continuing to perform L2 regularization on the second weight group to limit the square value of the weight;
[0021] 4) The first weight group and the second weight group are calculated through a convolutional layer to generate gradients and update the parameters of the model;
[0022] 5) Updating the weight values of the first weight group and the second weight group to minimize the loss function.
[0023] The dual-regularization network training method of the embodiment of the present invention is intended to solve the problem of weight bit width limitation in storage and computing circuits. This method combines the advantages of L1 and L2 regularization, divides the network weights into two groups, and performs different regularization treatments respectively to achieve fine control of the weights. Specifically, the Wa weights outside the restricted range converge quickly to zero through L1 regularization, reducing unnecessary weights and reducing model complexity; while the Wb weights that meet the standards converge smoothly through L2 regularization to maintain the stability and accuracy of the model. This method effectively controls the convergence speed and size of weights through the synergy of forward propagation and back propagation, and is suitable for deploying efficient neural network models on storage and computing circuits with limited weight bit width, while improving the generalization ability and anti-overfitting ability of the model.
[0024] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0026] Figure 1 The present invention is a flowchart of a dual regularization network training method provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION
[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0028] Figure 1 The present invention is a flowchart of a dual regularization network training method provided in accordance with an embodiment of the present invention.
[0029] like Figure 1 As shown, the double regularization network training method includes the following steps:
[0030] The network weights are grouped by forward propagation and back propagation to obtain a first weight group and a second weight group, wherein the first weight group includes weights outside the convolution kernel weight width limit range, and the second weight group includes weights that meet the convolution kernel weight width limit range standard;
[0031] The weights in the first weight group are subjected to L1 regularization of the absolute value of the weights, and the weights in the first weight group that meet the convolution kernel weight width limit range standard after L1 regularization are transferred to the second weight group. The second weight group is subjected to L2 regularization based on the square of the weights, so that both the first weight group and the second weight group converge.
[0032] In an embodiment of the present invention, weight grouping is completed by forward propagation and back propagation, and the L1 and L2 double regularization methods are adopted to achieve the control of weight convergence speed and size. The training method framework includes a convolutional layer (Conv), weight grouping, a double regularization module, and a loss function. The method combines L1 normalization based on the absolute value of the weight and L2 normalization based on the square of the weight. L1 normalization causes the weight to converge quickly to 0, while L2 normalization is adjusted smoothly at a slower speed.
[0033] In this double regularization training method, all weights are divided into two groups: the first weight group Wa and the second weight group Wb. The first weight group Wa contains weights that are outside the range of the convolution kernel [4n] and are not between 0 and 3, and these weights need to be adjusted. Correspondingly, the second weight group Wb contains those weights that have met the standard conditions. During the training process, the weights in the first weight group Wa are L1 normalized and converge quickly to 0. When the weights meet the standard conditions and enter the range of 0 to 3, these weights will be transferred to the second weight group Wb and L2 normalized. After multiple training cycles (epochs), most weights (more than 99%) will be adjusted. For most convolutional neural networks (CNNs), this process is usually completed in less than 20 training cycles (epochs), and the loss of accuracy is usually less than 1%.
[0034] In an embodiment of the present invention, forward propagation mainly includes weight input (W), position classification (Pos split), threshold classification (Thresh split), Wa weight group, Wb weight group, convolution layer, double regularization module and loss function. Among them, the position classification Pos split is performed based on whether the weight is located at the position of the convolution kernel kernel[4n], and the threshold classification Thresh split is based on whether the weight value is in the range of 0 to 3. The first weight group Wa weight group includes those weights that exceed the bit width limit (the value is not in the range of 0 to 3), and the second weight group Wb weight group includes weights that have met the standard (the value is in the range of 0 to 3).
[0035] In an embodiment of the present invention, forward propagation refers to the data processing process from the input layer to the output layer, and the prediction results are calculated layer by layer and generated. Each layer is calculated according to the output and parameters (such as weights and biases) of the previous layer, extracts features and passes them to the next layer. Finally, the output layer generates a predicted value, and the error between the predicted value and the true label is calculated by the loss function. The forward propagation of the training method is implemented through the following 5 steps: 1) The input weight (W) is first processed by the convolution layer; 2) The position classification Pos split and the threshold classification Thresh split divide the weights into two groups: the first weight group Wa and the second weight group Wb; 3) The first weight group Wa weight group is processed by L1 regularization to make it converge quickly to 0; 4) The second weight group Wb weight group is smoothly converged by L2 regularization; 5) The regularized weights are then output by the convolution layer and compared with the target value (GT), and the loss is finally calculated.
[0036] In an embodiment of the present invention, the main components of back propagation include convolutional layers, dual regularization modules, a first weight group Wa, a second weight group Wb, a weight merging module (Merge), and weight outputs. Back propagation starts from the output layer, back propagates errors, and calculates the gradients of the parameters of each layer by the chain rule. This process includes calculating the gradient of the loss function, back propagating the gradient to each layer, and using these gradients to update the parameters to reduce the error. By continuously iterating forward propagation and back propagation, the model parameters gradually converge and the performance is improved.
[0037] In an embodiment of the present invention, the specific processing flow of back propagation is as follows: 1) the weights in the first weight group Wa that have met the standard in the forward propagation process are transferred to the second weight group Wb through a merging operation; 2) the weights of the first weight group Wa are L1 regularized, and the absolute value of the weight is penalized to make it tend to zero, thereby achieving model sparsity, thereby simplifying the model structure and improving efficiency; 3) the weights of the second weight group Wb are further L2 regularized to limit the square value of the weight to prevent the weight from being too large, thereby ensuring the stability and generalization ability of the model; 4) the weights are calculated through the convolution layer and the gradient is generated, which is used to update the model parameters, reduce errors, and optimize the model; 5) finally, by updating the weights of the first weight group Wa and the second weight group Wb, the loss function is further minimized and the model performance is improved.
[0038] According to the dual regularization network training method proposed in the embodiment of the present invention, L1 normalization based on the absolute value of the weight and L2 normalization based on the square of the weight are adopted. L1 normalization causes the weight to converge quickly to 0, while L2 normalization achieves slower convergence. In this method, all weights are divided into a first weight group and a second weight group, the first weight group contains weights outside the range of the convolution kernel weight bit width limit, and the second weight group contains weights that have met the standard. During the dual regularization network training process, the weights in the first weight group are L1 normalized and quickly converge to meet the standard in the range of 0 to 3, and are transferred to the second weight group and L2 normalized. The present invention can eliminate more than 99.994% of error points and produce results close to full precision, solving the impact of weight bit width limitations in in-memory computing circuits, and has significant application value.
[0039] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0040] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
Claims
1. A double regularization network training method, It is characterized in that The following steps are involved: The network weights are grouped by forward propagation and back propagation to obtain a first weight group and a second weight group, wherein the first weight group includes weights outside the convolution kernel weight width limit range, and the second weight group includes weights that meet the convolution kernel weight width limit range standard; The weights in the first weight group are subjected to L1 regularization of the absolute value of the weights, the weights in the first weight group that meet the convolution kernel weight width limit range standard after L1 regularization are transferred to the second weight group, and the second weight group is subjected to L2 regularization based on the square of the weights, so that both the first weight group and the second weight group converge.
2. The method according to claim 1, It is characterized in that The first weight group includes weights that are outside the range of convolution kernel kernel[4n] and are not between 0 and 3.
3. The method according to claim 2, It is characterized in that The forward propagation includes weight input W, position classification Pos split, threshold classification Thresh split, the first weight group, the second weight group, convolution layer, double regularization module and loss function, wherein the condition of position classification Pos split is whether the weight is at the position of convolution kernel kernel[4n], the condition of threshold classification Thresh split is whether the weight value is in the range of 0 to 3, the first weight group represents the weight outside the weight width limit range, that is, the value is outside the range of 0 to 3, and the second weight group represents the weight that meets the standard, that is, the value is within the range of 0 to 3.
4. The method according to claim 3, It is characterized in that The specific process of forward propagation is as follows: 1) The input weight W is initially processed by the convolutional layer; 2) dividing the weights into the first weight group and the second weight group by using position classification Pos split and threshold classification Thresh split; 3) The first weight group is processed by L1 regularization to make it converge quickly to 0; 4) The second weight group is smoothly converged by L2 regularization; 5) The first weight group and the second weight group that have been regularized are then output through a convolutional layer, the output values are compared with the target values, and a loss function is calculated.
5. The method according to claim 4, It is characterized in that The back propagation includes a convolution layer, a double regularization module, the first weight group, the second weight group, a weight merging module and a weight output; the specific processing process of the back propagation is: 1) The weights in the first weight group that have met the criteria in the forward propagation are moved to the second weight group through a merging operation; 2) performing L1 regularization on the first weight group. L1 regularization penalizes the absolute value of the weight so that the weight value tends to zero, thereby promoting model sparsification; 3) continuing to perform L2 regularization on the second weight group to limit the square value of the weight; 4) The first weight group and the second weight group are calculated through a convolutional layer to generate gradients and update the parameters of the model; 5) Updating the weight values of the first weight group and the second weight group to minimize the loss function.