A small sample online training method, device and storage medium
By introducing inter-batch normalization and random weight averaging modules into deep neural networks, the computational complexity and insufficient sample size of deep neural networks online training on mobile devices are solved, and efficient and accurate model training and Generalization.
Patent Information
- Application Number
- CN202210189617.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-02-28
AI Technical Summary
In the prior art, optimization algorithms are limited by storage, power consumption and computing power, making it difficult to implement online training of deep neural networks on mobile devices and embedded platforms, and a small number of samples lead to problems of model overfitting and catastrophic forgetting.
Using the small sample online training method, convolution calculation and parameter iteration are performed through interval batch normalization module, RELU activation function, block floating point quantization module, maximum pooling layer, full connection layer, Softmax classifier and random weight averaging module. , reduce the amount of computing and data storage, and improve the generalization of the model.
It realizes efficient online training on mobile devices and embedded platforms, reduces computational complexity and memory consumption, avoids model overfitting and catastrophic forgetting, and improves training accuracy and accuracy.
Smart Images

Figure CN114662654B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a small sample online training method, device and storage medium, belonging to the technical field of calculation, estimation or counting. Background Art
[0002] In recent years, deep neural networks have achieved unprecedented success in speech recognition, image recognition, and medical applications. Convolutional neural networks are widely used in the field of visual recognition, such as handwritten digit recognition, character recognition, face detection, cross-media search, face recognition, facial expression analysis, and car detection. In view of the parameter optimization process of neural network training, different learning algorithms have been proposed to train neural networks. The most commonly used algorithm is the gradient-based algorithm, namely the gradient descent algorithm. However, implementing online training of deep neural networks on hardware has always been a difficult problem for researchers.
[0003] First, the current mainstream optimization algorithms have complex calculations and a large number of training parameters. They are limited by storage, power consumption and computing power, and are difficult to be used on mobile devices and embedded platforms. Secondly, in actual application scenarios, it is often difficult to collect more samples, and too little data will have a negative impact on model training. For example, a small amount of training data is difficult to provide enough information for the model to learn, resulting in the model failing to converge; the model learns completely irrelevant information about the training data, causing the model to overfit the training data; and it is easy to fall into local optimal values. Thirdly, in the process of continuous online training of new samples, catastrophic forgetting of old samples will occur. Summary of the invention
[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a small sample online training method, device and storage medium to solve the technical problems that the optimization algorithm in the prior art is limited by storage, power consumption and computing power, is difficult to be installed on mobile devices and embedded platforms, and a small number of samples can easily lead to overfitting. At the same time, in the process of online training of new samples, catastrophic forgetting of old samples will be caused.
[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:
[0006] In a first aspect, the present invention provides a small sample online training method, the method comprising the following steps:
[0007] The neural network is trained using the data set to obtain weight parameters, which are used as the original training parameters of the pre-trained model;
[0008] Collect and label the specified preset amount of samples;
[0009] Input the labeled samples and original training parameters into the pre-training model for training to update the original training parameters;
[0010] The pre-trained model is obtained based on an online training neural network, which includes a convolutional layer, an interval batch normalization module, a RELU activation function, a block floating point quantization module, a maximum pooling layer, a fully connected layer, a Softmax classifier, and a random weight averaging module;
[0011] Inputting the labeled samples and the convolution layer training parameters in the original training parameters into the convolution layer to perform convolution calculation to obtain an output feature map, and inputting the output feature map into the interval batch normalization module;
[0012] The interval batch normalization module performs batch normalization on the output feature map and inputs the batch normalization result into the RELU activation function;
[0013] The RELU activation function activates the batch normalization result, and inputs the activated result into the block floating point quantization module;
[0014] The block floating point quantization module performs low-precision quantization on the activated result, and inputs the low-precision quantized result into the maximum pooling layer;
[0015] The maximum pooling layer performs maximum pooling on the result after low-precision quantization to obtain a low-precision forward feature map, and inputs the low-precision forward feature map into the fully connected layer;
[0016] The fully connected layer calculates the low-precision forward feature map to obtain an output vector, and inputs the output vector into the Softmax classifier;
[0017] The Softmax classifier calculates the output vector to obtain a prediction result;
[0018] Calculate the error between the prediction result and the marked sample to get the corresponding error;
[0019] The corresponding error is input back into the pre-trained model to obtain the relative error between the fully connected layer and the convolutional layer;
[0020] The relative error of the fully connected layer and the relative error of the convolutional layer are convolved with the low-precision forward feature map and the output feature to obtain the weight gradient. The original training parameters are iteratively calculated through the weight gradient to obtain new training parameters, and the new training parameters are input into the random weight averaging module.
[0021] The random weight averaging module performs sliding average calculation on the new training parameters and the original training parameters to obtain the final training parameters;
[0022] Calculate the loss of the final training parameters and make a judgment on the loss.
[0023] Furthermore, the calculation formula for batch normalization of the output feature map by the interval batch normalization module is as follows:
[0024]
[0025] Where: x (d) is the output feature, μ (d) is the average output feature value, is the batch normalization result, is the maximum value of the output feature, is the minimum value of the output feature.
[0026] Furthermore, the weight gradient iteratively calculates the original training parameters, and the calculation formula for the new training parameters is as follows:
[0027]
[0028]
[0029] W t+1 =W t -η t G t
[0030] Where: l(W t ) is brought into the relative error E between the fully connected layer and the convolutional layer C and E F ,λψ(W t ) is the regularization factor, λ is the regularization coefficient, W t The original training parameters include W and w, η t is the learning rate, G t The weight gradient contains and W t+1 is the new training parameter and t is the number of iterations.
[0031] Furthermore, the calculation formula of the random weight average module for sliding average calculation of the new training parameters and the original training parameters is as follows:
[0032]
[0033] Where: g is the preset amount of sample collection, W t is the original training parameter, W t+1 is the training parameter after iteration, are the final training parameters.
[0034] Furthermore, methods for determining losses include:
[0035] Determine whether the loss is less than 0.1;
[0036] If the judgment result is yes, the final training parameters are output and applied;
[0037] If the judgment result is no, the final training parameters and labeled samples are input into the pre-training model for training iteration.
[0038] Furthermore, the prescribed preset number of collected samples is 32.
[0039] In a second aspect, the present invention provides a small sample online training device, including a memory and a processor;
[0040] The memory is used to store programs;
[0041] The processor is used to execute the program to implement the various steps of the small sample online training method described in the first aspect.
[0042] In a third aspect, the present invention provides a readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the various steps of the small sample online training method described in the first aspect are implemented.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] 1. The present invention performs batch normalization on the output feature map through the interval batch normalization module, which greatly reduces the amount of calculation and speeds up the training. The number of iterations required to reach convergence is reduced, and the process of online training calculation is more in line with the computing characteristics of the hardware.
[0045] 2. The present invention iteratively calculates the original training parameters through weight gradients, reduces the resources consumed by data storage, reduces the computational complexity to a certain extent, provides advantages such as scalability, memory saving, portability and energy efficiency for online training, while maintaining a high training accuracy, and facilitates the deployment of network models on mobile devices and embedded platforms.
[0046] 3. The present invention can obtain a more global training parameter result by performing sliding average calculation on the new training parameters and the original training parameters through the random weight averaging module, thereby improving the generalization of the pre-training model, avoiding the pre-training model from falling into the local optimal value, preventing the pre-training model from overfitting the new samples due to too small a sample size, solving the catastrophic forgetting of old samples in online training, and improving the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 A flowchart of a small sample online training method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0048] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0049] Embodiment 1:
[0050] This embodiment proposes a small sample online training method, which is implemented by the following technical solution:
[0051] The VGG16 neural network is trained using the CIFAR100 dataset to obtain weight parameters, which are used as the original training parameters of the pre-trained model.
[0052] Collect samples in actual application scenarios and mark them. It should be noted that the preset amount of collected samples batch_size=32. When the preset amount of collected samples batch_size=32, the samples are input for training;
[0053] Input the labeled samples and original training parameters into the pre-trained model for several training iterations to update the original training parameters;
[0054] The pre-trained model is obtained based on the online training neural network, which includes convolution layer, interval batch normalization module, RELU activation function, block floating point quantization module, maximum pooling layer, fully connected layer, Softmax classifier and random weight average module; the specific implementation plan steps are as follows:
[0055] S1: Input the 32 labeled samples and the convolution layer training parameters in the original training parameters into the convolution layer for convolution calculation to obtain 32 output feature maps, and input the 32 output feature maps into the interval batch normalization module.
[0056] The convolution calculation formula is as follows:
[0057]
[0058] 0≤u <M,0≤y<E,0≤x<F,E=(H-R+U) / U,F=(D-S+U) / U
[0059] Where: I is a sample, the size of sample I is H×D×C, H, D, C are the height, width and number of channels of sample I respectively. O is the output feature map, the size of output feature map O is E×F×M, E, F, M are the height, width and number of channels of output feature map O respectively. U is the step size of convolution, i, j are the horizontal and vertical coordinates of weight respectively, k is the kth input channel, u is the uth output channel, x, y are the horizontal and vertical coordinates of output feature map, W is the convolution layer training parameter, the size of convolution layer training parameter W is R×S, R, S are the height and width of convolution layer training parameter W respectively.
[0060] S2: The interval batch normalization module performs batch normalization on the 32 output feature maps. The specific process is as follows: find the maximum value and the minimum value of the output feature in each output feature map, calculate the average value of the output feature in each output feature map, calculate the output features in each output feature map, obtain the batch normalization results corresponding to the output features, and input the batch normalization results into the RELU activation function.
[0061] The calculation formula for calculating the output features in each output feature map is as follows:
[0062]
[0063] Where: x (d) is the output feature, μ (d) is the average output feature value, is the batch normalization result, is the maximum value of the output feature, is the minimum value of the output feature.
[0064] The output feature maps are batch normalized through the interval batch normalization module, which greatly reduces the amount of calculation and speeds up the training. The number of iterations required to reach convergence is reduced, and the online training calculation process is more in line with the computing characteristics of the hardware.
[0065] S3: The RELU activation function activates the batch normalization result and inputs the activated result p into the block floating-point quantization module. The calculation formula is as follows:
[0066]
[0067] S4: The block floating point quantization module performs low-precision quantization on the activated result x and inputs the low-precision quantized result into the maximum pooling layer.
[0068] The calculation formula for low-precision quantization of the activated result p is as follows:
[0069]
[0070] σ=2 -e+k-2
[0071] Among them: p is the result after activation, σ is the quantization gap, k is the number of bits after low-precision quantization, Q(p,k) is the result after low-precision quantization, round is the random rounding function, and clip is the truncation function. The matrix composed of the low-precision quantized data Q(x,k) is denoted as T, which is the output result of the block floating-point quantization module.
[0072] S5: The maximum pooling layer performs maximum pooling on the output results of the block floating-point quantization module to obtain a low-precision forward feature map, and inputs the low-precision forward feature map into the fully connected layer.
[0073] The calculation formula for the maximum pooling is as follows:
[0074] Y(c,h,w)=maxT(c,hK+k,wK+k),k∈[1,K]
[0075] Among them: T is the output result of the block floating-point quantization module, Y is the low-precision forward feature map, c is the number of channels of the Y matrix, h and w are the horizontal and vertical coordinates of the Y matrix data respectively, and K is the size of the pooling window.
[0076] S6: The fully connected layer calculates the low-precision forward feature map to obtain the output vector, and inputs the output vector into the Softmax classifier.
[0077] The calculation formula is as follows:
[0078]
[0079] Where: Y is the low-precision forward feature map, Z is the output vector, and w is the fully connected layer training parameter. The formula for the mth value Z of the output vector Z is m The calculation process is to convert the nth value Y n Multiply it by the nth value w[m][n] in the mth row of w. The value range of n is 1 to N, where N is the total number of Y. Sum all N products to get Z. m .
[0080] S7: The Softmax classifier calculates the output vector and obtains the prediction result.
[0081] The calculation formula is as follows:
[0082]
[0083] Where: Z m is the mth value of the output vector Z, S is the prediction result, S ii is the ii-th prediction result, ∑e Z To sum all the values of Z after taking the e exponent, is the ii-th value of the corresponding output vector Z.
[0084] S8: Calculate the error between the prediction result and the marked sample to obtain the corresponding error.
[0085] The calculation formula is as follows:
[0086]
[0087] Where: E is the corresponding error, I ii is the ii-th sample, S ii is the ii-th prediction result.
[0088] S9: Reverse the corresponding error into the S1-S6 steps in the pre-trained model to obtain the relative error E between the fully connected layer and the convolutional layer. F and E C Step S6 outputs the relative error E of the fully connected layer F , step S1 outputs the relative error E of the convolutional layer C .
[0089] S10: The relative error E of the fully connected layer F The relative error E of the convolutional layer C Convolution is performed with the low-precision forward feature map Y and the output feature Z to obtain the weight gradient G t .
[0090] The calculation formula is as follows,
[0091]
[0092]
[0093] in, is the weight gradient of the convolutional layer, is the weight gradient of the fully connected layer, om, oc, k x , k y for dimensional coordinates, p and q are E C The horizontal and vertical coordinates of P and Q are E C The horizontal and vertical coordinates of Calculate E F The bth value E F Multiplying [b] by the dth value of Z, Z[d], we can get the value of the bth row and dth column. value
[0094] The original training parameters are iteratively calculated through weight gradient to obtain new training parameters, and the new training parameters are input into the random weight averaging module. The calculation formula is as follows:
[0095]
[0096]
[0097] W t+1 =W t -η t G t
[0098] Where: l(W t ) is brought into the relative error E between the fully connected layer and the convolutional layer C and E F ,λψ(W t ) is the regularization factor, λ is the regularization coefficient, W t The original training parameters include W and w, η t is the learning rate, G t The weight gradient contains and W t+1 is the new training parameter and t is the number of iterations.
[0099] By iteratively calculating the original training parameters through weight gradients, the resources consumed by data storage are reduced, which reduces the computational complexity to a certain extent, providing advantages such as scalability, memory saving, portability and energy efficiency for online training, while maintaining a high training accuracy, and facilitating the deployment of network models on mobile devices and embedded platforms.
[0100] S11: The random weight average module performs sliding average calculation on the new training parameters and the original training parameters to obtain the final training parameters. The calculation process is as follows:
[0101]
[0102] Where: g is the preset amount of sample collection, W t is the original training parameter, W t+1 is the training parameter after iteration, are the final training parameters.
[0103] By performing sliding average calculation on the new training parameters and the original training parameters through the random weight averaging module, a more global training parameter result can be obtained, which can improve the generalization of the pre-trained model, prevent the pre-trained model from falling into the local optimal value, and prevent the pre-trained model from overfitting the new samples due to too small a sample size. It also solves the catastrophic forgetting of old samples in online training and improves the accuracy.
[0104] S12: Calculate the loss of the final training parameters, replace the original training parameters in the network with the final training parameters, re-input the sample to run steps S1 to S8 to calculate and obtain the corresponding error again. The corresponding error obtained again at this time is the loss of the final training parameters.
[0105] The loss of calculating the final training parameters is judged. The specific implementation steps are as follows:
[0106] S121: Determine whether the loss is less than 0.1;
[0107] S122: If the judgment result is yes, output the final training parameters and apply them;
[0108] S123: If the judgment result is no, the final training parameters and the labeled samples are input into the pre-training model for training iteration.
[0109] Embodiment 2:
[0110] An embodiment of the present invention provides a small sample online training device, comprising a memory and a processor;
[0111] The storage medium is used to store instructions;
[0112] The processor is used to operate according to the instructions to perform the steps of the following method:
[0113] The neural network is trained using the data set to obtain weight parameters, which are used as the original training parameters of the pre-trained model;
[0114] Collect and label the specified preset amount of samples;
[0115] The labeled samples and original training parameters are input into the pre-trained model for several training iterations to update the original training parameters.
[0116] Embodiment three:
[0117] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program, which, when executed by a processor, implements the steps of the following method:
[0118] The neural network is trained using the data set to obtain weight parameters, which are used as the original training parameters of the pre-trained model;
[0119] Collect and label the specified preset amount of samples;
[0120] The labeled samples and original training parameters are input into the pre-trained model for several training iterations to update the original training parameters.
[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0124] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A small sample online training method, characterized in that: The steps include: The VGG16 neural network is trained using the CIFAR100 dataset to obtain weight parameters, which are used as the original training parameters of the pre-trained model. Collect and label the specified preset amount of samples; Input the labeled samples and original training parameters into the pre-training model for training to update the original training parameters; The pre-trained model is obtained based on an online training neural network, which includes a convolutional layer, an interval batch normalization module, a RELU activation function, a block floating point quantization module, a maximum pooling layer, a fully connected layer, a Softmax classifier, and a random weight averaging module; Inputting the labeled samples and the convolution layer training parameters in the original training parameters into the convolution layer to perform convolution calculation to obtain an output feature map, and inputting the output feature map into the interval batch normalization module; The interval batch normalization module performs batch normalization on the output feature map and inputs the batch normalization result into the RELU activation function; The RELU activation function activates the batch normalization result, and inputs the activated result into the block floating point quantization module; The block floating point quantization module performs low-precision quantization on the activated result, and inputs the low-precision quantized result into the maximum pooling layer; wherein the calculation formula for performing low-precision quantization on the activated result is as follows: ; ; Among them: p is the result after activation, is the quantization gap, k is the number of bits after low-precision quantization, is the result of low-precision quantization, round is the random rounding function, and clip is the truncation function; the data after low-precision quantization The composed matrix is recorded as T, which is the output result of the block floating point quantization module; The maximum pooling layer performs maximum pooling on the result after low-precision quantization to obtain a low-precision forward feature map, and inputs the low-precision forward feature map into the fully connected layer; The fully connected layer calculates the low-precision forward feature map to obtain an output vector, and inputs the output vector into the Softmax classifier; The Softmax classifier calculates the output vector to obtain a prediction result; Calculate the error between the prediction result and the marked sample to get the corresponding error; The corresponding error is input back into the pre-trained model to obtain the relative error between the fully connected layer and the convolutional layer; The relative error of the fully connected layer and the relative error of the convolutional layer are convolved with the low-precision forward feature map and the output feature to obtain the weight gradient. The original training parameters are iteratively calculated through the weight gradient to obtain new training parameters, and the new training parameters are input into the random weight averaging module. The random weight averaging module performs sliding average calculation on the new training parameters and the original training parameters to obtain the final training parameters; Calculate the loss of the final training parameters and make a judgment on the loss.
2. The small sample online training method according to claim 1, characterized in that: The calculation formula for batch normalization of the output feature map by the interval batch normalization module is as follows: ; in: is the output feature, is the average output feature value, is the batch normalization result, is the maximum value of the output feature, is the minimum value of the output feature.
3. The small sample online training method according to claim 1, characterized in that: The weight gradient iteratively calculates the original training parameters, and the calculation formula for the new training parameters is as follows: ; ; ; in: Relative error between the fully connected layer and the convolutional layer , is the regularization factor, is the regularization coefficient, The original training parameters include W and w. is the learning rate, The weight gradient contains and , is the new training parameter and t is the number of iterations.
4. The small sample online training method according to claim 1, characterized in that: The calculation formula of the random weight average module for sliding average calculation of new training parameters and original training parameters is as follows: ; Where: g is the preset amount of sample collection, is the original training parameter, is the training parameter after iteration, are the final training parameters.
5. The small sample online training method according to claim 1, characterized in that: Methods for determining loss include: Determine whether the loss is less than 0.1; If the judgment result is yes, the final training parameters are output and applied; If the judgment result is no, the final training parameters and labeled samples are input into the pre-training model for training iteration.
6. The small sample online training method according to claim 1, characterized in that: The prescribed preset number of samples collected is 32.
7. A small sample online training device, characterized in that: including memory and processor; The storage medium is used to store instructions; The processor is used to operate according to the instruction to execute each step of the small sample online training method according to any one of claims 1-6.
8. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the small sample online training method as described in any one of claims 1 to 6 is implemented.
