Convolution kernel activation value regularization method and system based on convolutional neural network pre-training model

Abstract

A convolution kernel activation value regularization method and system based on a convolutional neural network pre-training model, the method comprising: 1) pre-training a convolutional neural network model; 2) calculating the importance of activation values ​​generated by the convolution kernel; 3) regularizing the activation values ​​generated by the convolution kernel to generate new activation values, and using the new activation values ​​to replace the original activation values; and 4) classifying images based on the convolutional neural network model after regularization. The present invention finds the importance of the activation values ​​generated by each convolution kernel in the convolutional neural network pre-training model, and regularizes the convolution kernel activation values ​​based on the importance of the activation values. A loss function is calculated based on the classification results, and the parameters of the convolutional neural network are updated using the loss function to improve the image classification performance of the pre-trained convolutional neural network.

Description

Technical Field

[0001] The present invention belongs to the field of model optimization, and specifically relates to a convolution kernel activation value regularization method and system for a convolutional neural network pre-training model. Background Art

[0002] A convolutional neural network (CNN) is a type of deep neural network that incorporates convolutional computations and has a deep hierarchical structure. Convolutional layers and pooling layers are alternately connected within the hidden layers of a CNN. These hidden layers form the core building block of a CNN, while higher layers typically consist of fully connected layers. Convolutional neural networks are currently widely used in areas such as image classification, similarity search, and medical imaging.

[0003] However, although convolutional neural networks have powerful image data processing capabilities, researchers believe that there is still a lot of room for optimization, such as designing different network structures, designing different loss functions, and using more complex optimizers.

[0004] In recent years, the progress of convolution kernel activation value regularization methods has mainly focused on model quantization, model pruning, model distillation, network structure design, loss function design, etc. The technical direction of regularizing the convolution kernel activation values ​​of pre-trained convolutional neural network models is still in a blank stage.

[0005] Model quantization significantly reduces computing and storage resource usage by reducing the precision of model parameters to a lower number of bits, while having a relatively small impact on model performance. Model pruning reduces the size of the model by removing unnecessary connections and nodes, thereby reducing resource requirements. Model distillation improves the performance of small models by transferring knowledge from large models to small models. Furthermore, network structure design allows the model to be reasoned on low-computing devices such as mobile phones. Targeted loss functions can be designed based on data distribution, allowing the model to converge better. Although these techniques optimize the model to a certain extent, they do not take into account the convolution kernel activation value and its impact on downstream tasks such as image classification. Summary of the Invention

[0006] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a convolution kernel activation value regularization method and system based on a convolutional neural network pre-training model.

[0007] Based on the importance of activation values ​​generated by convolution kernels in a convolutional neural network model, this paper designs a convolution kernel activation value regularization method based on a convolutional neural network pre-training model. The core idea of ​​this method is to analyze the importance of activation values ​​generated by convolution kernels in a convolutional neural network pre-training model, and further regularize the activation values ​​based on their importance to obtain new activation values. The new activation values ​​are then used to replace the original activation values, thereby improving the performance of the convolutional neural network pre-training model.

[0008] The convolution kernel activation value regularization method based on the convolutional neural network pre-training model includes the following steps:

[0009] S1. Pre-trained convolutional neural network model.

[0010] Given any convolutional neural network model (such as AlexNet or ResNet), use the cross entropy loss function to train it on an image classification dataset (such as CIFAR10 or CIFAR100) until the model converges and obtains a pre-trained weight; this weight will be used as the initialization of the model in the following steps.

[0011] S2. Calculate the importance of the activation value generated by the convolution kernel.

[0012] S2.1 Initialize the convolutional neural network model using pre-trained weights.

[0013] Use the same convolutional neural network as in step S1 and initialize the convolutional neural network model using the pre-trained weights obtained in step S1;

[0014] S2.2 obtains the mean and variance of the distribution corresponding to the convolution kernel activation value.

[0015] In convolutional neural networks, the activation values ​​of the convolution kernel after passing through the convolution kernel and batch normalization layer are Gaussian distributed. For the convolution kernel k on the convolution layer of the convolutional neural network, the mean of the Gaussian distribution corresponding to its activation value is set to μ k , the standard deviation is set to σ k , these two parameter values ​​are directly obtained from the model parameters of the batch normalization layer after the lth convolutional layer;

[0016] S2.3 calculates the importance of the activation value generated by the convolution kernel.

[0017] For a single image, the activation value generated by the convolution kernel k of the lth convolution layer is set to a k ; If the activation value a k The closer to the mean μ of the corresponding convolution kernel k k , this activation value needs to be given a larger weight; if the activation value a k The further away from the mean μ of the convolution kernel k k, then the activation value needs to be given a smaller weight; according to the definition of the importance of the convolution kernel activation value in this form, the importance of the activation value can be calculated using a variant of the Gaussian function integral: if the activation value is less than or equal to the mean, then take the integral of the Gaussian function at the activation value position; if the activation value is greater than the mean, then take the integral of the activation value symmetrically about the mean; the formula is as follows:

[0018] For a single image, the activation value a generated on the convolution kernel k k , if a k ≤μ k , then the weight value w k The calculation formula is as follows:

[0019]

[0020] Among them, μ k is the mean of the convolution kernel k, σ k is the variance of the convolution kernel k, exp represents the exponential function with the natural constant e as the base, and π represents pi; the above expression can be simplified using the error function:

[0021]

[0022] Among them, a k Represents the activation value generated by the convolution kernel k, μ k is the mean of the convolution kernel k, σ k is the variance of the convolution kernel k, erf represents the error function;

[0023] For a single image, the activation value a generated by the convolution kernel k k , if a k >μ k , weight value w k is the activation value a k About the mean μ k The Gaussian function integral at the symmetrical position is calculated as follows:

[0024]

[0025] In summary, the importance of the activation value generated by a single image on the convolution kernel k is as follows:

[0026]

[0027] S3. Regularize the activation value generated by the convolution kernel to generate a new activation value, and use the new activation value to replace the original activation value.

[0028] The activation value generated by the convolution kernel k of a single image in the lth convolution layer is a k , the new activation value after regularization is wk *a k , and use the new activation value to replace the original activation value, where w k is the activation value a calculated by formula (4) k The cross entropy loss function is then used to update the model parameters through the back propagation algorithm to improve the performance of the model.

[0029] The innovation of the present invention is that the mean and variance of the distribution of the convolution kernel activation value are directly extracted from the weights of the pre-trained convolutional neural network model, without adding other modules to the original convolutional neural network model and without increasing the number of model parameters; the weight of the activation value is calculated by the integral corresponding to the Gaussian function.

[0030] The working principle of the present invention is that when the activation value generated by the convolution kernel is closer to the mean of the distribution corresponding to the convolution kernel, it means that the activation value is within a reasonable range and its importance is high; if the activation value is further away from the mean of the distribution corresponding to the convolution kernel, it means that the activation value is an outlier and its importance is low. By calculating the importance of the activation value, the activation value can be regularized and the activation value after regularization is used instead of the original activation value, so that the convolutional neural network pays more attention to relevant features and filters out irrelevant features.

[0031] The advantages of the present invention are: this method does not require adding additional modules to the convolutional neural network model, and does not increase the number of model parameters; the new activation values ​​processed by this method can pay more attention to features related to the image classification task, while suppressing irrelevant features. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described below with reference to the accompanying drawings.

[0034] This embodiment provides an image classification method using the convolution kernel activation value regularization method based on a convolutional neural network pre-training model of the present invention, comprising the following steps:

[0035] S1. Pre-trained convolutional neural network model.

[0036] Given any convolutional neural network model (such as AlexNet or ResNet), use the cross entropy loss function to train it on an image classification dataset (such as CIFAR10 or CIFAR100) until the model converges and obtains a pre-trained weight; this weight will be used as the initialization of the model in the following steps.

[0037] S2. Calculate the importance of the activation value generated by the convolution kernel.

[0038] S2.1 Initialize the convolutional neural network model using pre-trained weights.

[0039] Use the same convolutional neural network as in step S1 and initialize the convolutional neural network model using the pre-trained weights obtained in step S1;

[0040] S2.2 obtains the mean and variance of the distribution corresponding to the convolution kernel activation value.

[0041] In convolutional neural networks, the activation values ​​of the convolution kernel after passing through the convolution kernel and batch normalization layer are Gaussian distributed. For the convolution kernel k on the convolution layer of the convolutional neural network, the mean of the Gaussian distribution corresponding to its activation value is set to μ k , the standard deviation is set to σ k , these two parameter values ​​are directly obtained from the model parameters of the batch normalization layer after the lth convolutional layer;

[0042] S2.3 calculates the importance of the activation value generated by the convolution kernel.

[0043] For a single image, the activation value generated by the convolution kernel k of the lth convolution layer is set to a k ; If the activation value a k The closer to the mean μ of the corresponding convolution kernel k k , this activation value needs to be given a larger weight; if the activation value a k The further away from the mean μ of the convolution kernel k k , then the activation value needs to be given a smaller weight; according to the definition of the importance of the convolution kernel activation value in this form, the importance of the activation value can be calculated using a variant of the Gaussian function integral: if the activation value is less than or equal to the mean, then take the integral of the Gaussian function at the activation value position; if the activation value is greater than the mean, then take the integral of the activation value symmetric to the mean position; the formula is as follows:

[0044] For a single image, the activation value a generated on the convolution kernel k k , if a k ≤μ k , then the weight value w k The calculation formula is as follows:

[0045]

[0046] Among them, μ k is the mean of the convolution kernel k, σ k is the variance of the convolution kernel k, exp represents the exponential function with the natural constant e as the base, and π represents pi; the above expression can be simplified using the error function:

[0047]

[0048] Among them, a k Represents the activation value generated by the convolution kernel k, μ k is the mean of the convolution kernel k, σ k is the variance of the convolution kernel k, erf represents the error function;

[0049] For a single image, the activation value a generated by the convolution kernel k k , if a k >μ k , weight value w k is the activation value a k About the mean μ k The Gaussian function integral at the symmetrical position is calculated as follows:

[0050]

[0051] In summary, the importance of the activation value generated by a single image on the convolution kernel k is as follows:

[0052]

[0053] S3. Regularize the activation value generated by the convolution kernel to generate a new activation value, and use the new activation value to replace the original activation value.

[0054] The activation value generated by the convolution kernel k of a single image in the lth convolution layer is a k , the new activation value after regularization is w k *a k , and use the new activation value to replace the original activation value, where w k is the activation value a calculated by formula (4) k the importance of.

[0055] S4. Image classification based on the convolutional neural network model after regularization processing.

[0056] Use the regularized convolutional neural network model to classify images. Calculate the cross-entropy loss based on the classification results, and use the loss function to update the convolutional neural network parameters to further improve the image classification performance of the pre-trained convolutional neural network.

[0057] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. The convolution kernel activation value regularization method based on the convolutional neural network pre-training model includes the following steps: S1. Pre-trained convolutional neural network model; Given any convolutional neural network model, use the cross entropy loss function to train it on an image classification dataset until the model converges and obtain a pre-trained weight; this weight will be used as the initialization of the model in the following steps; S2. Calculate the importance of the activation value generated by the convolution kernel; S2.1 Initialize the convolutional neural network model using pre-trained weights; Use the same convolutional neural network as in step S1 and initialize the convolutional neural network model using the pre-trained weights obtained in step S1; S2.2 obtains the mean and variance of the distribution corresponding to the convolution kernel activation value; For the convolution kernel k on the lth convolution layer of the convolutional neural network, the mean of the Gaussian distribution corresponding to its activation value is μ k , with a standard deviation of σ k , these two parameter values ​​are directly obtained from the model parameters of the batch normalization layer after the lth convolutional layer; S2.3 calculates the importance of the activation value generated by the convolution kernel; For a single image, the activation value generated by the convolution kernel k of the lth convolution layer is set to a k ; If the activation value a k The closer to the mean μ of the corresponding convolution kernel k k , this activation value needs to be given a larger weight; if the activation value a k The further away from the mean μ of the convolution kernel k k , then the activation value needs to be given a smaller weight; according to the definition of the importance of the convolution kernel activation value in this form, the importance of the activation value can be calculated using a variant of the Gaussian function integral: if the activation value is less than or equal to the mean, then take the integral of the Gaussian function at the activation value position; if the activation value is greater than the mean, then take the integral of the activation value symmetric to the mean position; the formula is as follows: For a single image, the activation value a generated on the convolution kernel k k , if a k ≤μ k , then the weight value w k The calculation formula is as follows: Among them, μ k is the mean of the convolution kernel k, σ k is the variance of the convolution kernel k, exp represents the exponential function with the natural constant e as the base, and π represents pi; the above expression can be simplified using the error function: Among them, a k Represents the activation value generated by the convolution kernel k, μ k is the mean of the convolution kernel k, σ k is the variance of the convolution kernel k, erf represents the error function; For a single image, the activation value a generated by the convolution kernel k k , if a k >μ k , weight value w k is the activation value a k About the mean μ k The Gaussian function integral at the symmetrical position is calculated as follows: In summary, the importance of the activation value generated by a single image on the convolution kernel k is as follows: S3. Regularize the activation value generated by the convolution kernel to generate a new activation value, and use the new activation value to replace the original activation value; The activation value generated by the convolution kernel k of a single image in the lth convolution layer is a k , the new activation value after regularization is w k *a k , and use the new activation value to replace the original activation value, where w k is the activation value a calculated by formula (4) k The importance of the model is then determined, and the cross entropy loss function is used. The model parameters are updated through the back propagation algorithm to improve the performance of the model.

2. The convolution kernel activation value regularization method based on a convolutional neural network pre-training model according to claim 1, characterized in that: The convolutional neural network model described in step S1 is AlexNet or ResNet.

3. The convolution kernel activation value regularization method based on a convolutional neural network pre-training model according to claim 1, characterized in that: The image classification dataset described in step S1 is CIFAR10 or CIFAR100.

Images