Model pruning method based on channel attention and adaptive learning

Abstract

The invention discloses a model pruning method based on channel attention and adaptive learning, and the method comprises the steps: firstly introducing a channel attention mechanism SENet, adding the SENet to each convolution layer of a target model, and training the SENet until the model converges; then, extracting the middle output of the SENet of the training sample, combining with a scaling factor in a BN layer to obtain a channel importance score, and then performing pruning according to the channel importance score; and finally, training the pruned model by using an adaptive learning loss function until convergence. According to the method, the model pruning principle and the channel attention principle are fully utilized, and the high-precision model pruning method in the remote sensing field is achieved. The method can greatly reduce the consumption of computing resources, improve the processing efficiency and reduce the hardware cost while maintaining the model precision or even exceeding the model precision before pruning.

Description

Technical Field

[0001] The present invention discloses a model pruning method based on channel attention and adaptive learning mechanism, belongs to the field of computer application, and relates to a model lightweight method. Background Art

[0002] Model pruning technology has the characteristics of reducing the amount of calculation, reducing memory usage and improving the inference speed. It can significantly reduce the complexity of the model while maintaining the accuracy of the model. The model pruning method is highly flexible and suitable for different deep learning models and task scenarios, such as image classification, object detection and speech recognition. Its application fields are wide and the effect is remarkable. In recent years, with the rapid development of artificial intelligence technology, model pruning technology has been widely used in practical applications. For example, lightweight models optimized by pruning can be deployed on embedded devices with limited resources to achieve real-time image processing; models with reduced parameters through pruning can process and analyze large-scale data more efficiently. In addition, model pruning is also applied to emerging fields such as autonomous driving, intelligent security, and edge computing in mobile devices, providing more efficient solutions for these fields. Therefore, combining model pruning technology with existing deep learning frameworks is the core and focus of the current artificial intelligence field, and has important theoretical and practical significance for promoting the development and application of deep learning technology.

[0003] Currently common model pruning methods often ignore the importance differences between model channels, which often leads to a significant decrease in model performance after pruning. For example, application publication number CN118504642A (A machine translation model training method and system based on parameter pruning and tuning) directly extracts channel importance from the trained model; while existing pruning methods only focus on the pruning process without considering the fine-tuning process, resulting in the pruned model having difficulty learning difficult sample features in the fine-tuning stage. For example, application publication number CN118409872A (A model pruning method, device and medium based on memory computing power perception) only considers the pruning process and does not make improvements to the post-pruning fine-tuning stage. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies described in the background technology and provide a model pruning method based on channel attention and adaptive learning. By combining the intermediate output of Squeeze-and-Excitation Network (SENet) with the scaling factor of the Batch Normalization (BN) layer, the channel importance is mapped to the attention space to amplify the channel importance difference, thereby improving the performance of the model after pruning; and introducing an adaptive learning loss function in the post-pruning fine-tuning stage, so that the model focuses on learning difficult samples and reduces the performance loss caused by difficult samples. Finally, our method realizes model pruning suitable for training on remote sensing data, effectively improves the accuracy of the model after pruning, and has good reliability.

[0005] In order to achieve the above object, the present invention is implemented by adopting the following technical solutions.

[0006] A model pruning method based on channel attention and adaptive learning includes the following steps:

[0007] Step 1: Add SENet to the remote sensing image classification model and train it on a remote sensing dataset (such as EuroSAT). i (i represents the training set image number, i.e. Image i Represents the i-th training set image) is input into the remote sensing image classification model, the output value of the image classification model is compared with the actual value, and each output value is derived through the loss function. Then, the error is transmitted to all neurons using the back propagation algorithm. According to the size of the transmitted error, the weights and bias values ​​of neurons in each layer are updated using the stochastic gradient descent method, and the adjustment amplitude is made according to the learning rate each time;

[0008] Step 2, repeat step 1 and set the number of iterations to be sufficient for the model parameters to converge to the optimal value;

[0009] Step 3: Use the remote sensing image classification model trained in step 2 to extract the intermediate outputs of SENet corresponding to the training samples and calculate their average value as the channel attention score;

[0010] Step 4: Use the channel attention score obtained in step 3 and combine it with the scaling factor in the BN layer to get the channel importance score;

[0011] Step 5: Use the channel importance scores obtained in step 4 to prune the channels with small scores according to the pruning rate;

[0012] Step 6: Use the adaptive learning loss function to train the pruned model obtained in step 5. i (i represents the training set image number, i.e. Imagei represents the i-th training set image) as the input of the remote sensing image classification model, and the back propagation algorithm is used to update the parameters;

[0013] Step 7: Repeat step 6 and set the number of iterations to be sufficient for the model parameters to converge to the optimal value.

[0014] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0015] 1.1. After adding SENet to each convolutional layer (including BN layer) of the model, taking ResNet18 as an example, add SENet after each convolutional layer (including BN layer and ReLU activation function (if any)) (a total of 17, excluding the convolutional layers added to keep the data size consistent during residual connection), and adjust the input size of SENet to match the output shape of the previous layer.

[0016] 1.2. Use the stochastic gradient descent method to update the model parameters according to the learning rate. Taking the learning rate of 0.1 as an example, the loss function is used to derive each output value. Calculated using the back propagation algorithm The reverse gradient of the parameters of each layer of the model is then multiplied by the learning rate 0.1 and added to the corresponding model parameters to complete the parameter update.

[0017] As a preferred solution of the present invention, the specific process of calculating the channel attention score in step 3 is:

[0018] 3-1. Extracted SENet intermediate output:

[0019] The operation is performed through two main steps: Squeeze and Excitation. Let the feature map of the mth sample output by the cth convolutional layer be X.

[0020] In the compression step, the feature map output by the cth convolutional layer is average pooled, which is defined as:

[0021]

[0022] in, is the aggregated feature value of the kth channel of the cth convolutional layer of the mth sample. H and W represent the height and width of the feature map, respectively. Represents the value of position (i, j) in the feature map of the kth channel.

[0023] In the excitation step, the output of the compression step After two fully connected layers, denoted as W1 and W2 respectively, the channel attention vector of the cth convolutional layer of the mth sample is obtained The process is expressed as:

[0024]

[0025] Where W1 and W2 are the weight matrices of the fully connected layer, ReLU represents the ReLU activation function, and σ represents the sigmoid activation function.

[0026] 3-2 Calculate channel attention score

[0027] Calculate the average of the channel attention vectors of all selected samples, that is, the channel importance score of the c-th convolutional layer The process is represented as follows:

[0028]

[0029] As a preferred solution of the present invention, the specific process of calculating the channel attention score in step 4 is as follows:

[0030] We take the channel importance score of the c-th convolutional layer The scaling factor γ in the corresponding BN layer c The latter essentially reflects the relative importance of each channel. Finally, the channel importance score I of the cth convolutional layer is c =[I c,1 ,I c,2 ,...,I c,C ]∈R C The calculation process is:

[0031]

[0032] in, represents the attention score of the kth channel of the cth convolutional layer, γ c,k Represents the scaling factor of the kth channel of the cth convolutional layer (including the BN layer).

[0033] As a preferred solution of the present invention, the specific process of step 5 is:

[0034] Assume the pruning rate is α, taking the i-th convolutional layer as an example.

[0035] 5-1 Calculate the critical value of pruning

[0036] Sort the importance scores of each channel of the i-th convolutional layer in descending order, and get the channel importance score S with the ranking percentage in 1-α i .

[0037] 5-2 Pruning using critical values

[0038] The pruning process is shown as follows:

[0039]

[0040] Among them, result i,k represents the pruning result of the kth channel of the i-th convolutional layer, 0 means pruning, 1 means keeping, x i,x Represents the channel importance score of the kth channel of the i-th convolutional layer. Perform the same pruning operation on all layers that need to be pruned, and then obtain the pruned model.

[0041] As a preferred solution of the present invention, the specific process of training using the adaptive learning loss function in step 6 is:

[0042] The adaptive learning loss function is expressed as follows:

[0043]

[0044] Among them, L CE is the cross entropy loss function, y is the label of the true value, is the predicted value, r1 and r2 are hyperparameters (specify specific values ​​before use).

[0045] 6-1 Cross Entropy Loss Function

[0046] The cross entropy loss function is one of the commonly used loss functions in classification tasks and is mainly used to stabilize the model training process. It helps the model achieve accurate classification by penalizing the difference between the predicted value and the true label.

[0047] 6-2 Exponential Function

[0048] Exponential function A weighted penalty mechanism is introduced, which increases with the prediction error As increases, the penalty also increases, which effectively enhances the model’s attention to difficult samples.

[0049] 6-3 Model Training

[0050] Image i (i represents the training set image number, i.e. Image i Represents the i-th training set image) is input into the pruned remote sensing image classification model, the output value of the pruned image classification model is compared with the actual value, and each output value is derived through the adaptive learning loss function. Then, the error is transmitted to all neurons using the back propagation algorithm. According to the size of the transmitted error, the weights and bias values ​​of neurons in each layer are updated using the stochastic gradient descent method, and the adjustment amplitude is made according to the learning rate each time.

[0051] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0052] 1. The model pruning method based on channel attention of the present invention solves the problem of insufficient channel importance difference in existing pruning methods when processing high-resolution remote sensing images by introducing the channel attention mechanism. This method not only improves the accuracy of pruning, but also effectively reduces the number of model parameters and significantly reduces the computational cost. It is particularly suitable for processing complex scenes such as remote sensing images and has strong adaptability and reliability.

[0053] 2. The present invention adopts a post-pruning fine-tuning method based on adaptive learning. After pruning the model, it focuses on optimizing samples that are difficult to learn, so that the pruned lightweight model can still effectively classify complex samples while maintaining a high inference speed. This method further improves the overall performance of the model by strengthening the model's attention to difficult samples, and is suitable for application scenarios that require efficient processing of diverse data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a model pruning method based on channel attention and adaptive learning of the present invention;

[0055] Figure 2 It is a structural diagram of the model pruning method based on channel attention and adaptive learning of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the attached Figures 1-2 The method of the present invention is described in detail with specific embodiments.

[0057] It is worth mentioning that the embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0058] With the increasing maturity of model pruning technology, more and more pruning technologies have been applied in deep learning model optimization, which has attracted widespread attention and high praise. However, existing model pruning techniques usually only focus on the importance measurement of channels, ignoring the importance differences between channels in different tasks, which may lead to the performance degradation of the pruned model in specific scenarios. In addition, when processing complex data, existing methods are difficult to balance the accuracy and inference speed of the pruned model. The combination of channel attention and adaptive learning can amplify the importance differences between channels, and at the same time further improve the overall performance of the model by strengthening the model's learning ability for difficult samples. Based on this idea, the present invention proposes a model pruning method based on channel attention and adaptive learning.

[0059] like Figure 1 As shown, the present invention proposes a model pruning method based on channel attention and adaptive learning, comprising the following steps:

[0060] Step 1: Add SENet to each convolutional layer (including BN layer) of the model. Taking ResNet18 as an example, add SENet after each convolutional layer (including BN layer and ReLU activation function (if any)) (a total of 17, excluding the convolutional layer added to keep the data size consistent when residual connection is used), and adjust the input size of SENet to match the output shape of the previous layer. i (i represents the training set image number, i.e. Image i Represents the i-th training set image) is input into the remote sensing image classification model, the output value of the image classification model is compared with the actual value, and each output value is derived through the cross entropy loss function. Then, the error is transmitted to all neurons using the back propagation algorithm. According to the size of the transmitted error, the weights and bias values ​​of neurons in each layer are updated using the stochastic gradient descent method, and the adjustment amplitude is made according to the learning rate each time.

[0061] Step 2, repeat step 1 and set the number of iterations to be sufficient for the model parameters to converge to the optimal value;

[0062] Step 3: Use the remote sensing image classification model trained in step 2 to extract the intermediate outputs of SENet corresponding to the training samples and calculate their average value as the channel attention score;

[0063] 3-1. Extracted SENet intermediate output:

[0064] The operation is performed through two main steps: Squeeze and Excitation. Let the feature map of the mth sample output by the cth convolutional layer be X.

[0065] In the compression step, the feature map output by the cth convolutional layer is average pooled, which is defined as:

[0066]

[0067] in, is the aggregated feature value of the kth channel of the cth convolutional layer of the mth sample. H and W represent the height and width of the feature map, respectively. Represents the value of position (i, j) in the feature map of the kth channel.

[0068] In the excitation step, the output of the compression step After two fully connected layers, denoted as W1 and W2 respectively, the channel attention vector of the cth convolutional layer of the mth sample is obtained The process is expressed as:

[0069]

[0070] Where W1 and W2 are the weight matrices of the fully connected layer, ReLU represents the ReLU activation function, and σ represents the sigmoid activation function.

[0071] 3-2 Calculate channel attention score

[0072] Calculate the average of the channel attention vectors of all selected samples, that is, the channel importance score of the c-th convolutional layer The process is represented as follows:

[0073]

[0074] Step 4: Use the channel attention score obtained in step 3 and combine it with the scaling factor in the BN layer to get the channel importance score;

[0075] We take the channel importance score of the c-th convolutional layer The scaling factor γ in the corresponding BN layer c The latter essentially reflects the relative importance of each channel. Finally, the channel importance score I of the cth convolutional layer is c =[I c,1 ,I c,2 ,...,I c,C ]∈R C The calculation process is:

[0076]

[0077] in, represents the attention score of the kth channel of the cth convolutional layer, γ c,k Represents the scaling factor of the kth channel of the cth convolutional layer (including the BN layer).

[0078] Step 5: Use the channel importance scores obtained in step 4 to prune the channels with small scores according to the pruning rate;

[0079] Assume the pruning rate is α, taking the i-th convolutional layer as an example.

[0080] 5-1 Calculate the critical value of pruning

[0081] Sort the importance scores of each channel of the i-th convolutional layer in descending order, and get the channel importance score S with the ranking percentage in 1-α i .

[0082] 5-2 Pruning using critical values

[0083] The pruning process is shown as follows:

[0084]

[0085] Among them, result i,k represents the pruning result of the kth channel of the i-th convolutional layer, 0 means pruning, 1 means keeping, x i,k Represents the channel importance score of the kth channel of the i-th convolutional layer. Perform the same pruning operation on all layers that need to be pruned, and then obtain the pruned model.

[0086] Step 6: Use the adaptive learning loss function to train the pruned model obtained in step 5. i (i represents the training set image number, i.e. Image i represents the i-th training set image) as the input of the remote sensing image classification model, and the back propagation algorithm is used to update the parameters;

[0087] The specific process of training using the adaptive learning loss function is:

[0088] The adaptive learning loss function is specifically expressed as follows:

[0089]

[0090] Among them, L CE is the cross entropy loss function, y is the label of the true value, is the predicted value, r1 and r2 are hyperparameters (specify specific values ​​before use).

[0091] 6-1 Cross Entropy Loss Function

[0092] The cross entropy loss function is one of the commonly used loss functions in classification tasks and is mainly used to stabilize the model training process. It helps the model achieve accurate classification by penalizing the difference between the predicted value and the true label.

[0093] 6-2 Exponential Function

[0094] Exponential function A weighted penalty mechanism is introduced, which increases with the prediction error As increases, the penalty also increases, which effectively enhances the model’s attention to difficult samples.

[0095] 6-3 Model Training

[0096] Image i (i represents the training set image number, i.e. Image iRepresents the i-th training set image) is input into the pruned remote sensing image classification model, the output value of the pruned image classification model is compared with the actual value, and each output value is derived through the adaptive learning loss function. Then, the error is transmitted to all neurons using the back propagation algorithm. According to the size of the transmitted error, the weights and bias values ​​of neurons in each layer are updated using the stochastic gradient descent method, and the adjustment amplitude is made according to the learning rate each time.

[0097] Step 7: Repeat step 6 and set the number of iterations to be sufficient for the model parameters to converge to the optimal value.

[0098] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A model pruning method based on channel attention and adaptive learning, characterized in that: The steps include: Step 1: Add SENet to the remote sensing image classification model and train it on the remote sensing dataset, then i Input into the remote sensing image classification model, compare the output value of the image classification model with the actual value, and derive each output value through the loss function, then use the back propagation algorithm to transfer the error to all neurons, and use the stochastic gradient descent method to update the weights and bias values ​​of each layer of neurons according to the size of the transferred error. Each adjustment is made according to the learning rate, and the learning rate is 0.1-0.3; The i represents the training set image number, Image i represents the i-th training set image; Step 2, repeat step 1 and set the number of iterations to be sufficient for the model parameters to converge to the optimal value; Step 3: Use the remote sensing image classification model trained in step 2 to extract the intermediate outputs of SENet corresponding to the training samples and calculate their average value as the channel attention score; Step 4: Use the channel attention score obtained in step 3 and combine it with the scaling factor in the BN layer to get the channel importance score; Step 5: Use the channel importance scores obtained in step 4 to prune the channels with small scores according to the pruning rate; Step 6: Use the adaptive learning loss function to train the pruned model obtained in step 5. i As the input of the remote sensing image classification model, the back propagation algorithm is used to update the parameters; Step 7: Repeat step 6 and set the number of iterations to be sufficient for the model parameters to converge to the optimal value.

2. A model pruning method based on channel attention and adaptive learning according to claim 1, characterized in that: The specific process described in step 1 is: 1.

1. Add SENet. The steps are as follows: After adding SENet to each convolutional layer of the model, add SENet after each convolutional layer and adjust the input size of SENet to match the output shape of the previous layer; The convolutional layers include BN layers, and the SENet adds 17 of them; 1.

2. Use the stochastic gradient descent method to update the model parameters according to the learning rate and derive each output value through the loss function. Calculated using the back propagation algorithm The reverse gradient of the parameters of each layer of the model is then multiplied by the learning rate and added to the corresponding model parameters to complete the parameter update. The learning rate can be any one of 0.1-0.

3.

3. A model pruning method based on channel attention and adaptive learning according to claim 1, characterized in that: The specific process described in step 3 is: 3-1. Extracted SENet intermediate output, which includes compression step and excitation step. In the compression step, first, the feature map of the mth sample output at the cth convolutional layer is set to X, and then the feature map output by the cth convolutional layer is average pooled. The operation is defined as: in, is the aggregated feature value of the kth channel of the cth convolutional layer of the mth sample. H and W represent the height and width of the feature map, respectively. Represents the value of position (i, j) in the feature map of the kth channel; In the excitation step, the output of the compression step After two fully connected layers, denoted as W1 and W2 respectively, the channel attention vector of the cth convolutional layer of the mth sample is obtained The process is expressed as: Where W1 and W2 are the weight matrices of the fully connected layer, ReLU represents the ReLU activation function, and σ represents the sigmoid activation function; 3-2. Calculate channel attention score Calculate the average of the channel attention vectors of all selected samples, that is, the channel importance score of the c-th convolutional layer The process is represented as follows:

4. A model pruning method based on channel attention and adaptive learning according to claim 1, characterized in that: The specific process described in step 4 is: The channel importance score of the c-th convolutional layer The scaling factor γ in the corresponding BN layer c Combined, the final channel importance score I of the cth convolutional layer c =[I c,1 ,I c,2 ,...,I c,C ]∈R C The calculation process is: in, represents the attention score of the kth channel of the cth convolutional layer, γ c,k represents the scaling factor of the kth channel of the cth convolutional layer including the BN layer.

5. A model pruning method based on channel attention and adaptive learning according to claim 1, characterized in that: The specific process described in step 5 is: First, set the pruning rate to α, taking the i-th convolutional layer as an example; 5-1 Calculate the critical value of pruning Sort the importance scores of each channel of the i-th convolutional layer in descending order, and get the channel importance score S with the ranking percentage in 1-α i ; 5-2 Pruning using critical values The pruning process is expressed as follows: Among them, result i,k represents the pruning result of the kth channel of the i-th convolutional layer, 0 means pruning, 1 means keeping, x i,k Represents the channel importance score of the kth channel of the i-th convolutional layer. Perform the same pruning operation on all layers that need to be pruned, and then obtain the pruned model.

6. A model pruning method based on channel attention and adaptive learning according to claim 1, characterized in that: The adaptive learning loss function L described in step 6 AM It is expressed as follows: Among them, L CE is the cross entropy loss function, y is the label of the true value, is the predicted value, is an exponential function, r1 and r2 are hyperparameters; 6-1 Cross Entropy Loss Function The cross entropy loss function is one of the commonly used loss functions in classification tasks. It is mainly used to stabilize the model training process. It helps the model achieve accurate classification by penalizing the difference between the predicted value and the true label. 6-2 Exponential Function Exponential function A weighted penalty mechanism is introduced, which increases with the prediction error As increases, the penalty also increases, which effectively enhances the model's attention to difficult samples; 6-3 Model Training Image i (i represents the training set image number, i.e. Image i Represents the i-th training set image) is input into the pruned remote sensing image classification model, the output value of the pruned image classification model is compared with the actual value, and each output value is derived through the adaptive learning loss function. Then, the back propagation algorithm is used to transfer the error to all neurons. According to the size of the transmitted error, the weights and bias values ​​of neurons in each layer are updated using the stochastic gradient descent method, and the amplitude of each adjustment is based on the learning rate.

Images