Complex network structure-oriented progressive model compression method

By aligning the filter pruning and channel count of complex network structures and replacing convolution operations, lightweight compression on complex structural models is achieved, solving the problem that the existing technology is difficult to high requirements at the same time in detection speed and accuracy. Significantly reduces the amount of parameters and calculations.

CN119047524AActive Publication Date: 2024-11-29HARBIN INST OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411158489.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The existing lightweight network design methods are difficult to apply to basic models with complex structures, and cannot meet high requirements at the same time as detection speed and detection accuracy.

Method used

A progressive model compression method for complex network structures is proposed, which aligns with the number of channels through filter pruning, and replaces the convolution operation in the convolution layer with lightweight convolution operation, combined with backpropagation algorithm optimization Loss function to implement model compression.

Benefits of technology

Without significantly reducing the detection accuracy, the parameter amount, calculation amount and memory usage of the model are significantly reduced, and are suitable for basic models with complex structures, especially in the motion object detection model in the remote sensing field, which achieves the improvement of detection speed and accuracy. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047524A_ABST
    Figure CN119047524A_ABST
Patent Text Reader

Abstract

The invention discloses a progressive model compression method for a complex network structure, belongs to the field of convolutional neural network lightweight, and relates to the progressive model compression method for the complex network structure. The invention aims to solve the problems that an existing lightweight network design method is difficult to apply to a basic model with a complex structure, and an existing lightweight network is not suitable for scenes with high requirements on detection speed and detection precision at the same time. The method comprises the following steps of: 1, performing filter pruning and channel number alignment on a target model for image recognition; 2, giving a loss function of the target model after finishing filter pruning and channel number alignment in the step 1; the feature map is input into the target model after filter pruning and channel number alignment are completed in the first step, yl is output by each convolution layer l of the target model, the gradient T1 of a loss function relative to yl is calculated through a back propagation algorithm, and according to the gradient, convolution operation in the convolution layers is replaced with lightweight convolution operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of lightweight convolutional neural networks and relates to a progressive model compression method for complex network structures. Background Art

[0002] With the continuous development of deep learning in recent years, more and more neural network models have been applied in various fields such as target detection and semantic segmentation. Compared with traditional algorithms, network models based on deep learning algorithms have the advantages of high accuracy and wide application range, but they also have obvious shortcomings such as large data input, complex network structure, and high model calculation consumption. For example, the moving target detection network for satellite videos has high detection accuracy, but the model is complex and occupies a large amount of memory. It is difficult to deploy directly on the satellite platform with limited resources, and cannot meet the requirements of real-time detection applications.

[0003] There are two main lightweight network design methods:

[0004] (I) For residual structures with simple structures and low coupling, such as the Resnet series, weight reset, parameter quantization, pruning and other methods are used to locally compress the model size. However, this method is not universal and is difficult to effectively apply to complex network structures with high front-to-back correlation and multi-level nonlinear output, such as DLA34.

[0005] (2) Directly use existing lightweight networks such as MoblieNet to replace the backbone network in the original model. However, this method often comes at the cost of significantly reducing model functionality and is not suitable for scenarios that have high requirements for both detection speed and detection accuracy. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the existing lightweight network design methods are difficult to apply to basic models with complex structures, and the existing lightweight networks are not suitable for scenarios with high requirements on both detection speed and detection accuracy, and to propose a progressive model compression method for complex network structures.

[0007] A progressive model compression method for complex network structures includes the following steps:

[0008] Step 1: Prune the filter and align the number of channels of the target model used for image recognition;

[0009] Step 2: Given the loss function L(θ) of the target model after filter pruning and channel number alignment in step 1;

[0010] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l, use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0011] According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation.

[0012] Preferably, in step 1, filter pruning and channel number alignment are performed on the target model for image recognition; the specific process is:

[0013] Step 11: Perform channel pruning on the filters in each convolutional layer except the output layer in the target model for image recognition to obtain the channel pruned filters in each convolutional layer;

[0014] Step 12: Align the number of channels of the weight matrix of the last filter after channel pruning in the last convolutional layer before the output layer in the target model with the number of channels of the output layer of the target model.

[0015] Preferably, in step 11, channel pruning is performed on the filters in each convolution layer except the output layer in the target model for image recognition to obtain the filters after channel pruning in each convolution layer;

[0016] The specific process is:

[0017] Select all J convolutional layers except the output layer in the target model;

[0018] The jth convolutional layer contains F I filter, filter F i The weight matrix is ​​represented as W i ;

[0019] j=1,2,…,J;i=1,2,…,I, i is the index of the filter;

[0020] Calculate the filter F i The weight matrix W i The L1 norm of is expressed as:

[0021]

[0022] Among them, L1 i Denotes the filter F i The weight matrix W i The L1 norm of || ||1 represents the L1 norm, W i [k] represents the weight matrix W i The kth element in ;

[0023] For all F contained in the jth convolutional layer I The L1 norms of the filters are sorted from large to small;

[0024] The mask M of the filter corresponding to the L1 norm of the first N% of the sorting i′ The value is set to 1, and the mask M of the remaining filter i′ The value is set to 0; i′=1, 2, …, I, i′ is the index of the mask;

[0025] Based on the filter F i The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ;

[0026] Set the weight matrix W to 0 i′ The corresponding filters are deleted to complete the filter pruning of the j-th convolutional layer;

[0027] This continues until all filters in all convolutional layers are pruned.

[0028] Preferably, the filter based F i The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ;

[0029] The calculation is as follows:

[0030] W i′ =W i ⊙M i′

[0031] Among them, ⊙ represents element-wise multiplication.

[0032] Preferably, in step 12, the number of channels of the weight matrix of the last filter after pruning in the last convolutional layer before the output layer in the target model is aligned with the number of channels of the output layer of the target model; the specific process is:

[0033] Perform 1×1 convolution on the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning to obtain a new feature tensor;

[0034] The new feature tensor X i ' is expressed as:

[0035]

[0036] Among them, X i Represents the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning;

[0037] Cin Indicates the number of channels of the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning;

[0038] C represents the number of channels;

[0039] W 1×1 (c,:) represents the weight of the 1×1 convolution kernel on the cth channel.

[0040] Preferably, in step 2, the loss function L(θ) of the target model after filter pruning and channel number alignment are completed in step 1 is given;

[0041] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0042] According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation;

[0043] The specific process is:

[0044] Given step 1, the loss function L(θ) of the target model after filter pruning and channel number alignment is completed;

[0045] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0046]

[0047] Calculate the gradient T l The L2 norm L2(T l );

[0048] The gradient T l The L2 norm L2(T l ) are sorted from large to small, and the convolution operations in the last M% of the convolutional layers are replaced with lightweight convolution operations.

[0049] Preferably, the calculated gradient T l The L2 norm L2(T l ); the expression is:

[0050]

[0051] In the formula, || ||2 represents the L2 norm;

[0052] i represents the feature map y l The width of j represents the feature map y l The height of k represents the feature map y l The number of channels.

[0053] Preferably, the lightweight convolution operation includes channel-by-channel convolution and point-wise convolution;

[0054] The specific processing process of lightweight convolution operation is:

[0055] Each channel of the input feature map passes through a dilation kernel with a dilation rate of r = 2, and the outputs of all convolution kernels are concatenated along the channel dimension to obtain the intermediate feature map;

[0056] Perform a 1×1 point convolution on the intermediate feature map and output the final feature map as the output of the lightweight convolution operation.

[0057] Preferably, the value of N is: 30≤N≤60;

[0058] The value of M is: 40≤M≤50.

[0059] The beneficial effects of the present invention are:

[0060] The purpose of the present invention is to solve the problem that the existing lightweight network design methods are difficult to apply to basic models with complex structures, and the existing lightweight networks are not suitable for scenarios with high requirements for both detection speed and detection accuracy. The present invention proposes a progressive model compression method for complex network structures. On the basis of preliminary compression of the original model by channel pruning, lightweight convolution operations are further designed and replaced by convolution operations. Finally, the basic model with complex structure is effectively simplified in terms of parameters, calculation amount and other indicators with a small sacrifice of accuracy. The method is suitable for scenarios with high requirements for both detection speed and detection accuracy.

[0061] The lightweight method described in the present invention combines pruning with the design and replacement of lightweight convolutional models to progressively compress complex network models. Compared with the existing single lightweight method, the model compression effect is more significant, and there is a significant reduction in the three indicators of parameter quantity, calculation amount, and memory usage. In addition, the existing lightweight methods have certain limitations and are difficult to apply to neural network models with complex structures. The present method has a wider range of applications and can be applied not only to basic models with simple structures, but also to basic models with complex structures (moving target detection models in the field of remote sensing).

[0062] The present invention is based on the idea of ​​redundancy removal and performs progressive compression without changing the original architecture of the model, thereby ensuring the detection effect of the compressed model to the greatest extent.

[0063] In order to verify the performance of the algorithm proposed in the present invention, an existing remote sensing image motion target detection model (DSFNet) is selected as the target model, and a lightweight experiment is performed on it on the "Jilin-1" video satellite dataset. The experimental results verify the implementation of the network lightweight method based on model compression proposed in the present invention on a basic model with complex structure (motion target detection model in the remote sensing field), and its effectiveness in terms of parameter quantity, computational complexity, and memory usage, which meets the scenario with high requirements for both satellite video image detection speed and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the implementation process of the present invention. DETAILED DESCRIPTION

[0065] Specific implementation method 1: This implementation method is a progressive model compression method for complex network structures, comprising the following steps:

[0066] Step 1: Prune the filter and align the number of channels of the target model used for image recognition;

[0067] Step 2: Given the loss function L(θ) of the target model after filter pruning and channel number alignment in step 1;

[0068] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0069] According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation;

[0070] The complex network structure is a target model;

[0071] The target model is a deep learning model including a target detection model, a target tracking model, a semantic segmentation model, etc., such as a satellite video motion target detection model (DSFNet)

[0072] The image is a satellite video image;

[0073] The present invention proposes a progressive model compression method for complex network structures. Based on the idea of ​​de-redundancy, the model is firstly compressed based on channel pruning and channel alignment while retaining the original architecture. On this basis, lightweight convolution is used to replace ordinary convolution to further compress the model, ultimately achieving the purpose of sacrificing accuracy within a reasonable range, greatly reducing the number of complex model parameters and calculations, and reducing storage space occupancy.

[0074] In step 1, filter pruning and channel number alignment are performed on the target model for image recognition; the specific process is as follows:

[0075] Step 11: Perform channel pruning on the filters in each convolutional layer except the output layer in the target model for image recognition to obtain the channel pruned filters in each convolutional layer;

[0076] Step 12: Align the number of channels of the weight matrix of the last filter after channel pruning in the last convolutional layer before the output layer in the target model with the number of channels of the output layer of the target model.

[0077] In the step 11, channel pruning is performed on the filters in each convolutional layer except the output layer in the target model for image recognition to obtain the filter after channel pruning in each convolutional layer (the filter is pruned);

[0078] The specific process is:

[0079] Select all J convolutional layers except the output layer in the target model;

[0080] The jth convolutional layer contains F I filter, filter F i The weight matrix is ​​represented as W i ;

[0081] j=1,2,…,J;i=1,2,…,I, i is the index of the filter;

[0082] Calculate the filter F i The weight matrix W i The L1 norm of is expressed as:

[0083]

[0084] Among them, L1 i Denotes the filter F i The weight matrix W i The L1 norm of || ||1 represents the L1 norm, W i [k] represents the weight matrix W i The kth element in ;

[0085] For all F contained in the jth convolutional layer I The L1 norms of the filters are sorted from large to small; the sorting result reflects the importance of each filter in the entire network;

[0086] The mask M of the filter corresponding to the L1 norm of the first N% of the sorting i′ The value is set to 1, and the mask M of the remaining filter i′The value is set to 0; i′=1, 2, …, I, i′ is the index of the mask (each filter corresponds to a mask);

[0087] Based on the sorting results, create corresponding masks M for these filters i , the M of the filter that needs to be retained i The value is set to 1, and the clipped value is set to 0.

[0088] Based on the filter F i The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ;

[0089] Set the weight matrix W to 0 i′ The corresponding filters are deleted to complete the filter pruning of the j-th convolutional layer;

[0090] This continues until all filters in all convolutional layers are pruned.

[0091] The number of filters in the jth convolutional layer is reduced, but the number of convolutional layers is not reduced.

[0092] The filter based on F i The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ;

[0093] The calculation is as follows:

[0094] W i′ =W i ⊙M i′

[0095] Among them, ⊙ represents element-wise multiplication.

[0096] In step 12, the number of channels of the weight matrix of the last filter after pruning in the last convolutional layer before the output layer in the target model is aligned with the number of channels of the output layer of the target model; the specific process is:

[0097] Perform 1×1 convolution on the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning to obtain a new feature tensor;

[0098] The number of channels of the weight matrix of the last filter of the last convolutional layer before the output layer in the target model after channel pruning is aligned with the number of channels of the output layer of the target model; the number of channels of the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning is the number of channels of the weight matrix of the last filter of the last convolutional layer before the output layer in the target model after channel pruning, and the number of channels of the new feature tensor is the number of channels of the output layer of the target model;

[0099] The new feature tensor X i ' is expressed as:

[0100]

[0101] Among them, X i Represents the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning;

[0102] C in Indicates the number of channels of the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning;

[0103] C represents the number of channels;

[0104] W 1×1 (c,:) represents the weight of the 1×1 convolution kernel on the cth channel.

[0105] To avoid the mismatch between the number of channels of the output layer and the number of channels of the output layer after channel cropping, the input tensor X is convolved using a convolution kernel W of size 1×1. i The number of channels from C in Change to C out (The number of channels of the output layer of the target model) to achieve channel number alignment.

[0106] In step 2, the loss function L(θ) of the target model after the filter pruning and channel number alignment are completed in step 1 is given;

[0107] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0108] According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation;

[0109] The specific process is:

[0110] Given step 1, the loss function L(θ) of the target model after filter pruning and channel number alignment is completed;

[0111] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0112]

[0113] Calculate the gradient T l The L2 norm L2(T l );

[0114] The gradient T l The L2 norm L2(T l ) are sorted from large to small, and the convolution operations in the last M% of the convolutional layers are replaced with lightweight convolution operations.

[0115] The calculated gradient T l The L2 norm L2(T l ); the expression is:

[0116]

[0117] In the formula, || ||2 represents the L2 norm;

[0118] i represents the feature map y l The width of j represents the feature map y l The height of k represents the feature map y l The number of channels.

[0119] The lightweight convolution operation includes channel-by-channel convolution and point convolution;

[0120] The specific processing process of lightweight convolution operation is:

[0121] Each channel of the input feature map passes through a dilation kernel with a dilation rate of r = 2, and the outputs of all convolution kernels are concatenated along the channel dimension to obtain the intermediate feature map;

[0122] Perform a 1×1 point convolution on the intermediate feature map and output the final feature map as the output of the lightweight convolution operation.

[0123] Realize changing the output channels to the specified number;

[0124] The conventional convolution operation process is changed to a two-step implementation of channel-by-channel convolution and point convolution. First, perform channel-by-channel convolution, use a dilation rate r = 2 for each channel of the input feature map X, and concatenate the outputs of all convolution kernels along the channel dimension to obtain the intermediate feature map X'. Then perform a 1×1 point convolution on the intermediate feature map X' to change the output channel to the specified number.

[0125] Compare the ordinary convolution and the lightweight convolution designed in this step. When the number of input channels is C in , the number of output channels is C out , when the convolution kernel size is n×n.

[0126] The parameters and receptive fields of ordinary convolution are:

[0127] C in ×C out ×n×n

[0128] n×n

[0129] The parameters and receptive fields of the lightweight convolution designed in this step are:

[0130] C in ×n×n+C in ×C out

[0131] [(n-1)r+1] 2

[0132] Among them, r is the expansion rate. By comparison, it can be seen that the number of parameters of lightweight convolution is smaller than that of ordinary convolution, the receptive field is larger than that of ordinary convolution, and its practicality is better than that of ordinary convolution.

[0133] According to the order of ranking, the importance of each convolutional layer to the loss function is evaluated, and the convolutional layers ranked in the bottom 50% are selected. The ordinary convolutions in these convolutional layers are replaced with lightweight convolutions.

[0134] The ordinary convolution operation is a one-step process, such as ordinary convolution kernels 3×3 and 1×1; the present invention is a two-step process, including channel-by-channel convolution and point convolution in sequence, and the present invention reduces the amount of calculation.

[0135] The value of N is: 30≤N≤60;

[0136] The value of M is: 40≤M≤50.

[0137] Specific implementation method 2: This implementation method is a recognition method based on a progressive compression model, which uses a progressive model compression method for complex network structures to obtain a progressive compression model, and then recognizes the image based on the constructed progressive compression model.

[0138] Specific implementation method three: This implementation method is a progressive model compression and identification method for complex network structures, including the following steps:

[0139] Step 1: construct a progressive compression model;

[0140] Step 11: prune the filter and align the number of channels of the target model used for image recognition;

[0141] Step 12: Given the loss function L(θ) of the target model after filter pruning and channel number alignment in step 11;

[0142] The image is input into the target model after filter pruning and channel number alignment in step 11. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0143] According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation;

[0144] Step 2: Recognize the image based on the constructed progressive compression model.

[0145] The complex network structure is a target model;

[0146] The target model is a deep learning model including a target detection model, a target tracking model, a semantic segmentation model, etc., such as a satellite video motion target detection model (DSFNet)

[0147] The image is a satellite video image;

[0148] The present invention proposes a progressive model compression method for complex network structures. Based on the idea of ​​de-redundancy, the model is firstly compressed based on channel pruning and channel alignment while retaining the original architecture. On this basis, lightweight convolution is used to replace ordinary convolution to further compress the model, ultimately achieving the purpose of sacrificing accuracy within a reasonable range, greatly reducing the number of complex model parameters and calculations, and reducing storage space occupancy.

[0149] In step 11, filter pruning and channel number alignment are performed on the target model for image recognition; the specific process is as follows:

[0150] Step 111: performing channel pruning on the filters in each convolution layer except the output layer in the target model for image recognition to obtain the filter after channel pruning in each convolution layer;

[0151] Step 112: Align the number of channels of the weight matrix of the last filter after channel pruning in the last convolutional layer before the output layer in the target model with the number of channels of the output layer of the target model.

[0152] In the step 111, channel pruning is performed on the filters in each convolution layer except the output layer in the target model for image recognition to obtain the channel pruned filters in each convolution layer (the pruned filters);

[0153] The specific process is:

[0154] Select all J convolutional layers except the output layer in the target model;

[0155] The jth convolutional layer contains F I filter, filter F i The weight matrix is ​​represented as W i ;

[0156] j=1,2,…,J;i=1,2,…,I, i is the index of the filter;

[0157] Calculate the filter F i The weight matrix W i The L1 norm of is expressed as:

[0158]

[0159] Among them, L1 i Denotes the filter F i The weight matrix W i The L1 norm of || ||1 represents the L1 norm, W i [k] represents the weight matrix W i The kth element in ;

[0160] For all F contained in the jth convolutional layer I The L1 norms of the filters are sorted from large to small; the sorting result reflects the importance of each filter in the entire network;

[0161] The mask M of the filter corresponding to the L1 norm of the first N% of the sorting i′ The value is set to 1, and the mask M of the remaining filter i′ The value is set to 0; i′=1, 2, …, I, i′ is the index of the mask (each filter corresponds to a mask);

[0162] Based on the sorting results, create corresponding masks M for these filters i , the M of the filter that needs to be retained i The value is set to 1, and the clipped value is set to 0.

[0163] Based on the filter Fi The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ;

[0164] Set the weight matrix W to 0 i′ The corresponding filters are deleted to complete the filter pruning of the j-th convolutional layer;

[0165] This continues until all filters in all convolutional layers are pruned.

[0166] The number of filters in the jth convolutional layer is reduced, but the number of convolutional layers is not reduced.

[0167] The filter based on F i The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ;

[0168] The calculation is as follows:

[0169] W i′ =W i ⊙M i′

[0170] Among them, ⊙ represents element-wise multiplication.

[0171] In step 112, the number of channels of the weight matrix of the last filter after pruning in the last convolutional layer before the output layer in the target model is aligned with the number of channels of the output layer of the target model; the specific process is:

[0172] Perform 1×1 convolution on the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning to obtain a new feature tensor;

[0173] The number of channels of the weight matrix of the last filter of the last convolutional layer before the output layer in the target model after channel pruning is aligned with the number of channels of the output layer of the target model; the number of channels of the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning is the number of channels of the weight matrix of the last filter of the last convolutional layer before the output layer in the target model after channel pruning, and the number of channels of the new feature tensor is the number of channels of the output layer of the target model;

[0174] The new feature tensor X i ' is expressed as:

[0175]

[0176] Among them, X i Represents the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning;

[0177] C in Indicates the number of channels of the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning;

[0178] C represents the number of channels;

[0179] W 1×1 (c,:) represents the weight of the 1×1 convolution kernel on the cth channel.

[0180] To avoid the mismatch between the number of channels of the output layer and the number of channels of the output layer after channel cropping, the input tensor X is convolved using a convolution kernel W of size 1×1. i The number of channels from C in Change to C out (The number of channels of the output layer of the target model) to achieve channel number alignment.

[0181] In the step 12, the loss function L(θ) of the target model after the filter pruning and the channel number alignment are completed in step 11 is given;

[0182] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0183] According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation;

[0184] The specific process is:

[0185] Given step 1, the loss function L(θ) of the target model after filter pruning and channel number alignment is completed;

[0186] The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l :

[0187]

[0188] Calculate the gradient T l The L2 norm L2(T l );

[0189] The gradient T lThe L2 norm L2(T l ) are sorted from large to small, and the convolution operations in the last M% of the convolutional layers are replaced with lightweight convolution operations.

[0190] The calculated gradient T l The L2 norm L2(T l ); the expression is:

[0191]

[0192] In the formula, || ||2 represents the L2 norm;

[0193] i represents the feature map y l The width of j represents the feature map y l The height of k represents the feature map y l The number of channels.

[0194] The lightweight convolution operation includes channel-by-channel convolution and point convolution;

[0195] The specific processing process of lightweight convolution operation is:

[0196] Each channel of the input feature map passes through a dilation kernel with a dilation rate of r = 2, and the outputs of all convolution kernels are concatenated along the channel dimension to obtain the intermediate feature map;

[0197] Perform a 1×1 point convolution on the intermediate feature map and output the final feature map as the output of the lightweight convolution operation.

[0198] Realize changing the output channels to the specified number;

[0199] The conventional convolution operation process is changed to a two-step implementation of channel-by-channel convolution and point convolution. First, perform channel-by-channel convolution, use a dilation rate r = 2 for each channel of the input feature map X, and concatenate the outputs of all convolution kernels along the channel dimension to obtain the intermediate feature map X'. Then perform a 1×1 point convolution on the intermediate feature map X' to change the output channel to the specified number.

[0200] Compare the ordinary convolution and the lightweight convolution designed in this step. When the number of input channels is C in , the number of output channels is C out , when the convolution kernel size is n×n.

[0201] The parameters and receptive fields of ordinary convolution are:

[0202] C in ×C out ×n×n

[0203] n×n

[0204] The parameters and receptive fields of the lightweight convolution designed in this step are:

[0205] C in ×n×n+C in ×C out

[0206] [(n-1)r+1] 2

[0207] Among them, r is the expansion rate. By comparison, it can be seen that the number of parameters of lightweight convolution is smaller than that of ordinary convolution, the receptive field is larger than that of ordinary convolution, and its practicality is better than that of ordinary convolution.

[0208] According to the order of ranking, the importance of each convolutional layer to the loss function is evaluated, and the convolutional layers ranked in the bottom 50% are selected. The ordinary convolutions in these convolutional layers are replaced with lightweight convolutions.

[0209] The ordinary convolution operation is a one-step process, such as ordinary convolution kernels 3×3 and 1×1; the present invention is a two-step process, including channel-by-channel convolution and point convolution in sequence, and the present invention reduces the amount of calculation.

[0210] The value of N is: 30≤N≤60;

[0211] The value of M is: 40≤M≤50.

[0212] The following examples are used to verify the beneficial effects of the present invention:

[0213] Embodiment 1:

[0214] The experiment selected the existing satellite video moving target detection model (DSFNet) as the target model, and tested the model's parameter quantity and computational complexity when the input image size was fixed at 3×512×512. It was also tested on the "Jilin-1" video satellite dataset. The target model parameters before progressive compression are shown in Table 1;

[0215] Table 2 shows the various indexes of the remote sensing image moving target detection model (MDSFNet) after preliminary compression after pruning and channel alignment;

[0216] Table 3 shows the various indicators of the lightweight model (LDSFNet) obtained by replacing ordinary convolution with the designed lightweight convolution.

[0217] By comparison, the progressive model compression method proposed in the present invention effectively reduces the number of model parameters, the amount of calculation, and the memory usage at the cost of a small sacrifice of detection accuracy.

[0218] Table 1

[0219]

[0220] Table 2

[0221]

[0222] Table 3

[0223]

[0224] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A progressive model compression method for complex network structures, characterized by: The following steps are involved: Step 1: Prune the filter and align the number of channels of the target model used for image recognition; Step 2: Given the loss function L(θ) of the target model after filter pruning and channel number alignment in step 1; The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l : According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation.

2. The progressive model compression method for complex network structures according to claim 1, characterized in that: In step 1, filter pruning and channel number alignment are performed on the target model for image recognition; the specific process is as follows: Step 11: Perform channel pruning on the filters in each convolutional layer except the output layer in the target model for image recognition to obtain the channel pruned filters in each convolutional layer; Step 12: Align the number of channels of the weight matrix of the last filter after channel pruning in the last convolutional layer before the output layer in the target model with the number of channels of the output layer of the target model.

3. The progressive model compression method for complex network structures according to claim 2, characterized in that: In the step 11, channel pruning is performed on the filters in each convolution layer except the output layer in the target model for image recognition to obtain the filter after channel pruning in each convolution layer; The specific process is: Select all J convolutional layers except the output layer in the target model; The jth convolutional layer contains F I filter, filter F i The weight matrix is ​​represented as W i ; j=1,2,…,J;i=1,2,…,I, i is the index of the filter; Calculate the filter F i The weight matrix W i The L1 norm of is expressed as: Among them, L1 i Denotes the filter F i The weight matrix W i The L1 norm of || ||1 represents the L1 norm, W i [k] represents the weight matrix W i The kth element in ; For all F contained in the jth convolutional layer I The L1 norms of the filters are sorted from large to small; The mask M of the filter corresponding to the L1 norm of the first N% of the sorting i′ The value is set to 1, and the mask M of the remaining filter i′ The value is set to 0; i′=1, 2, …, I, i′ is the index of the mask; Based on the filter F i The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ; Set the weight matrix W to 0 i′ The corresponding filters are deleted to complete the filter pruning of the j-th convolutional layer; This continues until all filters in all convolutional layers are pruned.

4. The progressive model compression method for complex network structures according to claim 3 is characterized in that: The filter based on F i The weight matrix W i and mask M i′ Value, get the new filter F of the j-th convolutional layer i The weight matrix W i′ ; The calculation is as follows: IN i′ =In i ⊙M i′ Among them, ⊙ represents element-wise multiplication.

5. The progressive model compression method for complex network structures according to claim 4, characterized in that: In step 12, the number of channels of the weight matrix of the last filter after pruning in the last convolutional layer before the output layer in the target model is aligned with the number of channels of the output layer of the target model; the specific process is: Perform 1×1 convolution on the feature tensor output by the last convolution layer before the output layer in the target model after channel pruning to obtain a new feature tensor; The new feature tensor X i ' is expressed as: Among them, X i Represents the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning; C in Indicates the number of channels of the feature tensor output by the last convolutional layer before the output layer in the target model after channel pruning; C represents the number of channels; W 1×1 (c,:) represents the weight of the 1×1 convolution kernel on the cth channel.

6. The progressive model compression method for complex network structures according to claim 5, characterized in that: In the step 2, the loss function L(θ) of the target model after the filter pruning and the channel number alignment are completed in step 1 is given; The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l : According to the gradient, the convolution operation in the convolution layer is replaced by a lightweight convolution operation; The specific process is: Given step 1, the loss function L(θ) of the target model after filter pruning and channel number alignment is completed; The image is input into the target model after filter pruning and channel number alignment in step 1. Each convolution layer l of the target model outputs y l , use the back-propagation algorithm to calculate the loss function relative to y l The gradient T l : Calculate the gradient T l The L2 norm L2(T l ); The gradient T l The L2 norm L2(T l ) are sorted from large to small, and the convolution operations in the last M% of the convolutional layers are replaced with lightweight convolution operations.

7. The progressive model compression method for complex network structures according to claim 6, characterized in that: The calculated gradient T l The L2 norm L2(T l ); the expression is: In the formula, || ||2 represents the L2 norm; i represents the feature map y l The width of j represents the feature map y l The height of k represents the feature map y l The number of channels.

8. The progressive model compression method for complex network structures according to claim 7, characterized in that: The lightweight convolution operation includes channel-by-channel convolution and point convolution; The specific processing process of lightweight convolution operation is: Each channel of the input feature map passes through a dilation kernel with a dilation rate of r = 2, and the outputs of all convolution kernels are concatenated along the channel dimension to obtain the intermediate feature map; Perform a 1×1 point convolution on the intermediate feature map and output the final feature map as the output of the lightweight convolution operation.

9. The progressive model compression method for complex network structures according to claim 8, characterized in that: The value of N is: 30≤N≤60.

10. The progressive model compression method for complex network structures according to claim 9, characterized in that: The value of M is: 40≤M≤50.

Citation Information

Patent Citations

  • Pruning-based lightweight network structure image classification method

    CN117197524A

  • Robust pruned neural networks via adversarial training

    US20190244103A1