A static-dynamic joint-based convolutional neural network pruning method and device

Abstract

The application discloses a kind of convolutional neural network pruning method and device based on static and dynamic combination, and the method comprises: static pruning is carried out to convolutional neural network parameter, and the parameter with low weight is screened out to be pruned, and the convolutional neural network parameter is optimized;During the training process of convolutional neural network, according to the specific characteristics of input sample, the importance of the remaining channel is dynamically evaluated, and the convolutional neural network parameter is dynamically pruned and optimized again;Static pruning is combined with dynamic pruning, until the convolutional neural network pruning is completed.The application combines static pruning and dynamic pruning, makes full use of the complementary advantages of the two pruning strategies, realizes the reduction of model size and the improvement of computing efficiency, while trying to keep the performance of the model from being damaged, not only more flexible than static pruning, but also reduces the additional computing cost and time overhead compared with dynamic pruning.

Description

Technical Field

[0001] This invention relates to the field of compression and pruning technology for convolutional neural networks, and more particularly to a method and apparatus for pruning convolutional neural networks based on static and dynamic combination. Background Technology

[0002] Convolutional neural networks (CNNs) have achieved remarkable performance in many computer vision tasks, such as image classification, anomaly detection, and object detection. However, CNNs require significant memory overhead and high-performance computing units, making their deployment and operation on resource-constrained embedded devices difficult. This presents a considerable obstacle to their widespread practical application.

[0003] Existing network model compression methods generally reduce the size and computational complexity of convolutional neural networks (CNNs) through techniques such as pruning, quantization, low-rank decomposition, and knowledge distillation, thereby reducing the resource requirements of the network model. Among these, pruning, by identifying and removing unimportant weights from the neural network to reduce its size, is currently a popular method in the field of network model compression.

[0004] Pruning is divided into static pruning and dynamic pruning. Static pruning trims the parameters of the network model, designing a fixed sparse structure for all samples. This method is widely used in most weight pruning, channel pruning, and filter pruning, and has shown significant effects in network model compression and acceleration. However, static pruning is usually irreversible; once some parts are pruned, these parts cannot be recovered in subsequent processes, potentially causing the network model to permanently lose some important parameters, adversely affecting the accuracy and generalization ability of the network model. Simultaneously, static pruning ignores the dynamic relationship between channel saliency and input samples, failing to consider the different importance of channels in different input samples. Compared to static pruning, dynamic pruning takes a more flexible approach. Dynamic pruning does not permanently remove any parameters from the network, but dynamically determines which part of the convolutional neural network should participate in the computation based on the features of each input image, thus effectively reducing unnecessary computation and redundancy. However, dynamic pruning requires repeating certain additional operations for each input sample, such as index copying or weight copying, incurring additional computational costs and time overhead. Therefore, although dynamic pruning can theoretically significantly improve inference efficiency, in practice, the actual inference speed may be much slower than the theoretical speed due to these additional operations.

[0005] In summary, current pruning methods have the following problems: First, static pruning generates a fixed pruning structure, which cannot prune flexibly, affecting the accuracy and generalization of the network model and ignoring the dynamic relationship between channel saliency and input samples. Second, dynamic pruning requires repeated operations such as index copying or weight copying for each input sample, reducing the actual inference speed.

[0006] For example, application number 202410904785.9 discloses a model pruning method and a computer-readable storage medium. This application prunes the filter channels of the first preset number in the sorting based on a preset global pruning rate; it obtains the model to be pruned after pruning each set of convolutional layers, thus obtaining the target model. Another example is application number 202410601678.9, which discloses a fast pruning method, system, device, and storage medium for convolutional neural networks. This application uses gradient descent to generate the pruning rate, enabling the search for the optimal pruning rate in fewer rounds, and requiring only one fine-tuning during the pruning process, greatly improving pruning efficiency.

[0007] While the above approach improves the pruning effect on the model, it also has drawbacks: it does not combine static and dynamic pruning, fails to realize the complementary advantages of the two pruning strategies, and is prone to damage to model performance, increasing additional computational costs and time overhead. Summary of the Invention

[0008] To address the aforementioned problems, the present invention aims to provide a convolutional neural network pruning method and apparatus based on a combination of static and dynamic pruning. By leveraging the complementary advantages of static and dynamic pruning strategies, the method achieves model size reduction and computational efficiency improvement while minimizing any compromise in model performance.

[0009] This invention provides a method and apparatus for pruning convolutional neural networks based on static and dynamic combination.

[0010] First aspect: A convolutional neural network pruning method based on static and dynamic joint methods, including the following steps:

[0011] S1. Perform static pruning on the parameters of the convolutional neural network, select parameters with low weights for removal, and optimize the parameters of the convolutional neural network.

[0012] S2. During the training of the convolutional neural network, the importance of the remaining channels is dynamically evaluated based on the specific characteristics of the input samples, and the parameters of the convolutional neural network are dynamically pruned and optimized.

[0013] S3. Repeat steps S1 to S2 until the convolutional neural network pruning is completed.

[0014] Furthermore, in the static pruning stage of S1, parameters with low importance are selected based on a trainable threshold T.

[0015] Furthermore, in the S2 stage, the channel attention mechanism is optimized to dynamically evaluate the importance of the remaining channels.

[0016] Furthermore, the step of filtering out parameters of low importance based on a trainable threshold T includes the following steps:

[0017] S11. Use a trainable threshold T to evaluate the importance of the convolution weight parameters, and obtain the corresponding mask M for each convolution weight parameter through the threshold function. i The formula is:

[0018]

[0019] Where M∈R K*K Let T be the set of masks for each convolutional kernel, σ be the threshold, and M be the sigmoid function. ij ∈M i , where i represents the number of convolutional layers and j represents the number of convolutional kernels.

[0020] S12. If the convolution weight parameter exceeds the threshold, the corresponding mask is set to 1, and the weight is retained and participates in subsequent training; otherwise, the convolution weight is pruned.

[0021] S13, change the original weight parameters The dot product of the obtained mask parameters is given by the following formula:

[0022]

[0023] in, M represents the weight parameters of the i-th convolutional layer. i is the kernel mask for the i-th convolutional layer.

[0024] Furthermore, the optimization of the channel attention mechanism, dynamically evaluating the importance of the remaining channels, includes the following steps:

[0025] S21. For input features X∈R C*H*W Global features F are extracted through global average pooling and global max pooling. a With texture feature F m The formula is expressed as:

[0026]

[0027] Where H is the height of the input feature, W is the width of the input feature, and F... a ∈R C*1*1 For global information of the input features, Fm ∈R C*1*1 The texture information is input from the features.

[0028] S22, against F a and F m One-dimensional convolution is used to achieve local interaction between channels, and the attention parameters are calculated using the sigmoid activation function. The concatenation results in the attention weights F of the i-th convolutional layer. i :

[0029] F i =σ(conv(F) a ))+σ(conv(F m ))

[0030] Where conv represents one-dimensional convolution and σ represents the Sigmoid activation function;

[0031] Furthermore, the dynamic pruning and re-optimization of the convolutional neural network parameters in S2 includes: adjusting the attention weights F generated by the dynamic pruning... i Weight parameters obtained from static pruning Multiplying point by point yields the weight parameters.

[0032]

[0033] Where i represents the number of convolutional layers, These are the convolution weight parameters after static pruning.

[0034] Furthermore, in the static-dynamic pruning process, the trainable threshold T is optimized using a loss function, as shown in the formula:

[0035]

[0036] Among them, L c λ represents the cross-entropy loss, λ represents the coefficient factor, N represents the number of convolutional layers, and C represents the number of filters in the convolutional layer.

[0037] The second aspect: an apparatus for a convolutional neural network pruning method based on static and dynamic joint methods, comprising:

[0038] The static pruning module performs static pruning on the parameters of the convolutional neural network, selecting and removing parameters with low weights to optimize the parameters of the convolutional neural network.

[0039] The dynamic pruning module dynamically evaluates the importance of the remaining channels based on the specific features of the input samples during the training process of the convolutional neural network, and dynamically prunes and optimizes the parameters of the convolutional neural network.

[0040] The static pruning module, together with the dynamic pruning module, jointly completes the pruning of the convolutional neural network.

[0041] Third aspect: An electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the method provided in the first aspect.

[0042] Fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0043] The beneficial effects of this invention are:

[0044] 1. This invention removes parameters and structures with low contribution by applying static pruning. In the static pruning stage, a sparse method based on trainable thresholds is used to identify convolution weights with low contribution. By setting an appropriate threshold, parameters with low importance can be effectively screened out and pruned, thus completing the static pruning of the model without significantly impairing model performance.

[0045] 2. This invention optimizes the traditional channel attention mechanism by applying dynamic pruning during the dynamic pruning stage, enabling it to generate attention weights for the input feature map more accurately. This allows for a more accurate assessment of the importance of the remaining channels after static pruning, and the pruning process of the model is dynamically adjusted according to the specific features of each input sample.

[0046] 3. This invention combines static pruning and dynamic pruning, making full use of the complementary advantages of the two pruning strategies to achieve model size reduction and computational efficiency improvement, while keeping the model performance as unimpaired as possible. It is not only more flexible than static pruning, but also reduces additional computational costs and time overhead compared to dynamic pruning.

[0047] 4. This invention improves the channel attention mechanism in the dynamic pruning stage, generating more accurate attention weights to guide further pruning work, and dynamically adjusts the pruning structure according to different input samples; it significantly reduces the computational complexity and number of parameters of the network model, while maintaining the accuracy of the convolutional neural network. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the network pruning method of the present invention;

[0049] Figure 2 This is a schematic diagram of the network pruning device of the present invention;

[0050] Figure 3 This is a flowchart illustrating the principle of the network pruning method of the present invention;

[0051] Figure 4 This is a schematic diagram of the electronic device structure of the present invention. Detailed Implementation

[0052] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0053] Current methods for pruning convolutional neural networks use static pruning to generate fixed pruning structures, which cannot prune flexibly and affect the accuracy and generalization of the network model. Dynamic pruning, on the other hand, requires repeated operations such as index copying or weight copying for each input sample, which reduces the actual inference speed.

[0054] To address the above problems, this invention provides a convolutional neural network pruning method based on static and dynamic joint methods. Figure 1 This is a flowchart illustrating a convolutional neural network pruning method based on static and dynamic joint operation provided in an embodiment of the present invention. The method includes:

[0055] S1. Perform static pruning on the parameters of the convolutional neural network, select parameters with low weights for removal, and optimize the parameters of the convolutional neural network.

[0056] like Figure 2 As shown, in the static pruning process of convolutional neural network parameters, during the static pruning stage, parameters with low importance are selected based on a trainable threshold T, and the weights of each convolutional layer are passed through a smooth threshold function to obtain a corresponding mask M. i The formula is defined as:

[0057]

[0058] Where M∈R K*K Let T be the set of masks for each convolutional kernel, σ be the threshold, and M be the sigmoid function. ij ∈M i , where i represents the number of convolutional layers and j represents the number of convolutional kernels, i.e. the number of output channels.

[0059] The mask M is generated based on the comparison between the weight parameters of the convolution kernel and the threshold T. If the convolution weight parameters exceed the threshold T, the corresponding mask is set to 1, and the weight is retained and participates in subsequent training. Otherwise, the convolution weight is pruned.

[0060] Change the original weight parameters With the obtained mask M i This process is achieved by performing a dot product:

[0061]

[0062] in, M represents the weight parameters of the i-th convolutional layer. i is the kernel mask for the i-th convolutional layer.

[0063] S2. During the training of the convolutional neural network, the importance of the remaining channels is dynamically evaluated based on the specific characteristics of the input samples, and the parameters of the convolutional neural network are dynamically pruned and optimized.

[0064] Dynamic pruning process such as Figure 3 As shown. First, for the input feature X∈R C*H*W It extracts important information from the input features from different perspectives, and extracts the global features F of the image through global average pooling and global max pooling operations, respectively. a With texture feature F m The formula is expressed as:

[0065]

[0066] Where H represents the height of the input feature, W represents the width of the input feature, and F... a ∈R C*1*1 F reflects global information about the input features. m ∈R C*1*1 This highlights the texture information in the input features.

[0067] Traditional channel attention typically uses two fully connected layers to perform dimensionality reduction followed by dimensionality increase, generating attention weights for each channel. However, this design adds extra parameters, increasing the burden on the network model. Furthermore, the dimensionality reduction operation in the first fully connected layer may lead to the loss of important information, affecting the accuracy of the network model.

[0068] This invention uses one-dimensional convolution to replace two fully connected layers to process the channel dimension of the input feature map. Through one-dimensional convolution, the attention weight of each channel is affected by the channel itself and the adjacent channels, and the network model can capture the dynamic relationship between channels more effectively.

[0069] Therefore, by adjusting F a and F m One-dimensional convolution is used to achieve local interaction between channels, and the sigmoid activation function is used to calculate attention parameters. The concatenation results in the attention weights F of the i-th convolutional layer. i This indicates the importance of each channel, expressed by the formula:

[0070]

[0071] Where conv represents one-dimensional convolution and σ represents the Sigmoid activation function.

[0072] S3. Repeat steps S1 to S2 until the convolutional neural network pruning is completed.

[0073] Finally, the weight parameters obtained from static pruning are... Attention weights F generated by dynamic pruning i Multiplying point by point yields the weight parameters.

[0074]

[0075] By combining dynamic and static pruning, the network model parameters are further compressed while ensuring that the performance of the network model is not significantly affected, until the pruning of the convolutional neural network is completed.

[0076] In the static-dynamic pruning process, the trainable threshold T is optimized using a loss function, the formula of which is:

[0077]

[0078] Among them, L c λ represents the cross-entropy loss, λ represents the coefficient factor, N represents the number of convolutional layers, and C represents the number of filters in the convolutional layer.

[0079] During training, the threshold T is updated during the backpropagation of the loss function, and it self-adjusts according to the network model's performance on the training data in order to find the optimal threshold setting. Ultimately, the threshold T in each filter will reach an optimal state, which not only reduces the computational load of the network model, but also maintains the network model's performance.

[0080] like Figure 2 As shown, the present invention also discloses a convolutional neural network pruning device for the above method, the device comprising:

[0081] The static pruning module performs static pruning on the parameters of the convolutional neural network, selecting and removing parameters with low weights to optimize the parameters of the convolutional neural network.

[0082] The dynamic pruning module dynamically evaluates the importance of the remaining channels based on the specific features of the input samples during the training process of the convolutional neural network, and dynamically prunes and optimizes the parameters of the convolutional neural network.

[0083] The static pruning module, together with the dynamic pruning module, jointly completes the pruning of the convolutional neural network.

[0084] In this invention, a static pruning module is first applied to remove parameters and structures that contribute less, and then a dynamic pruning module is used to further sparsify and adjust the model during training.

[0085] In the static pruning stage, a sparse method based on trainable thresholds is used to identify convolutional weights with lower contributions. By setting an appropriate threshold, parameters with lower importance can be effectively filtered out and pruned, completing static pruning of the model without significantly impairing model performance. In the dynamic pruning stage, the traditional channel attention mechanism is optimized. Through one-dimensional convolution, the attention weight of each channel is influenced by both the channel itself and its neighbors. The model can more effectively capture the dynamic relationships between channels and more accurately generate attention weights for the input feature map, thereby more accurately evaluating the importance of the remaining channels after static pruning. The pruning process of the model is dynamically adjusted according to the specific features of each input sample. This invention fully utilizes the complementary advantages of the two pruning strategies to achieve model size reduction and computational efficiency improvement, while minimizing the impact on model performance.

[0086] The present invention also provides an electronic device, Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory, for example, to execute the following method:

[0087] S1. Perform static pruning on the parameters of the convolutional neural network, select parameters with low weights for removal, and optimize the parameters of the convolutional neural network.

[0088] S2. During the training of the convolutional neural network, the importance of the remaining channels is dynamically evaluated based on the specific characteristics of the input samples, and the parameters of the convolutional neural network are dynamically pruned and optimized.

[0089] S3. Repeat steps S1 to S2 until the convolutional neural network pruning is completed.

[0090] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments, including, for example:

[0092] S1. Perform static pruning on the parameters of the convolutional neural network, select parameters with low weights for removal, and optimize the parameters of the convolutional neural network.

[0093] S2. During the training of the convolutional neural network, the importance of the remaining channels is dynamically evaluated based on the specific characteristics of the input samples, and the parameters of the convolutional neural network are dynamically pruned and optimized.

[0094] S3. Repeat steps S1 to S2 until the convolutional neural network pruning is completed.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A convolutional neural network pruning method based on static and dynamic joint operation, characterized in that, include: S1. Perform static pruning on the parameters of the convolutional neural network, select parameters with low weights for removal, and optimize the parameters of the convolutional neural network. S2. During the training of the convolutional neural network, the importance of the remaining channels is dynamically evaluated based on the specific characteristics of the input samples, and the parameters of the convolutional neural network are dynamically pruned and optimized. S3. Repeat steps S1 to S2 until the convolutional neural network pruning is completed; In S2, during the dynamic pruning phase, the channel attention mechanism is optimized to dynamically evaluate the importance of the remaining channels. The optimization of the channel attention mechanism, which dynamically evaluates the importance of the remaining channels, includes the following steps: S21. For input features X∈R C*H*W Global features F of the image are extracted through global average pooling and global max pooling. a With texture feature F m The formula is expressed as: Where H is the height of the input feature, W is the width of the input feature, and F... a ∈R C*1*1 For global information of the input features, F m ∈R C*1*1 The texture information is input from the features. S22, against F a and F m One-dimensional convolution is used to achieve local interaction between channels, and the attention parameters are calculated using the sigmoid activation function. The concatenation results in the attention weights F of the i-th convolutional layer. i : F i =σ(conv(F a ))+σ(conv(F m )) Where conv represents one-dimensional convolution and σ represents the Sigmoid activation function; The S2 section describes the dynamic pruning and re-optimization of the convolutional neural network parameters, including... The attention weight F generated by dynamic pruning i The weight parameter W obtained by static pruning i 'Multiply point by point to obtain the weight parameters' Where i represents the number of convolutional layers, These are the convolution weight parameters after static pruning.

2. The method according to claim 1, characterized in that, In the static pruning stage of S1, parameters with low importance are selected based on a trainable threshold T.

3. The method according to claim 2, characterized in that, The step of filtering out parameters of low importance based on a trainable threshold T includes the following steps: S11. Use a trainable threshold T to evaluate the importance of the convolution weight parameters, and obtain the corresponding mask M for each convolution weight parameter through the threshold function. i The formula is: Where M∈R K*K Let T be the set of masks for each convolutional kernel, σ be the threshold, and M be the sigmoid function. ij ∈M i , where i represents the number of convolutional layers and j represents the number of convolutional kernels; S12. If the convolution weight parameter exceeds the threshold, the corresponding mask is set to 1, and the weight is retained and participates in subsequent training; otherwise, the convolution weight is pruned. S13, change the original weight parameters The dot product of the obtained mask parameters is given by the following formula: in, M represents the weight parameters of the i-th convolutional layer. i is the kernel mask for the i-th convolutional layer.

4. The method according to claim 1, characterized in that, In the static-dynamic pruning process, the trainable threshold T is optimized using a loss function, as shown in the formula: Among them, L c λ represents the cross-entropy loss, λ represents the coefficient factor, N represents the number of convolutional layers, and C represents the number of filters in the convolutional layer.

5. A convolutional neural network pruning device based on the method of any one of claims 1 to 4, characterized in that, The device includes: The static pruning module performs static pruning on the parameters of the convolutional neural network, selecting and removing parameters with low weights to optimize the parameters of the convolutional neural network. The dynamic pruning module dynamically evaluates the importance of the remaining channels based on the specific features of the input samples during the training process of the convolutional neural network, and dynamically prunes and optimizes the parameters of the convolutional neural network. The static pruning module, together with the dynamic pruning module, jointly completes the pruning of the convolutional neural network.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as claimed in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as claimed in any one of claims 1 to 4.

Images