Method and device for realizing convolutional neural network pruning based on wavelet transform, processor and computer readable storage medium thereof

Abstract

The present application relates to a kind of based on wavelet transform implementation convolutional neural network pruning processing method, comprising the following steps: obtaining image classification dataset and doing pretreatment;Obtain the pre-training model of convolutional neural network disclosed on the network;The feature map output by spatial domain convolutional neural network is converted to frequency domain;Characteristic map importance score is calculated;Sort and carry out pruning operation.The present application also relates to a kind of for implementing based on wavelet transform convolutional neural network pruning processing device, processor and its computer readable storage medium.The wavelet transform implementation convolutional neural network pruning processing method, device, processor and its computer readable storage medium of the present application have not been improved or changed to the pooling layer operation in network, the integrity and simplicity of network structure are maintained to the greatest extent.The energy weighting coefficient of high and low frequency components after wavelet transform of the present application can more effectively retain valuable feature information and improve image classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to the field of model pruning system, and more particularly to a method and device for realizing pruning processing of convolutional neural network based on wavelet transform, a processor and a computer readable storage medium thereof. BACKGROUND

[0002] The development of convolutional neural network has a great impact on many computer vision tasks. However, as the difficulty of computer vision tasks increases, the size of the corresponding convolutional neural network also gradually increases, and the training and deployment of these networks require more and more computing power and memory consumption. Therefore, these advanced convolutional neural networks cannot be directly deployed in embedded systems and mobile platforms, which seriously affects the application and promotion of convolutional neural networks. In response to this, many solutions have been proposed, including model pruning methods.

[0003] Model pruning is a method of realizing compact and lightweight network by removing redundant network structures and parameters from a pre-trained model. Among them, filter pruning can prune redundant convolution kernels in the original neural network, thereby obtaining a neural network with small parameter size. At the same time, filter pruning can effectively reduce the number of model parameters and computational cost while maintaining a certain accuracy, thereby realizing the compression and acceleration of convolutional neural network. In addition, existing deep learning libraries can support filter pruning without the need for special hardware and software. Therefore, filter pruning has become the most popular structured pruning. And many filter pruning methods have been proposed and achieved satisfactory experimental results.

[0004] Most of the existing pruning methods are performed in the spatial domain, by setting different parameter importance evaluation criteria to determine which parameters need to be pruned and which need to be retained, in order to obtain a more compact and efficient neural network model. However, the spatial frequency of an image is actually complex, with a lot of redundant information. In this case, if the pruning criteria are directly determined by the original image, the result may not be able to extract effective information or may contain redundant information. Since most of the information in the image is manifested as low-frequency information with high energy in the frequency domain, and a small part of the detail information in the image is manifested as high-frequency information with low energy in the frequency domain, if the filter in the neural network is considered as an image, the energy distribution of the image after frequency domain transformation will also show a large difference. Therefore, if the internal structure of the model (i.e. the feature map) is transformed into the frequency domain, the change of the internal entropy of the model can be analyzed in the frequency domain, thereby better reflecting the mapping of the potential relationship between the parameters or features inside the model.

[0005] Therefore, how to better explore the relationship between the parameters or features inside the model under the premise of ensuring the overall performance of the model becomes the key to more effective model pruning. SUMMARY

[0006] The present application aims to overcome the above-mentioned shortcomings of the prior art, and provides a method, device, processor and computer readable storage medium for implementing convolutional neural network pruning based on wavelet transform, which meets the requirements of low error, high accuracy and wide application range.

[0007] To achieve the above-mentioned purpose, the method, device, processor and computer readable storage medium for implementing convolutional neural network pruning based on wavelet transform of the present application are as follows:

[0008] The method for implementing convolutional neural network pruning based on wavelet transform mainly comprises the following steps:

[0009] (1) Obtain an image classification data set and perform preprocessing;

[0010] (2) Obtain a publicly available convolutional neural network pre-training model;

[0011] (3) Convert the feature map output by the convolutional neural network in the spatial domain to the frequency domain;

[0012] (4) Calculate the feature map importance score;

[0013] (5) Sort the feature map importance score, prune the filters corresponding to the sorted feature map according to the set pruning threshold, and obtain the pruned model.

[0014] Preferably, step (3) specifically comprises the following steps:

[0015] Wavelet decomposition is performed using CWT;

[0016] The output feature map in the convolutional neural network is subjected to wavelet transform and four frequency domain components are obtained, namely low frequency information component LL, horizontal high frequency information component HL, vertical high frequency information component LH and diagonal high frequency information component HH.

[0017] Preferably, in step (1), wavelet decomposition is performed using CWT, specifically as follows:

[0018] Wavelet decomposition is performed using CWT according to the following formula:

[0019]

[0020] Wherein, a is called the scale factor, which controls the stretching of the wavelet function corresponding to the frequency, b is the translation factor, which controls the position of the wavelet function on the time axis, f(t) is the input signal, Haar wavelet is the wavelet transform base function, is the complex function of the Haar wavelet function.

[0021] Preferably, step (4) specifically includes the following steps:

[0022] (4.1) Calculate the similarity of the corresponding high and low frequency components by decomposing different feature maps of a certain convolutional layer based on WT;

[0023] (4.2) Calculate the weights of high-frequency and low-frequency frequencies based on the energy of the feature map;

[0024] (4.3) By comparing the energy of the low-frequency components and the energy of the high-frequency components with the total energy of the feature map, the energy weighting coefficients of the low-frequency and high-frequency components are derived respectively;

[0025] (4.4) Calculate the weighted similarity between different feature maps;

[0026] (4.5) The importance score of each feature map is calculated by summing the similarities between the feature maps.

[0027] Preferably, the calculation of the similarity between the corresponding high and low frequency components in step (4.1) is specifically as follows:

[0028] The similarity between the corresponding high and low frequency components is calculated using the following formula:

[0029]

[0030]

[0031] in, Indicates calculation and The cosine similarity value of the low-frequency components between them. Indicates the calculation of high-frequency components and The cosine similarity value between them The wavelet transform yields the first... The low-frequency component of the j-th feature map in layer 1. It represents the sum of three high-frequency components. and They represent the first... The sum of the low-frequency component and the three high-frequency components obtained by wavelet transforming the k-th feature map of the convolutional layer.

[0032] Preferably, the calculation of the weighted similarity between different feature maps in step (4.4) is specifically as follows:

[0033] The weighted similarity between different feature maps is calculated using the following formula:

[0034]

[0035] in, represent and Perform energy-weighted calculations, ew LF and ew HF These represent the energy weighting coefficients for the low-frequency and high-frequency components, respectively. and These represent the j-th and k-th (j≠k) feature maps of the l-th convolutional layer, respectively.

[0036] Preferably, the calculation of the importance score of the feature map in step (4.5) is specifically as follows:

[0037] The importance score of the feature map is calculated using the following formula:

[0038]

[0039] in, Indicates from the first The importance score is obtained by comparing the similarity between the j-th feature map of the convolutional layer and all other feature maps.

[0040] The apparatus for implementing wavelet transform-based convolutional neural network pruning is characterized in that the apparatus comprises:

[0041] A processor is configured to execute computer-executable instructions;

[0042] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the various steps of the above-described method for pruning convolutional neural networks based on wavelet transform.

[0043] The processor used to implement wavelet transform-based convolutional neural network pruning is characterized in that the processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the various steps of the above-described method for wavelet transform-based convolutional neural network pruning.

[0044] The main feature of this computer-readable storage medium is that it stores a computer program thereon, which can be executed by a processor to implement the various steps of the above-described method for implementing convolutional neural network pruning based on wavelet transform.

[0045] The method, device, processor and computer readable storage medium of the application based on wavelet transform to realize convolutional neural network pruning process do not improve or change the pooling layer operation in the network, and the integrity and simplicity of the network structure are maintained to the maximum extent. The energy weighting coefficient of the high and low frequency components after wavelet transform can more effectively retain valuable feature information and improve image classification accuracy. The application calculates the cosine similarity between the corresponding high and low frequency components, and fuses and calculates the energy weighting coefficient of the high and low frequency components, which further prunes the feature map containing redundant information. Since the architecture of the convolutional neural network is not modified, the pruned convolutional neural network architecture belongs to structured pruning and can be directly deployed on edge devices. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The overall framework diagram of the method for realizing convolutional neural network pruning processing based on wavelet transform of the application.

[0047] Figure 2 The feature map processed by wavelet transform of the method for realizing convolutional neural network pruning processing based on wavelet transform of the application.

[0048] Figure 3 The overall flowchart of the method for realizing convolutional neural network pruning processing based on wavelet transform of the application.

[0049] Figure 4 The comparative result schematic diagram of different pruning methods of the application in pruning DenseNet40 on CIFAR-10.

[0050] Figure 5 The comparative result schematic diagram of different pruning methods of the application in pruning ResNet-56 on CIFAR-10.

[0051] Figure 6 The acceleration ratio schematic diagram of different convolutional neural networks with a batch size of 128 on CIFAR-10 on CPU (Intel(R) Xeon(R) E5-1650 v3 @ 3.50GHz) and GPU (GTX-1080TI) of the application.

[0052] Figure 7 The pruning result schematic diagram of the application and the existing pruning method on VGGNet-16 architecture on CIFAR-10 dataset.

[0053] Figure 8 The pruning result schematic diagram of the application and the existing pruning method on ResNet-110 architecture on CIFAR-10 dataset based on wavelet transform to realize convolutional neural network pruning processing of the application.

[0054] Figure 9 The experimental results of the effectiveness of the application based on the wavelet transform to realize the pruning processing of the convolutional neural network method of the application on a large-scale complex image dataset ImageNet represent the intention. DETAILED DESCRIPTION

[0055] In order to more clearly describe the technical content of the application, further description will be made below in combination with specific embodiments.

[0056] The method for realizing the pruning processing of the convolutional neural network based on the wavelet transform of the application, wherein the method comprises the following steps:

[0057] (1) obtaining an image classification dataset and performing preprocessing;

[0058] (2) obtaining a publicly disclosed convolutional neural network pre-training model;

[0059] (3) converting the feature map output by the convolutional neural network in the spatial domain to the frequency domain;

[0060] (4) calculating the feature map importance score;

[0061] (5) sorting according to the feature map importance score, pruning the filters corresponding to the sorted feature maps according to the set pruning threshold, and obtaining the pruned model.

[0062] As a preferred embodiment of the application, the step (3) specifically comprises the following steps:

[0063] wavelet decomposition is performed by using CWT;

[0064] The output feature map in the convolutional neural network is subjected to wavelet transform and four frequency domain components are obtained, which are low frequency information component LL, horizontal high frequency information component HL, vertical high frequency information component LH and diagonal high frequency information component HH.

[0065] As a preferred embodiment of the application, the wavelet decomposition in the step (1) is performed by using CWT, and specifically:

[0066] The wavelet decomposition is performed by using CWT according to the following formula:

[0067]

[0068] wherein a is called a scale factor, controls the stretching of the wavelet function corresponding to the frequency, b is a translation factor, controls the position of the wavelet function on the time axis, f(t) is an input signal, Haar wavelet is a wavelet transform base function, is a complex function of the Haar wavelet function.

[0069] In a preferred embodiment of the present invention, step (4) specifically includes the following steps:

[0070] (4.1) Calculate the similarity of the corresponding high and low frequency components by decomposing different feature maps of a certain convolutional layer based on WT;

[0071] (4.2) Calculate the weights of high-frequency and low-frequency frequencies based on the energy of the feature map;

[0072] (4.3) By comparing the energy of the low-frequency components and the energy of the high-frequency components with the total energy of the feature map, the energy weighting coefficients of the low-frequency and high-frequency components are derived respectively;

[0073] (4.4) Calculate the weighted similarity between different feature maps;

[0074] (4.5) The importance score of each feature map is calculated by summing the similarities between the feature maps.

[0075] In a preferred embodiment of the present invention, the calculation of the similarity between the corresponding high and low frequency components in step (4.1) specifically involves:

[0076] The similarity between the corresponding high and low frequency components is calculated using the following formula:

[0077]

[0078]

[0079] in, Indicates calculation and The cosine similarity value of the low-frequency components between them. Indicates the calculation of high-frequency components and The cosine similarity value between them The wavelet transform yields the first... The low-frequency component of the j-th feature map in layer 1. It represents the sum of three high-frequency components. and They represent the first... The sum of the low-frequency component and the three high-frequency components obtained by wavelet transforming the k-th feature map of the convolutional layer.

[0080] In a preferred embodiment of the present invention, the calculation of the weighted similarity between different feature maps in step (4.4) specifically involves:

[0081] The weighted similarity between different feature maps is calculated using the following formula:

[0082]

[0083] wherein, represent and performing energy weighting calculation, ew LF and ew HF represent energy weighting coefficients of low frequency components and high frequency components, respectively, and represent the jth and kth (j≠k) feature maps of the lth convolutional layer, respectively.

[0084] As a preferred embodiment of the present application, the importance score of the feature map in step (4.5) is calculated, specifically:

[0085] The importance score of the feature map is calculated according to the following formula:

[0086]

[0087] wherein, denotes the importance score obtained from the similarity between the jth feature map of the lth convolutional layer and all other feature maps.

[0088] The device for implementing the pruning processing of the convolutional neural network based on wavelet transform of the present application, wherein the device comprises:

[0089] a processor configured to execute computer executable instructions;

[0090] a memory storing one or more computer executable instructions, which, when executed by the processor, implement each step of the method for implementing the pruning processing of the convolutional neural network based on wavelet transform.

[0091] The processor for implementing the pruning processing of the convolutional neural network based on wavelet transform of the present application, wherein the processor is configured to execute computer executable instructions, which, when executed by the processor, implement each step of the method for implementing the pruning processing of the convolutional neural network based on wavelet transform.

[0092] The computer readable storage medium of the present application, which stores a computer program that can be executed by a processor to implement each step of the method for implementing the pruning processing of the convolutional neural network based on wavelet transform.

[0093] ​The application provides a pruning convolutional neural network classification method based on wavelet transform.

[0094] As Figure 1 The overall framework of the application is shown in the figure. First, the feature map is obtained by traversing the convolutional layer. Second, the frequency domain transformation is performed on the feature map, and then the cosine similarity and the energy weighting coefficient of the high and low frequency components are used to calculate the similarity between the feature maps, and the importance score of the feature map is accumulated. In the third step, the unimportant filters are pruned according to the importance score. In the fourth step, the pruned network is fine-tuned.

[0095] The application is a convolutional neural network pruning method based on wavelet transform, which comprises the following steps:

[0096] (1) Obtain an image classification dataset and perform preprocessing: a publicly available dataset with labels (CIFAR-10, ImageNet) is used. The dataset used in the experiment (which has been divided into training set, validation set and test set) is rotated, scaled and operated.

[0097] (2) Obtain a publicly available convolutional neural network pre-trained model (such as VGGNet-16, ResNet-56, ResNet-110, etc.).

[0098] (3) Convert the feature map output by the convolutional neural network in the spatial domain to the frequency domain, and the specific operation is as follows:

[0099] Wavelet transform is a local transform in space and frequency, which analyzes functions at multiple scales through scaling and translation operation functions, has the unique characteristics of multi-resolution analysis, and can effectively extract information from signals, has strong analysis ability. In the time domain and frequency domain, detailed information of the signal is represented.

[0100] Wavelet transform has two types: discrete wavelet transform (DWT) and continuous wavelet transform (CWT), the main difference between which is the scaling and translation operation. CWT has the ability of all scaling and translation operations. It is widely used in computer signal research. DWT uses a specific set to distribute scaling and translation operations, which is more suitable for signal decomposition and reconstruction in signal coding.

[0101] The feature information in the convolutional neural network researched by the present application belongs to the category of computer signals, and is more suitable for being researched and analyzed by using CWT. Therefore, the present application uses CWT for wavelet decomposition. As shown in equation (1).

[0102]

[0103] where a is called a scale factor, controlling the stretching of the wavelet function corresponding to the frequency, and b is a translation factor, controlling the position of the wavelet function on the time axis. f(t) is the input signal. The Haar wavelet is a wavelet that has tight support, decay and orthogonality at the same time. Therefore, the Haar wavelet is used as the wavelet transform basis function. The function is a complex function of the Haar wavelet function.

[0104] As shown in Figure 2 , the output feature map in the convolutional neural network is subjected to wavelet transform to obtain four frequency domain components, which are low frequency information component LL, horizontal high frequency information component HL, vertical high frequency information component LH and diagonal high frequency information component HH, respectively. The low frequency information component LL corresponds to the region of the image that is degenerated and gentle, and retains the main information structure of the feature map. The high frequency components LH, HL and HH correspond to the region of the feature map that is subjected to strong transformation, and retain the detail information of the feature map. In addition, there is a significant difference between the energy of the high frequency information component and the low frequency information component of the feature map. Most of the energy of the feature map is concentrated in the low frequency information component LL, while less energy is distributed in the high frequency information components HL, LH and HH. Considering various factors, the present application sets LF to represent the low frequency information component LL (i.e. LF = LL), and HF to represent the sum of the high frequency components LH, HL and HH (i.e. HF = LH + HL + HH).

[0105] (4) Feature map importance score calculation: the present application uses the cosine similarity and the energy weighting coefficients of the high and low frequency components to calculate the importance of the feature map in the frequency domain.

[0106] First, by decomposing different feature maps of a certain convolutional layer based on WT, the similarity of the corresponding high and low frequency components is calculated as follows:

[0107]

[0108]

[0109] wherein, represents the calculation of the cosine similarity value of the low frequency component between and . represents the calculation of the cosine similarity value between the high frequency components and . And represents the first high frequency component obtained by wavelet transform. The low-frequency component of the j-th feature map in layer 1. It represents the sum of three high-frequency components. and They represent the first... The sum of the low-frequency component and the three high-frequency components obtained by wavelet transforming the k-th feature map of the convolutional layer.

[0110] After wavelet decomposition, general information of the feature map is stored in the low-frequency components, while the high-frequency components store the detailed signal of the feature map. To more accurately calculate and analyze the similarity between different feature maps for effective filter pruning, the weights of high-frequency and low-frequency components are calculated based on the energy of the feature map.

[0111] The energy weighting coefficient is defined as follows.

[0112]

[0113]

[0114]

[0115] Among them ew LF and ew HF These represent the energy weighting coefficients for the low-frequency and high-frequency components, respectively. E represents the energy of a certain feature map. LF and E HF These represent the low-frequency and high-frequency energies of a certain feature map, respectively. H and W represent the height and width of the feature map, respectively.

[0116] As shown above, by comparing the energy of the low-frequency components and the energy of the high-frequency components with the total energy of the feature map, the energy weight coefficients of the low-frequency and high-frequency components are derived respectively. The weighted similarity calculation between different feature maps can be obtained by combining equation (2), which is defined as follows.

[0117]

[0118] in, represent and Perform energy-weighted calculations. LF and ew HF These represent the energy weighting coefficients for the low-frequency and high-frequency components, respectively. and These represent the j-th and k-th (j≠k) feature maps of the l-th convolutional layer, respectively.

[0119] The above describes the similarity between two feature maps in the frequency domain. Then, the importance score of a feature map is calculated by summing the similarities between the feature maps. The importance score is defined as follows.

[0120]

[0121] in, Indicates from the first The importance score is obtained by comparing the similarity between the j-th feature map of the convolutional layer and all other feature maps.

[0122] The importance score for each feature map can be obtained through the above operations.

[0123] (5) Pruning operation: The importance score of each feature map is obtained according to the operations in (1)-(4). The feature maps in the same convolutional layer are sorted according to their importance scores, and then the filters corresponding to the sorted feature maps are pruned according to the set pruning threshold. Finally, the pruned model (i.e., the compressed model) is obtained.

[0124] In a specific embodiment of the present invention, the following is an example of the convolutional neural network pruning method based on wavelet transform:

[0125] (1) Pre-trained convolutional neural network model and image classification dataset setup: The convolutional neural network model used in this embodiment is the VGGNet-16 architecture, using the CIFAR-10 dataset. This dataset has 10 categories and a total of 60,000 color images, of which 50,000 color images are used for training and 10,000 color images are used for testing. Data augmentation processes such as horizontal flipping and normalization were applied to the dataset in this embodiment.

[0126] (2) Pre-trained model parameter settings: Iterative training for 200 epochs, with a batch size of 128. The initial learning rate was set to 0.01 and divided by 10 at the 100th and 150th training epochs. The network loss function used the SGD optimizer, with a momentum term coefficient of 0.9 and a weight decay coefficient of 5e-3.

[0127] (3) Obtain the output feature map of VGGNet-16 architecture: Use the hook function to obtain the output feature map of a total of 13 convolutional layers in the VGGNet-16 architecture and save it.

[0128] (4) Based on wavelet transform, the acquired feature map is converted to the frequency domain: Using wavelet transform formula (1), each feature map can be decomposed into four frequency domain components, namely, low-frequency information component LL, horizontal high-frequency information component HL, vertical high-frequency information component LH, and diagonal high-frequency information component HH. The low-frequency information component LL corresponds to the region of gradual degradation in the feature map, preserving the main information structure of the feature map. The high-frequency components LH, HL, and HH correspond to the region of strong transformation in the feature map, preserving the detailed information of the feature map. In addition, there is a significant difference between the energy of the high-frequency information components and the low-frequency information components of the feature map. Most of the energy of the feature map is concentrated in the low-frequency information component LL, while less energy is distributed in the high-frequency information components HL, LH, and HH. Considering various factors, this invention sets LF to represent the low-frequency information component LL (i.e., LF = LL), and HF to represent the sum of the high-frequency components LH, HL, and HH (i.e., HF = LH + HL + HH).

[0129]

[0130] (5) Calculate the importance score of the feature map: Based on the high and low frequency components obtained from the operation in (4), this embodiment first uses cosine similarity to calculate the similarity between high frequency components and the similarity between low frequency components. The calculation formula is shown in equation (2).

[0131]

[0132]

[0133] in Indicates calculation and The cosine similarity value of the low-frequency components between them. Indicates the calculation of high-frequency components and The cosine similarity value between them. The wavelet transform yields the first... The low-frequency component of the j-th feature map in layer 1. It represents the sum of three high-frequency components. and They represent the first... The sum of the low-frequency component and the three high-frequency components obtained by wavelet transforming the k-th feature map of the convolutional layer.

[0134] After wavelet decomposition, general information of the feature map is stored in the low-frequency components, while the high-frequency components store the detailed signal of the feature map. To more accurately calculate and analyze the similarity between different feature maps for effective filter pruning, the weights of high-frequency and low-frequency components are calculated based on the energy of the feature map.

[0135] The energy weighting coefficient is defined as follows.

[0136]

[0137]

[0138]

[0139] Among them ew LF and ew HF These represent the energy weighting coefficients for the low-frequency and high-frequency components, respectively. E represents the energy of a certain feature map. LF and E HF These represent the low-frequency and high-frequency energies of a certain feature map, respectively. H and W represent the height and width of the feature map, respectively.

[0140] As shown above, by comparing the energy of the low-frequency components and the energy of the high-frequency components with the total energy of the feature map, the energy weight coefficients of the low-frequency and high-frequency components are derived respectively. The weighted similarity calculation between different feature maps can be obtained by combining equation (2), which is defined as follows.

[0141]

[0142] in represent and Perform energy-weighted calculations. LF and ew HF These represent the energy weighting coefficients for the low-frequency and high-frequency components, respectively. and These represent the j-th and k-th (j≠k) feature maps of the l-th convolutional layer, respectively.

[0143] The above describes the similarity between two feature maps in the frequency domain. Then, the importance score of a feature map is calculated by summing the similarities between the feature maps. The importance score is defined as follows.

[0144]

[0145] in Indicates from the first The importance score is obtained by comparing the similarity between the j-th feature map of the convolutional layer and all other feature maps.

[0146] The importance score for each feature map can be obtained through the above operations. Indicates the first Feature map importance score set for each convolutional layer:

[0147] (6) Prune according to importance score and pruning rate: For the set of importance scores obtained in (5) Sort in descending order to get For the VGGNet-16 architecture, the layer pruning rate set in this embodiment is [0.45]*7+[0.70]*5, that is, the feature maps in the first 7 convolutional layers are pruned by 45%, and the feature maps in the last 5 convolutional layers are pruned by 70%. The pruning order is based on the sorted set. Finally, the experimental results are obtained based on the pruning results.

[0148] (7) Performance recovery: Fine-tune the pruned network to recover the lost performance.

[0149] This invention proposes a filter pruning method based on wavelet transform, in which the high-frequency and low-frequency components of wavelet transform decomposition can better represent the potential interrelationships of feature maps in the frequency domain.

[0150] This invention utilizes cosine similarity and energy weighting coefficients of high and low frequency components to more accurately calculate the similarity between feature maps, thereby pruning the filter based on the importance score obtained therefrom to reduce the accumulation of redundant similarity in the network.

[0151] The experimental results of this invention also show that this invention can be effectively applied to edge devices (such as mobile phones, watches, etc.) and embedded devices.

[0152] To demonstrate the technical effectiveness of this invention, image classification validation was performed on an image classification dataset, and relevant experimental data were recorded. Specifically, the Top-1 accuracy, FLOPs pruning rate (FPR), and parameter pruning rate (PPR) of the pruning model were recorded on the CIFAR-10 test dataset. Furthermore, the Top-5 accuracy was also added to the records on the ImageNet dataset. The Top-1 accuracy and Top-5 accuracy represent the accuracy of the first-ranked class matching the actual result and the accuracy of the top five classes containing the actual result, respectively. Under similar image classification accuracy conditions, higher FLOPs pruning rate and parameter pruning rate are better, indicating that the method prunes more FLOPs and parameters.

[0153] Figure 7 The results of pruning the VGGNet-16 architecture on the CIFAR-10 dataset, comparing the present invention with existing pruning methods, are shown. Figure 7 As can be seen, with similar FLOPs and reduced parameters, the present invention achieved a Top-1 score of 93.57%, which is better than all the comparison pruning algorithms.

[0154] Figure 8 The results of this invention and existing pruning methods on the ResNet-110 architecture on the CIFAR-10 dataset are shown. Figure 8 The report shows that when 42.9% of FLOPs and 38.2% of parameters were pruned, the present invention improved Top-1 accuracy by 0.62%. Furthermore, the present invention outperforms current pruning methods in all performance metrics. Compared to the L1 method, the present invention has an overall advantage. The present invention prunes more FLOPs (42.9% vs. 38.7%) and parameters (38.2% vs. 32.6%) while maintaining better Top-1 accuracy (94.12% vs. 93.30%). Compared to HRank, the present invention achieves 2.1% higher acceleration and 0.67% higher Top-1 accuracy, and its network structure is more compact than HRank's. Moreover, when the pruning rate exceeds 50%, the present invention also achieves better Top-1 accuracy (92.70% vs. 92.65%) and acceleration (73.2% vs. 68.6%) than HRank. In conclusion, this invention is applicable to pruned multi-branch network structures (residual networks) and has significant advantages in compressing and accelerating residual network models.

[0155] Figure 9 Experiments demonstrating the effectiveness of this invention on the large-scale complex image dataset ImageNet are presented. This invention trims 53.8% of FLOPs and 49.7% of parameters, with a Top-5 accuracy loss of only 0.37% and a Top-1 accuracy loss of 0.37%. This result outperforms adaptive importance GAL-0.5, SSS-32, and He et al. This invention outperforms the GDP method in all aspects. Compared to SSS-26, this invention trims 34% more FLOPs and 29.2% more parameters, while achieving higher accuracy. Compared to HRank, this invention achieves lower Top-1 accuracy loss (3.16% vs. 4.00%) and Top-5 accuracy loss (1.68% vs. 1.81%), while trimming more FLOPs (77.0% vs. 62.1%) and parameters (68.2% vs. 46.1%). In conclusion, this invention is also applicable to large-scale complex image datasets.

[0156] To further demonstrate the advantages of this invention, its Top-1 accuracy was compared with several advanced pruning methods at different FLOP pruning rates. The comparative experiments are as follows: Figure 4 and Figure 5The experimental results of ResNet-56 and DenseNet40 on CIFAR-10 show that, despite the changes in the pruning rate of FLOPs, the present invention consistently maintains a high Top-1 accuracy, which well demonstrates the correctness of exploring the potential correlations between feature maps in the frequency domain.

[0157] Accelerated comparison after model pruning:

[0158] The actual speedup performance of a pruned convolutional neural network architecture is affected by various factors, such as the percentage reduction in FLOPs, the number of available CPU / GPU cores, and the I / O latency of data exchange. This invention calculates the CPU / GPU inference speed on different convolutional neural networks to represent the speedup ratio after model pruning.

[0159] The actual acceleration of this invention on pruned networks was experimentally tested using CPUs and GPUs, and the reported results are as follows: Figure 6 As shown, for ResNet-56, ResNet-110, and DenseNet-40, the actual speedups achieved by the CPU after pruning according to this invention are 1.11×, 1.29×, and 1.42×, respectively. Furthermore, the actual speedups achieved by the GPU are 1.02×, 1.09×, and 1.37×, respectively. These results demonstrate the effectiveness of this invention in practical deployments.

[0160] For the specific implementation scheme of this embodiment, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0161] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0162] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0163] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0164] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0165] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0166] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0167] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0168] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0169] The present invention employs a method, apparatus, processor, and computer-readable storage medium for pruning convolutional neural networks based on wavelet transform. It does not modify or alter the pooling layer operations in the network, thus maintaining the integrity and simplicity of the network structure to the greatest extent. The energy weighting coefficients of the high- and low-frequency components after wavelet transform in this invention can more effectively preserve valuable feature information and improve image classification accuracy. This invention performs cosine similarity calculation between corresponding high- and low-frequency components and fuses this cosine similarity with the energy weighting coefficients of the high- and low-frequency components, further pruning feature maps containing redundant information. Since this invention does not modify the convolutional neural network architecture, the pruned convolutional neural network architecture is a structured pruning method that can be directly deployed on edge devices.

[0170] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for pruning convolutional neural networks based on wavelet transform, characterized in that, The method includes the following steps: (1) Obtain the image classification dataset and perform preprocessing; (2) Obtain publicly available pre-trained convolutional neural network models online; (3) Transform the feature map output by the convolutional neural network in the spatial domain to the frequency domain; (4) Calculate the feature map importance score; (5) Sort the feature maps according to their importance scores, and prune the filters corresponding to the sorted feature maps according to the set pruning threshold to obtain the pruned model. Step (4) specifically includes the following steps: (4.1) Calculate the similarity of the corresponding high and low frequency components by decomposing different feature maps of a certain convolutional layer based on WT; (4.2) Calculate the weights of high-frequency and low-frequency frequencies based on the energy of the feature map; (4.3) By comparing the energy of the low-frequency components and the energy of the high-frequency components with the total energy of the feature map, the energy weighting coefficients of the low-frequency and high-frequency components are derived respectively; (4.4) Calculate the weighted similarity between different feature maps; (4.5) The importance score of each feature map is calculated by summing the similarities between the feature maps.

2. The method for pruning convolutional neural networks based on wavelet transform according to claim 1, characterized in that, Step (3) specifically includes the following steps: Wavelet decomposition was performed using CWT; The output feature map in the convolutional neural network is transformed by wavelet transform to obtain four frequency domain components, namely the low-frequency information component LL, the horizontal high-frequency information component HL, the vertical high-frequency information component LH, and the diagonal high-frequency information component HH.

3. The method for pruning convolutional neural networks based on wavelet transform according to claim 2, characterized in that, In step (1), wavelet decomposition is performed using CWT, specifically as follows: Wavelet decomposition is performed using CWT according to the following formula: Where 'a' is called the scaling factor, which controls the scaling of the wavelet function corresponding to the frequency; 'b' is the translation factor, which controls the position of the wavelet function on the time axis; 'f(t)' is the input signal; and 'Haar wavelet' is the wavelet transform basis function. The function is a complex function of the Haar wavelet function.

4. The method for pruning convolutional neural networks based on wavelet transform according to claim 1, characterized in that, In step (4.1), the similarity between the corresponding high and low frequency components is calculated as follows: The similarity between the corresponding high and low frequency components is calculated using the following formula: in, Indicates calculation and The cosine similarity value of the low-frequency components between them. Indicates the calculation of high-frequency components and The cosine similarity value between them This represents the low-frequency component of the j-th feature map in the l-th layer obtained by wavelet transform. It represents the sum of three high-frequency components. and These represent the low-frequency component and the sum of three high-frequency components obtained by performing wavelet transform on the k-th feature map of the l-th convolutional layer, respectively.

5. The method for pruning convolutional neural networks based on wavelet transform according to claim 1, characterized in that, The weighted similarity between different feature maps is calculated in step (4.4) as follows: The weighted similarity between different feature maps is calculated using the following formula: in, represent and Perform energy-weighted calculations, ew LF and ew HF These represent the energy weighting coefficients for the low-frequency and high-frequency components, respectively. and These represent the j-th and k-th (j≠k) feature maps of the l-th convolutional layer, respectively.

6. The method for pruning convolutional neural networks based on wavelet transform according to claim 1, characterized in that, In step (4.5), the importance score of the feature map is calculated as follows: The importance score of the feature map is calculated using the following formula: in, This represents the importance score obtained from the similarity between the j-th feature map of the l-th convolutional layer and all other feature maps.

7. An apparatus for implementing wavelet transform-based convolutional neural network pruning, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for pruning convolutional neural networks based on wavelet transform as described in any one of claims 1 to 6.

8. A processor for implementing wavelet transform-based convolutional neural network pruning processing, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for pruning convolutional neural networks based on wavelet transform as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the various steps of the method for implementing convolutional neural network pruning based on wavelet transform as described in any one of claims 1 to 6.

Images