A flexible deep learning network model compression method based on channel gradient pruning

Through the flexible pruning method based on channel gradient, the problem of difficulty in deploying deep neural network models on resource-constrained platforms is solved, and stronger pruning prediction and accuracy is achieved, and real-time object recognition is realized in the smartphone APP.

CN112396179BActive Publication Date: 2024-11-26ZHEJIANG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011308919.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-20
Publication Date
2024-11-26
Estimated Expiration
2040-11-20

AI Technical Summary

Technical Problem

The difficulty in deploying deep neural network models on resource-constrained platforms leads to reduced application value, and existing pruning methods cannot effectively predict compression rates and lead to performance degradation.

Method used

Using a flexible pruning method based on channel gradient, the mask array is dynamically updated through the first-order Taylor expansion-based correlation loss function changes and parameter gradients to achieve more controllable model compression and allow the pruned channels to be re-issued when needed. activation.

Benefits of technology

It achieves stronger pruning prediction and accuracy, avoids performance degradation, is suitable for deep learning model deployment of resource-constrained platforms, and realizes real-time object recognition in smartphone APP.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112396179B_ABST
    Figure CN112396179B_ABST
Patent Text Reader

Abstract

A flexible deep learning network model compression method based on channel gradient pruning, comprising: step 1, adding a mask layer constraint to the original network to obtain a deep convolutional neural network model to be pruned, step 2, training the network model on a data set, updating the channel mask layer constraint according to the absolute value of the product of the channel gradient and the weight as the importance criterion, obtaining a mask and a sparse model, step 3, pruning the sparse model based on the mask, and step 4, retraining a compact deep convolutional neural network model. The present invention also provides the application effect of the flexible deep learning network model compression method based on channel gradient pruning on an actual object recognition APP. After pruning, the model greatly improves the recognition speed of objects, solves the technical problem that the deep neural network model cannot be deployed on embedded devices, smart phones and the like due to high storage space occupancy, high memory occupancy, and high computing resource occupancy, and expands the application scope of the deep neural network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a flexible deep learning network model compression method, and in particular to a practical application of object recognition on a smartphone APP. Background Art

[0002] Deep learning has been applied in many tasks such as image recognition, target detection, image segmentation, speech recognition, and QA question answering, and has achieved better results than traditional methods. However, the excellent performance of deep neural networks is closely related to their complex structure and the large number of parameters. And with the advancement of technology, there is a trend of complex neural network model structures in the field of deep learning. Therefore, high-performance deep neural network models have very high requirements for hard disk storage space, memory bandwidth, and platform computing resources. The contradiction between the high resource requirements of deep neural network models and the limited resources of platforms such as mobile phones, bracelets, and self-driving vehicles restricts their deployment on such platforms and reduces the application value of neural networks. In order to deploy deep neural network models on resource-constrained platforms, the network model needs to be further compressed to achieve the goal of reducing model complexity, model calculation, and improving inference speed while ensuring performance.

[0003] Pruning some redundant network parameters is a common method to compress convolutional neural network models and thus speed up model reasoning. Depending on the granularity of the modification and whether the network structure changes after pruning, pruning can be divided into unstructured pruning and structured pruning. The target of unstructured pruning is the internal weight parameter matrix of neurons (channels). During the pruning process, the weights in the channel will be deleted according to some rules to reduce the number of model parameters. In the paper Learning both weights and connections for efficient neural network (Han S, Pool J, Tran J, et al. Learning both weights and connections for efficient neural network [C]. Advances in neural information processing systems. 2015: 1135-1143.), Has et al. regarded weights with a size less than a threshold as redundant parameters and pruned them to obtain sparse channels. However, sparse channels cannot be directly accelerated by general hardware, making the actual pruned network rely on specially designed software and hardware for reasoning. The target of structured pruning is neurons (channels). Under the premise of ensuring stable performance and regular structure, structured pruning reduces the number of model parameters and accelerates network inference on general hardware by reducing the number of channels in each layer. In the patent (Guo Lie, Gao Jiandong, Zhao Jian, Liu Pengbo, Shi Zhenzhou. A deep learning network model compression method based on network layer pruning [P]. Liaoning Province: CN111401523A, 2020-07-10), Guo Lie et al. proposed a channel pruning method that uses the size of the Gamma parameter of the batch normalization (Batch Normalization, BN) layer as a discrimination criterion, and deletes channels whose Gamma parameters are less than the threshold. However, it is inappropriate for this method to use the Gamma parameter of the BN layer as a standard to reflect the impact of the actual channel on the result. The two are not completely consistent. In the paper Dynamic network structured pruning based on layer fusion feature coefficients (Lu Haiwei, Yuan Xiaotong. Dynamic network structured pruning based on layer fusion feature coefficients [J]. Pattern Recognition and Artificial Intelligence, 2019, 32(11): 1051-1059), Lu Haiwei et al. proposed a dynamic network structured pruning method based on layer fusion feature coefficients, while considering the influence of BN layer parameters and Dropout operation on feature maps, and using multiple dynamic parameters to find and delete redundant channels to obtain the optimal network structure.However, after this method transforms the structure of convolutional layer + BN layer into a fully connected layer structure, the introduction of hyperparameters to control the compression rate cannot directly reflect the actual compression rate, that is, it is impossible to predict the final floating-point operations per second (FLOPs) compression rate before the pruning is completed. In addition, the pruned channels no longer extract features, which means that the pruned model will inevitably experience a certain degree of performance degradation. Summary of the invention

[0004] The present invention overcomes the above-mentioned shortcomings of the prior art and proposes a flexible deep learning network model compression method based on channel gradient pruning.

[0005] The present invention considers the influence of pruning neurons on the change of loss function during the pruning process, and associates the change value of the loss function with the parameter size and parameter gradient based on the first-order Taylor expansion, so as to obtain the channel importance evaluation standard that only depends on the parameter value and parameter gradient. A mask layer tensor is set for each convolution kernel, and its value is determined by the channel importance evaluation standard, so that the parameters can be selectively involved in the network calculation. At the same time, flexible pruning allows the channels containing pruned parameters to be reactivated when necessary to relearn and extract features. The entire network pruning is more controllable, and the difference between the FLOPs and the target FLOPs corresponding to the number of channels saved after each pruning can be compared, and the number of channels that need to be pruned can be automatically updated. The present invention also provides a compact model after pruning that is imported into a smart device to realize object recognition through the application APP, including real-time object recognition function and object recognition function in photos.

[0006] A flexible deep learning network model compression method based on channel gradient pruning, the specific steps are as follows:

[0007] Step 1: Get the initial model of the deep convolutional neural network to be pruned. The specific implementation of this process is to add an additional masking layer after the output of the convolutional layer. The output of the previous convolutional layer is the input of the current convolutional layer. The input can be recorded as z l-1 , then the current convolutional layer operation can be recorded as f(z l-1 ), initialize a weight of 1, and the dimension is the same as f(z l-1 ) results in a consistent mask layer tensor m mask As the weight of the mask layer, the mask layer is then added after the corresponding convolutional layer to obtain the output constrained by the mask layer:

[0008] z l =f(z l-1 )×m mask(1.1) The output of the channel with mask 1 is the original output, while the output of the channel with mask 0 is 0 (the output of this channel has no effect on the result and the channel weight cannot be updated). When all the convolutional layers considered for pruning in the original deep convolutional neural network model are constrained, the mask array M is obtained. mask and the deep convolutional neural network to be pruned;

[0009] Step 2: Train the network model, update the model mask information, set the compression rate s of the model (which can refer to the percentage of the number of remaining channels in the network to the total number of channels, or the percentage of the remaining FLOPs to the original FLOPs), and use the training set to train the deep convolutional neural network model to be pruned under the compression rate limit;

[0010] Step 2-1: During the training process, calculate the global channel importance index: Based on the prior knowledge that the better the model performance, the lower the loss function, the impact of redundant channels on model changes is relatively small. The change in the loss function value is used to reflect the importance of the current channel. The mathematical expression is:

[0011]

[0012] represents the importance index of the i-th channel in the l-th layer, L(x, y), They represent the loss functions before and after pruning the output of the i-th channel of the l-th layer, respectively. x represents the data, and y represents the label corresponding to the data x. The smaller the value, the smaller the impact of the channel on the loss function. When calculating the change value of the loss function after cutting off a single channel, the present invention uses Taylor's first-order expansion to approximate the actual change amplitude of the loss function to reduce the amount of search calculation. The importance index of a single channel is:

[0013]

[0014] Refers to the output value of the ith channel of the lth layer, It represents the return gradient of the output value of the ith channel of the lth layer. In the specific implementation, the return gradient and output value are obtained by inserting a Hook in the forward channel of the network, and then the absolute value of the product obtained by multiplying the return gradient and the output value is normalized to obtain the final channel importance index.

[0015] Step 2-2: During the training process, the network weight parameter W is updated by stochastic gradient descent, and the mask array M is updated by the iterative threshold selection method. The process of generating thresholds and adjusting masks is to sort the channels to be considered for pruning from small to large according to the channel importance index, and generate a table including index, key and value. Index is the sequence subscript, key is the channel coordinate (l, i) ((l, i) represents the i-th channel of the l-th layer), and value is the channel importance index. The three correspond one to one, and the total number of channels n is counted, then the threshold is

[0016] T I =I s×n (1.4) For channels whose importance index is less than the threshold, the mask of the channel with the corresponding coordinate is set to 0 according to the corresponding relationship in the table, and the mask of the remaining channels is set to 1;

[0017] Step 2-3: During the training process, the mask information is updated flexibly and iteratively. For the mth round of training, the mask is reset to all 1, and the entire network is trained on the data set. According to step 2-2, step 2-3 updates the mask information and saves it. For the m+1th round of training, the corresponding channels are shielded according to the mask information of the previous round, and the residual network is trained on the data set. For the m+2th round of training, according to the mth round of training process, the entire network is trained on the data set. At this time, the pruned channels in the m+1th round are restored, and the above iterative process is repeated until all training rounds are completed and the training is stopped. The sparse model and mask information are finally saved.

[0018] Step 3: The mask information obtained in steps 2-3 is a table, including key and value, where key is the channel coordinate (l, i) ((l, i) represents the i-th channel of the l-th layer), and value is the mask value 0 or 1. Count the number of channels with a mask value of 1 in each layer, reconstruct a new smaller network according to the number of remaining channels in each layer, correspond the channel coordinates of the new network to the channel coordinates in the mask information in order, and then save the weight parameters that are still activated in the sparse model to the new network one by one according to the coordinate correspondence, and obtain a compact deep convolutional neural network model;

[0019] Step 4: Complete the retraining of the compact deep convolutional neural network model. Retrain the pruned compact deep convolutional neural network model on the original data set. The hyperparameters are set to select 128 batches for each iteration, and a total of 50 to 100 epochs are iterated. The optimization method is stochastic gradient descent with a momentum of 0.90 to 0.99, and the weight decay is 0.0001 to 0.001. The initial learning rate is set to 0.01 to 0.1, and then the learning rate is divided by 10 after 3 epochs when the loss does not change significantly. The final network model with good performance and a small number of model parameters is output.

[0020] The present invention also applies the compact neural network model compressed by the flexible deep learning network model compression method based on channel gradient pruning to the object recognition APP of the smartphone to realize real-time object recognition in the form of video and recognition of objects in a single picture. The steps of importing the model into the APP to realize object recognition are as follows:

[0021] Step 1. Load and call the model. Convert the pruned model to torchScript format on the computer and save it as a model file with the suffix .pt. Put it in the assets folder of the Android project. At the same time, import the Pytorch1.4 dependency package in Android Studio on the Android side, and import the model file under the assets path through the Module.load function. The present invention will run the model file through the Pytorch Mobile mobile terminal framework that comes with Ptorch.

[0022] Step 2. You need to call the camera to load the image and import the dependency package CameraX of androidx.camera. Request permission to open the phone's camera. If allowed, you can use the camera and call the camera's preview image to display on the UI interface. If not allowed, a prompt message will be displayed. Next, open the camera, register ConnectionCallback and OnImageAvailableListener, set some output parameters for the camera to obtain images, including image preview size and camera direction, etc. Then, call the camera and take a picture of the most recent queue taken by the camera every 0.5s. Adjust the obtained image to an image with a width of 224 and a height of 224, input it into the neural network for processing, call the child thread to convert the image format and implement model calculation and UI interface update. Finally, when the preprocessing is completed, the camera device will call back the previously registered OnImageAvailableListener;

[0023] Step 3. Obtain the model results. First, convert the Image type image in YUV420 format into a Tensor input with a dimension of (1, 3, 224, 224) through the imageYUV420CenterCropToFloatBuffer function. Preprocess the input data according to the standard preprocessing method of the ImageNet data set. Input the converted image into the model in Tensor form for forward calculation. Copy the output nodes and output values ​​of the neural network to obtain the classification category with the highest probability. Finally, save the reasoning time, FPS, average reasoning time, and output category with the highest probability of a single frame. Display the results on the screen through the textview component provided by Android, thereby realizing real-time object recognition. In addition, the present invention also adds the function of importing a single picture for image classification, importing the image from the album in the form of a bitmap, and then performing real-time object recognition.

[0024] The advantages of the present invention are as follows: compared with the traditional pruning method based on L1 or L2 norm, the target compression rate is directly related to the actual compression rate, the pruning result is more predictable, and the optimal structure can be obtained more quickly and conveniently. Compared with the traditional method, the channels with weights close to 0 still have an actual impact on the network prediction results. The channels that need to be pruned based on the gradient evaluation standard and the mask layer constraint do not directly participate in the training and thus do not interfere with the prediction results. This means that the importance differences of channels can be distinguished more accurately, thereby improving the pruning accuracy. The iterative update threshold makes the mask array dynamically changing, which means that the algorithm is allowed to repair anomalies such as "wrong pruning" and "missed pruning", while the traditional method does not have this mechanism. The present invention actually applies the pruned neural network model to a smartphone APP for object recognition, further proving the application value of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic flow chart of the method of the present invention.

[0026] Figure 2 This is a diagram of adding a masking layer constraint to the convolutional layer being considered for pruning.

[0027] Figure 3a to Figure 3c is a schematic diagram of updating mask information according to channel importance index during network training of the present invention, wherein Figure 3a At the kth training epoch, some mask information is modified according to the channel importance index (Value mask From 1 to 0), Figure 3b At the k+1th training epoch, due to channel output shielding, some channels are no longer updated, and the importance index changes. Figure 3cAt the k+nth training epoch, the masked channel is reactivated (the Value mask From 0 to 1), a new round of mask update is performed.

[0028] Figure 4a to Figure 4b is the result of object identification in the specific implementation of the present invention, Figure 4a is the original model recognition result, Figure 4b is the compact model identification result. DETAILED DESCRIPTION

[0029] In order to more clearly understand the purpose, features and advantages of the present invention, the technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] The present invention is a flexible deep learning network model compression method based on channel gradient pruning, and the specific process is as follows:

[0031] Step 1: Get the deep convolutional neural network model to be pruned

[0032] Step 1-1: Add mask layer constraints to the convolutional layers that need to be pruned in the original deep convolutional neural network. Import the pre-trained weight parameters obtained from the network into the original deep convolutional neural network with constraints added to form a deep convolutional neural network model to be pruned. The number of channels in the mask layer corresponds to the number of channels in the convolutional layer to be pruned, and also corresponds to the mask length. The mask represents the importance of the corresponding channel.

[0033] Specifically, Figure 2 As shown in the figure, consider pruning the first and second convolutional layers in the Bottleneck structure of ResNet, then only a mask layer needs to be added after these two layers. In order to ensure that the number of feature maps of the bypass connection is the same as the number of output feature maps of the third convolutional layer, the third convolutional layer is not included in the range of network layers considered to be pruned.

[0034] Step 1-2: Initialize the mask tensor m mask Assuming that the number of layers to be pruned is N, the mask tensor is a tensor with the same dimension as the previous convolutional layer. The value of its first dimension is 0 or 1. The mathematical expression of the process of constraining the output is

[0035]

[0036] where z l represents the output of the first layer, f(z l-1) represents the convolution operation of the lth layer, and the i-th channel of the nth mask layer corresponds to the i-th channel of the lth convolution layer. The output of the channel with mask 1 is the output of the corresponding channel of the previous convolution layer, and the output of the channel with mask 0 is 0. After the code is updated, the number of feature maps output by the current network structure changes. The reduced part can be understood as the channel subset that contributes very little or even no contribution to the prediction result and is redundant.

[0037] Step 2: Train the network model and update the model mask information

[0038] Step 2-1: Based on the prior knowledge that the better the model performance, the lower the loss function, intuitively, the redundant channels have a relatively small impact on the model change. Therefore, if the magnitude of the change in the loss function value is used to reflect the importance of the current channel, its mathematical expression is:

[0039]

[0040] represents the importance index of the i-th channel in the l-th layer, L(x, y), They represent the loss function before and after pruning the output of the i-th channel of the l-th layer, respectively. x represents the data, and y represents the label corresponding to the data x. When calculating the change in the loss function after pruning a single channel, the present invention uses Taylor's first-order expansion to approximate the actual change in the loss function, so as to reduce the amount of calculation required to search for the channel to be pruned. Specifically, the loss function L(x, y) is Taylor expansion, we have:

[0041]

[0042] in is the high-order term in the Taylor expansion. Combining equations (2.2) and (2.3), the absolute value of the product obtained by multiplying the return gradient and the output value is used as the channel importance index. The index reflects the contribution of the parameter to the prediction result. The mathematical expression of the index is as follows:

[0043]

[0044] Refers to the output value of the ith channel of the lth layer, It represents the return gradient of the output value of the ith channel of the lth layer. In the specific implementation, the return gradient and output value are obtained by inserting a Hook into the forward channel of the network. Considering the influence of network depth on channel gradients and channel parameters, normalization is introduced to eliminate the depth influence.

[0045]

[0046] represents the importance index of the i-th channel of the l-th layer after normalization, and the importance index of all channels in the l-th layer The sum is 1.

[0047] Step 2-2: Sort the contribution of each convolution kernel according to the importance index of each channel, calculate the threshold according to the compression rate, set the corresponding mask of the channel with importance value lower than the threshold to 0, and the channel output will be shielded by the mask layer, such as Figure 3a As shown in Figure 1, at the end of the kth epoch, the masking layer shields the channels with low contribution to the prediction, and obtains a mask array and a sparsely structured network model.

[0048] Step 2-3: If Figure 3b As shown in , during the k+1th to k+n-1th epochs, the network model is trained on the data set according to the network structure after redundant channel shielding. Figure 3c As shown in Figure 1, at the k+nth epoch, the first dimension of the mask layer weights is all set to 1, at which time all masked channels are activated and the channel parameters are updated. According to step 2-2, the threshold is updated iteratively to achieve dynamic update of the mask array and the sparse network model.

[0049] Step 2-4: At the end of step 2-3, the compression level of the pruned model is predicted based on the mask information to see whether it has reached the expected target, such as the remaining FLOPs and the number of remaining channels. If so, step 2 ends; otherwise, the compression rate is adjusted and steps 2-2 and 2-3 are iterated.

[0050] Step 3: Complete model pruning based on mask information

[0051] Determine in turn whether each mask in the mask array M is 0, reorganize the mask array according to the layer name of the mask to obtain a mask information table, including key and value, key is the channel coordinate (indicates the channel of the layer), value is the mask value 0 or 1. Count the number of channels with a mask value of 1 in each layer, reconstruct a new smaller network according to the number of remaining channels in each layer, correspond the channel coordinates of the new network to the channel coordinates in the mask information in order, and then save the weight parameters that are still in the activated state in the sparse model to the new network one by one according to the coordinate correspondence, and obtain a compact deep convolutional neural network model.

[0052] Step 4: Complete the compact deep convolutional neural network model retraining

[0053] Step 4-1: Based on the retained network weight parameters and the compact deep convolutional neural network model, use the data set to retrain the network model. The retraining process is consistent with the pre-training process, but the hyperparameters need to be adjusted according to the compact deep convolutional neural network model so that the model converges to the optimal solution, that is, the final network model is obtained.

[0054] Step 4-2: Implementation conditions and result evaluation: In the embodiment of the present invention, the code implementation and deep learning framework are Python and PyTorch respectively. The data set is ImageNet-1k, which contains 1.28 million training images and 50,000 verification images, with a total of 1,000 categories. Each channel of the image is normalized. In data enhancement, the image is padded with zeros, randomly flipped, and randomly cropped to obtain an image block with a length and width of 224. During the training process, the number of batches for each iteration is selected as 128, and a total of 100 epochs are iterated. The optimization method selects stochastic gradient descent with 0.90 momentum, the weight decay is 0.0001, and the initial learning rate is set to 0.01. Then, after 3 epochs when the loss does not change significantly, the learning rate is divided by 10.

[0055] The evaluation index of convolutional neural network model pruning generally uses the accuracy of the original network on the test set to reflect the change in the performance of the model before and after pruning. At the same time, the FLOPs of the model reflects the pruning and acceleration effect of the model.

[0056] The results of the comparison between the examples of the present invention and the existing methods on the ResNet-50 model and the ImageNet-1k dataset are shown in Table 1, where the cropping targets of all methods except the original model are set to cut off 50% of the channels.

[0057] Table 1. Results of various methods for compressing the ResNet-50 model on the ImageNet-1k dataset

[0058]

[0059] Step 5: As shown in Figure 4, the compact deep convolutional neural network can be applied to smartphones and can effectively complete tasks such as object recognition in photos and real-time object recognition. It can realize real-time object recognition in the form of video and recognition of objects in a single picture. The steps to import the model into the APP to realize object recognition are as follows:

[0060] Step 5-1. Load and call the model. Convert the pruned model to torchScript format on the computer and save it as a model file with the suffix .pt. Put it in the assets folder of the Android project. At the same time, import the Pytorch1.4 dependency package in Android Studio on the Android side, and import the model file under the assets path through the Module.load function. The present invention will run the model file through the Pytorch Mobile mobile terminal framework that comes with Ptorch.

[0061] Step 5-2. You need to call the camera to load the image and import the dependency package CameraX of androidx.camera. Request permission to open the phone's camera. If allowed, you can use the camera and call the camera's preview image to display on the UI interface. If not allowed, a prompt message will be displayed. Next, open the camera, register ConnectionCallback and OnImageAvailableListener, set some output parameters for the camera to obtain images, including image preview size and camera direction, etc. Then, call the camera and take a picture of the most recent queue taken by the camera every 0.5s. Adjust the obtained image to an image with a width of 224 and a height of 224, input it into the neural network for processing, call the child thread to convert the image format and implement model calculation and UI interface update. Finally, when the preprocessing is completed, the camera device will call back the previously registered OnImageAvailableListener;

[0062] Step 5-3. Obtain model results. First, convert the image type image in YUV420 format into a Tensor input with dimensions of (1, 3, 224, 224) through the imageYUV420CenterCropToFloatBuffer function. Preprocess the input data according to the standard preprocessing method of the ImageNet data set. The converted image is input into the model in Tensor form for forward calculation. The output nodes and output values ​​of the neural network are copied to obtain the classification category with the highest probability. Finally, save the reasoning time, FPS, average reasoning time, and the output category with the highest probability of a single frame image. Display the result on the screen through the textview component provided by Android, thereby realizing real-time object recognition. Practice has proved that the present invention can greatly reduce the size of the network model while ensuring network performance, and accelerate the operation of the network model, and has practical application value.

[0063] The present invention also provides the application effect of the flexible deep learning network model compression method based on channel gradient pruning on the actual object recognition APP, and the recognition speed of the model after pruning is greatly improved. Through the technical solution in the present invention, the existing main deep convolutional neural network model is effectively compressed under the premise of maintaining the original network accuracy, solving the technical problem that the deep neural network model cannot be deployed on embedded devices, smart phones and other devices due to high storage space occupation, high memory occupation, and high computing resource occupation, and expanding the application scope of deep neural networks.

[0064] The contents described in the embodiments of this specification are merely examples of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A flexible deep learning network model compression method based on channel gradient pruning, the specific steps are as follows: Step 1: Get the initial model of the deep convolutional neural network to be pruned. The specific implementation of this process is to add an additional masking layer after the output of the convolutional layer. The output of the previous convolutional layer is the input of the current convolutional layer, and the input is recorded as z l-1 , then the current convolutional layer operation is recorded as f(z l-1 ), initialize a weight of 1, with the same dimension as f(z l-1 ) results in a consistent mask layer tensor m mask As the weight of the mask layer, the mask layer is then added after the corresponding convolutional layer to obtain the output constrained by the mask layer: With l =f(z l-1 )×m mask (1) The output of the channel with a mask of 1 is the original output, while the output of the channel with a mask of 0 is 0. The output of this channel has no effect on the result and the channel weight cannot be updated. When all the convolutional layers considered for pruning in the original deep convolutional neural network model are constrained, the mask array M is obtained. mask and the deep convolutional neural network to be pruned; Step 2: Train the network model, update the model mask information, and set the model compression rate s, which refers to the percentage of the remaining channels in the network to the total number of channels, and also refers to the percentage of the remaining FLOPs to the original FLOPs. s The percentage of,under the compression rate limit, the training set is used to train the deep convolutional neural network model to be pruned; Step 2-1: During the training process, calculate the global channel importance index: Based on the prior knowledge that the better the model performance, the lower the loss function, the impact of redundant channels on model changes is relatively small. The change in the loss function value is used to reflect the importance of the current channel. The mathematical expression is: represents the importance index of the i-th channel in the l-th layer, L(x, y), They represent the loss functions before and after pruning the output of the i-th channel of the l-th layer, respectively. x represents the data, and y represents the label corresponding to the data x. The smaller the value, the smaller the impact of the channel on the loss function. When calculating the change in the loss function after cutting off a single channel, Taylor's first-order expansion is used to approximate the actual change in the loss function to reduce the amount of search calculations. The importance index of a single channel is: Refers to the output value of the ith channel of the lth layer, It represents the return gradient of the output value of the ith channel of the lth layer. The return gradient and output value are obtained by inserting a Hook in the forward channel of the network, and then the absolute value of the product obtained by multiplying the return gradient and the output value is normalized to obtain the final channel importance index. Step 2-2: During the training process, the network weight parameter W is updated by stochastic gradient descent, and the mask array M is updated by the iterative threshold selection method. The process of generating thresholds and adjusting masks is to sort the channels to be considered for pruning from small to large according to the channel importance index, and generate a table including index, key and value. Index is the sequence subscript, key is the channel coordinate (l, i), (l, i) represents the i-th channel of the l-th layer, and value is the channel importance index. The three correspond one to one, and the total number of channels n is counted, then the threshold is: T I =I s×n (4) For channels whose importance index is less than the threshold, the mask of the channel with the corresponding coordinate is set to 0 according to the corresponding relationship in the table, and the mask of the remaining channels is set to 1; Step 2-3: During the training process, the mask information is updated flexibly and iteratively. For the mth round of training, the mask is reset to all 1, and the entire network is trained on the data set. According to step 2-2, step 2-3 updates the mask information and saves it. For the m+1th round of training, the corresponding channels are shielded according to the mask information of the previous round, and the residual network is trained on the data set. For the m+2th round of training, according to the mth round of training process, the entire network is trained on the data set. At this time, the pruned channels in the m+1th round are restored, and the above iterative process is repeated until all training rounds are completed and the training is stopped. The sparse model and mask information are finally saved. Step 3: The mask information obtained in steps 2-3 is a table, including key and value, where key is the channel coordinate (l, i) and value is the mask value 0 or 1; count the number of channels with mask value 1 in each layer, reconstruct a new smaller network according to the number of remaining channels in each layer, correspond the channel coordinates of the new network to the channel coordinates in the mask information in sequence, and then save the weight parameters that are still activated in the sparse model to the new network one by one according to the coordinate correspondence, and obtain a compact deep convolutional neural network model; Step 4: Complete the retraining of the compact deep convolutional neural network model. Retrain the pruned compact deep convolutional neural network model on the original data set. The hyperparameter is set to 128 batches for each iteration, and a total of 50 to 100 epochs are iterated. The optimization method is stochastic gradient descent with a momentum of 0.90 to 0.99, and the weight decay is 0.0001 to 0.

001. The initial learning rate is set to 0.01 to 0.1, and then the learning rate is divided by 10 after 3 epochs when the loss does not change significantly; the final network model with good performance and a small number of model parameters is output; Step 5. Apply the compact neural network model compressed by the flexible deep learning network model compression method based on channel gradient pruning to the object recognition APP of the smartphone to realize real-time object recognition in the form of video and recognition of objects in a single picture. The steps for the object recognition APP to realize object recognition are as follows: Step 51. Load and call the model. Convert the pruned model to torchScript format on the computer and save it as a model file with the suffix .pt. Put it in the assets folder of the Android project. At the same time, import the Pytorch1.4 dependency package in AndroidStudio on the Android side, and import the model file under the assets path through the Module.load function. The model file will be run through the Pytorch Mobile framework that comes with Ptorch. Step 52. The camera needs to be called to load the image, and the dependent package CameraX of androidx.camera is imported; request permission to open the camera of the mobile phone. If allowed, the camera can be used, and the preview image of the camera is called to be displayed on the UI interface. If not allowed, a prompt message will be displayed. Next, the camera is opened, ConnectionCallback and OnImageAvailableListener are registered, and the output parameters of the camera to obtain the image are set, including the image preview size and the camera direction. Then, the camera is called, and a picture of the most recent queue taken by the camera is taken every 0.5s. The obtained image is adjusted to an image with a width of 224 and a height of 224, and is input into the compact neural network model for processing. The child thread is called to convert the image format and implement the model calculation and update the UI interface. Finally, when the preprocessing is completed, the camera device will call back the previously registered OnImageAvailableListener; Step 53. Get the model results. First, use the imageYUV420CenterCropToFloatBuffer function to convert the YUV420 Image type image into a Tensor input with a dimension of (1, 3, 224, 224). Preprocess the input data according to the standard preprocessing method of the ImageNet dataset. Input the converted image into the model in Tensor form for forward calculation. Copy the output nodes and output values ​​of the neural network to obtain the classification category with the highest probability. Finally, save the inference time, FPS, average inference time, and output category with the highest probability for a single frame. Display the results on the screen through the textview component provided by Android, thereby achieving real-time object recognition.