Garbage classification identification method based on network pruning and knowledge distillation

Abstract

The garbage classification image recognition method based on network pruning and knowledge distillation belongs to the field of computer vision, and researches a garbage classification recognition method for fixed angle camera under daily illumination by using time interval video frame interception. Firstly, a direct connection type image classification model is constructed by using a convolutional neural network. Secondly, the network is pruned by using a pruning standard. The pruning standard focuses on the similarity between the feature map after channel pre-pruning and the original feature map, and uses the two norm to normalize the influence of the feature map size, so that the cross-layer automatic pruning based on the pruning standard can be realized. Finally, knowledge distillation is used to make up for the precision drop caused by over-pruning, and the knowledge of the original network is migrated to the pruned network. The classifier constructed by the application can reduce the model calculation resource consumption by 77% without losing the network precision, reduce the hard disk space occupation ratio by 50%, and the recognition accuracy can reach 94.4%.

Description

Technical Field

[0001] This invention, based on deep learning technology, studies a garbage classification and identification method using timed frame capture from a fixed-angle camera under everyday lighting conditions. First, a direct-connection image classification model is constructed using a convolutional neural network. Second, the network is pruned using a pruning criterion. This pruning criterion focuses on the similarity between the pre-pruned feature map and the original feature map, and uses a second normal form to normalize the influence of feature map size, thus enabling automated cross-layer pruning based on the pruning criterion. Finally, knowledge distillation is used to compensate for the accuracy loss caused by over-pruning, transferring knowledge from the original network to the pruned network. This invention belongs to the field of computer vision, specifically involving deep learning and image classification technologies. Background Technology

[0002] The purpose of waste sorting is to increase the resource and economic value of waste, reduce the amount of waste requiring treatment and the use of treatment equipment, lower treatment costs, and reduce the consumption of land resources, thus achieving social, economic, and ecological benefits. The standard for waste sorting is generally to classify household waste into four categories: recyclables, hazardous waste, kitchen waste, and other waste. However, the public's understanding of specific waste classification is not yet thorough. Therefore, establishing a waste identification and classification model through technology can guide the public to pay more attention to waste sorting.

[0003] With the booming development of artificial intelligence in the information society, the demand for deep convolutional neural networks (hereinafter referred to as convolutional networks) is exploding due to their remarkable high-precision capabilities in intelligent tasks such as computer vision, natural language understanding, and speech recognition. To achieve increasingly higher precision in real-world tasks, the depth of convolutional networks is continuously increased, leading to a significant increase in network space and computational overhead. However, with limited space and computing resources in mobile devices, wearable devices, and IoT devices, these high-precision but high-cost convolutional networks have faced obstacles in practical applications.

[0004] The impact of datasets on waste sorting systems is common. A self-made waste sorting dataset considers different lighting conditions, angles, distances, and occlusions, making the model more universal. Related technologies are characterized by a large number of model parameters and enormous computational resources. Comparative documents include: Kang Zhuang, Yang Jie, Guo Haoqi. Design of Automatic Waste Sorting System Based on Machine Vision [J]. Journal of Zhejiang University (Engineering Science), 2020, 54(07):1272-1280+1307. Wang Li, He Mutian, Xu Shuo, Yuan Tian, ​​Zhao Tianyi, Liu Jianfei. Waste Sorting and Detection Based on YOLOv5s Network [J]. Packaging Engineering, 2021, 42(08):50-56.

[0005] Network pruning significantly reduces the number of model parameters and computational cost while maintaining model accuracy. However, most pruning metrics cannot be compared across layers, requiring the specification of pruning ratios for each layer, which introduces suboptimal optimization issues. Examples include the comparison document Lin, Mingbao, et al. "Hrank: Filter pruning using high-rank feature map." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020 (Lin et al. proposed a pruning metric based on high-rank feature maps at the IEEE / CVF conference on computer vision and pattern recognition in 2020); and Sui, Yang, et al. "Chip: Channel independence-based pruning for compact neural networks." Advances in Neural Information Processing Systems 34(2021):24604-24616 (Sui et al. proposed a channel independence-based pruning metric in the journal Advances in Neural Information Processing Systems in 2021).

[0006] Comparing pruning metrics across layers facilitates automated pruning, where the overall pruning ratio is determined manually, and then the pruning metrics are compared to allocate the pruning amount to each layer of the network. However, a problem with cross-layer comparisons is the inconsistency in feature map sizes across different layers, which affects the pruning metrics. To achieve classification models with small parameter counts and high accuracy, automating network pruning significantly reduces the workload of manually determining pruning ratios, facilitating the discovery of more efficient structures within convolutional networks. Therefore, finding a pruning metric that can be compared across layers is a feasible solution for automated pruning. Summary of the Invention

[0007] This invention proposes a waste sorting and identification method based on network pruning and knowledge distillation, utilizing deep learning model compression techniques. First, a directly connected waste sorting network N is constructed using a convolutional network. a The self-built garbage classification dataset is input into the classification network N. a Obtain the training network N b Next, the pruning criteria for all channels in the network are calculated, and the training network N is adjusted according to the magnitude of the pruning criteria. b Perform channel pruning to obtain the pruned network N. c Finally, knowledge distillation was used to recover the trained network N.c The accuracy decrease caused by over-pruning is used to obtain the target network N. d Video frames recorded by a camera at a fixed angle under normal lighting conditions are captured at regular intervals, cropped, and then input into the constructed target network N. d The system performs waste sorting and identification detection. Based on the detection results, an audible alarm is triggered for any sorting errors.

[0008] By periodically capturing video frames, the computational resource consumption of the waste sorting system can be reduced, while enabling rapid response. After pruning and knowledge distillation training, the model's size and recognition speed are further improved. The waste sorting and recognition architecture of this method is shown in Appendix 1, and the main process is as follows. Figure 2 As shown, it can be divided into the following three parts: training directly connected convolutional networks using self-constructed datasets, automated pruning based on pruning metrics, and accuracy recovery using knowledge distillation.

[0009] (1) Self-constructed waste sorting dataset. Waste sorting systems are generally affected by the dataset. The self-constructed waste sorting dataset takes into account different lighting, angles, distances, and occlusion conditions, making the model more universal. Targeting the latest waste sorting and processing methods, this invention constructs a dataset containing four categories: recyclables, kitchen waste, hazardous waste, and other waste. This differs from traditional waste sorting categories, which include only a single category and lack practical value.

[0010] (2) Automated Pruning Based on Pruning Indicators. The pruning index proposed in this invention evaluates channel importance by utilizing the similarity between the feature map generated by channel pre-pruning and the original feature map. Furthermore, it uses second-normative normalization of the pruning index, thereby enabling automated cross-layer pruning. Unlike traditional pruning indices that cannot perform cross-layer comparisons and require manual specification of the pruning ratio for each layer, this invention, after specifying the overall model pruning ratio, automatically determines the number of prunes for each layer based on the numerical comparison of the pruning index.

[0011] (3) Accuracy recovery based on knowledge distillation. Excessive pruning can lead to a decrease in the accuracy of convolutional network models. Traditional methods only use fine-tuning training to recover accuracy. The method provided in this invention uses the original training network as a guide model to transfer knowledge to the pruned network. The accuracy of garbage classification and recognition is significantly improved, while the computational cost and number of parameters of the model are reduced.

[0012] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0013] This invention utilizes a strategy of periodically capturing video frames to reduce the recognition frequency. Furthermore, data acquired from garbage videos taken at fixed angles under normal lighting conditions is generally applicable to scenarios meeting relevant constraints. A lightweight garbage classification and recognition direct connection network is constructed using automated pruning based on pruning metrics and training based on knowledge distillation. First, based on pruning metrics that ensure the feature maps of the pre-pruned channels are similar to the original feature maps, the manual setting of the pruning ratio for each layer is reduced, achieving automatic allocation of the pruning quantity for each layer. Then, a knowledge distillation strategy is used on the pruned network to transfer knowledge from the original feature maps to the feature maps of the pruned channels, compensating for the accuracy loss due to pruning. Without sacrificing model accuracy, the computational cost and number of parameters are further reduced.

[0014] Experiments have shown that the deep neural network based on VGGNet19 can achieve a model FLOPs of 90M and a model size of 1.1M. Compared to the original network, computational resource consumption is reduced by 77%, disk space usage is reduced by 50%, and recognition accuracy reaches 94.4%. This is feasible and has significant application value for achieving efficient, accurate, and real-time waste sorting detection. Attached Figure Description

[0015] Figure 1 Architecture diagram of waste sorting and identification method.

[0016] Figure 2 A flowchart of the waste sorting network pruning process.

[0017] Figure 3 A diagram illustrating the structure of a waste sorting network.

[0018] Figure 4 A diagram illustrating the calculation method for pruning metrics.

[0019] Figure 5 The structure diagram of the loss function for knowledge distillation.

[0020] Figure 6 Network pruning flowchart.

[0021] Figure 7 Knowledge distillation flowchart. Detailed Implementation

[0022] Based on the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.

[0023] Step 1: First, construct a directly connected garbage sorting network N using convolutional networks. a The VGGNet19 convolutional neural network proposed in this invention is implemented based on the mainstream open-source deep learning framework PyTorch. A detailed structural diagram of the convolutional neural network proposed in this step is attached. Figure 2The self-built garbage classification dataset is input into the classification network N. a Obtain the training network N b Classification network N a It contains 21 modules of three types: convolutional module M1, pooling module M2, and linear classification module M3. Modules are directly connected via data transfer. Convolutional module M1 consists of a ReLU activation layer, a batch normalization layer (BN), and a convolutional layer (Conv). The kernel size of the convolutional layer (Conv) is 3×3. The classification network N... a The input image X0 is 224×224 in size. Except for modules 3, 6, 11, and 16, which are pooling modules M2, and module 21, which is a linear classification module M3, the classification network N... a The other modules are all convolutional modules M1.

[0024] M1:X i =ReLU(BN(Conv(X) i-1 )))

[0025] M2:X i =MaxPool(X i-1 )

[0026] M3:Y = Linear(avgPool(X) n ))

[0027] N a =[(M1,64)×2+M2+(M1,128)×2+M2+(M1,256)×4+M2+(M1,512)×4+M2+(M1,512)×4+(M3,4)]

[0028] Among them, X i-1 X represents the feature map output by the (i-1)th module. i This represents the feature map output by the i-th module (i.e., the feature map output by the next module after the (i-1)-th module), where X0 represents the input image. Y represents the label corresponding to the input network image. + indicates direct connection between modules. ×n indicates direct connection between n identical modules. (M1,64) indicates that the number of channels in convolutional module M1 is fixed at 64. Similarly, (M1,128) and (M1,512) indicate that the number of channels in convolutional module M1 is 128 and 512, respectively. (M3,4) indicates that the final result of linear classification has only 4 categories. The target classification categories are recyclables, kitchen waste, hazardous waste, and other waste.

[0029] Step 2: Train the self-constructed garbage classification image classification network N aFirst, a fixed-angle camera was used to record video of the daily garbage disposal process. Then, images of the video frames were extracted at a rate of 5 frames per second. Finally, the images of the video frames were cropped to obtain images with a size of 224×224. The garbage sorting dataset contains four categories: recyclables, kitchen waste, hazardous waste, and other waste. Each category contains 600 images. These datasets were then input into the classification network N. a Training is performed to obtain the training network N. b The network was trained with a learning rate of 0.1 for 160 epochs. At 80 and 120 epochs, the learning rate decayed by 10%, while the momentum was 0.9 and the decay weights were 0.0001. Stochastic gradient descent was then used for optimization.

[0030] Step 3: Automated pruning based on pruning indices generates a pruning network N. c .

[0031] Step 3.1 Calculate the training network N b Pruning indices for all channels. The calculation method for the pruning indices of this invention can be found in the appendix. Figure 3 The pruning process can be found in the appendix. Figure 5 60 random garbage images from each class are selected and input into the training network N. b Record the feature maps F of the ReLU outputs of all activation layers in the network. Calculate the pruning index γ for all channels, and the feature map F of the k-th layer. k ∈R B ×C×H×W The pruning index γ for all channels includes the following steps:

[0032] The first step is to process the feature map F of the k-th layer. k Summing the absolute values ​​along the channel direction.

[0033]

[0034] The second step is to sum the absolute values ​​of the feature map of the pre-pruned j-th channel along the channel direction.

[0035]

[0036] The third step is to combine the above two indicators A k and Normalization is performed to eliminate the influence of feature map size.

[0037]

[0038] The fourth step is to subtract the two normalized indices and calculate their L2 norm.

[0039]

[0040] Where R represents a real number, B represents the number of data batches, C represents the number of channels, H represents the feature map height, W represents the feature map width, and m k Represents the feature map F of the k-th layer k The number of channels, γ k,j Let ||·||2 represent the pruning index of the j-th channel of the k-th layer feature map, and let ||·||2 represent the L2 norm.

[0041] Step 3.2: Prune the network according to the pruning ratio μ = 0.75 and the pruning index. Sort the pruning indexes of all channels, and determine the pruning position r according to the sorting position and the pruning ratio μ. * Determine the pruning position r. * The steps are as follows:

[0042] Step 1: Calculate and train network N based on the pruning criterion γ and the feature map variable F. b The pruning index γ for all channels.

[0043] γ=[γ1,γ2,…,γ k ],

[0044] Where k = 13 represents the training network N b There are 16 convolutional layers.

[0045] Step 2: Based on the pruning index γ, sort the pruning indexes of each channel in descending order to obtain the sorted position r.

[0046] r = [r1, r2, ..., r k ],

[0047] in, Indicates the m-th element within the k-th layer. k The sorted positions of each channel.

[0048] Step 3: Statistically train network N b The number of central channels is determined, and the pruning position r is obtained based on the pruning ratio μ = 0.75. * .

[0049]

[0050] Where m k This represents the number of channels in the k-th convolutional layer.

[0051] Step 4: Determine the pruning position r based on the sorting position and the pruning ratio μ. * The sorting position γ of the j-th channel in the k-th layer of the feature map. k,j Greater than or equal to the pruning position r* If the channel is not selected, it should be retained; otherwise, it should be removed. This is to ensure the training of network N. b The module structure remains unchanged, and the pruned network N b At least one channel should be retained for each layer, and the channel with the highest pruning standard should be retained.

[0052] Step 4: Use knowledge distillation to restore the pruned network N c The decrease in accuracy is due to over-pruning. The knowledge distillation training process proposed in this invention is described in the appendix. Figure 6 .

[0053] Step 1: Determine the training network N b With pruning network N c Aligned Each module uses variables. and variables Record the feature maps output by these modules. Indicates training network N b The feature map output by the second activation layer. Represents a pruned network N c The feature map output by the 4th activation layer.

[0054] Step 2: Based on the feature map variable F t and F s Calculate the similarity index δ of the feature maps k .

[0055]

[0056] Where δ k Let |||||2 represent the feature map similarity index of the k-th layer; ||·||2 represents the L2 norm.

[0057] Step 3: Use a balanced pruning network N with hyperparameter α = 0.9 d With pruning network N b The loss is such that the hyperparameter β = 0.1 balances the relationship between pruning loss and similarity index. The knowledge distillation loss calculation method proposed in this invention can be found in the appendix. Figure 4 .

[0058]

[0059] CrossEntropy(p,q)=-∑ x∈X p(x)*log(q(x))

[0060] Where CrossEntropy represents cross-entropy, and p represents the pruned network N. d The output q represents the output of the pruning network N. bThe output of y represents the label of the dataset; CrossEntropy represents the cross-entropy, where p(x) represents the probability of x in the p distribution, q(x) represents the probability of x in the q distribution, X represents all possible values, and x represents a certain value.

[0061] Step 4: Based on the loss function and the garbage dataset, train the pruned network N using stochastic gradient descent. c That is, the target network N d .

Claims

1. A waste sorting image recognition method based on network pruning and knowledge distillation firstly constructs a directly connected waste sorting network using a convolutional network. Input the self-built garbage classification dataset into the classification network. Obtain the training network Next, calculate the pruning criteria for all channels in the network, and adjust the training network according to the magnitude of the pruning criteria. Perform channel pruning to obtain the pruned network. Finally, knowledge distillation was used to restore the pruned network. The decrease in accuracy due to over-pruning affects the acquisition of the target network. Video frames recorded by a camera at a fixed angle under normal lighting conditions are captured at regular intervals, cropped, and then input into the constructed target network. The system performs waste sorting and identification detection; based on the detection results, it provides an audible alarm for any sorting errors. Its features are, The training network The specific steps for pruning the passageway are as follows: Step 1: Randomly select 60 garbage images from each class and input them into the training network. Record all activation layers of the network Output feature map ; Step 2: Determine the pruning ratio ; Step 3: Calculate the pruning index for all channels ;No. Feature map of layer Pruning parameters for all channels Includes the following steps: The first step is to... Feature map of layer Summing the absolute values ​​along the channel direction; ； The second step is to remove the pre-pruned branches. Sum the absolute values ​​of the feature maps of each channel along the channel direction; ； The third step is to combine the above two indicators. and Normalization is performed to eliminate the influence of feature map size; ； The fourth step is to subtract the two normalized indicators and calculate their L2 norm. ； in Represent real numbers, Indicates the number of data batches. Indicates the number of channels. Indicates the feature map height. Indicates the width of the feature map. Indicates the first Layer feature map The number of channels, Indicates the first Layer feature map Pruning indicators for each channel Represents the L2 norm; Step 4: Train the network After pruning, a pruning network is obtained. ; Sort the pruning indices of all channels and prune them according to their ranking position and pruning ratio. Determine the pruning location ; Feature map Layer The sorting position of each channel Greater than or equal to the pruning position Keep the channel or cut it.

2. The method according to claim 1, characterized in that, The fourth step in the pruning process is to determine the sorting position and pruning ratio. Determine the pruning location Determine the pruning location. The steps are as follows: Step 1: According to the pruning standards and feature map variables Computational training network Pruning indices for all channels ; ； in Indicates training network It has 16 convolutional layers; Step 2: Based on the pruning indicators The pruning indices of each channel are sorted within each layer in descending order to obtain the sorted positions. ; ； in, Indicates the first The first layer The sorted positions of each channel; Step 3: Statistically train the network The number of central passages and the pruning ratio To obtain the pruning position ; ； in Indicates the first The number of channels in a convolutional layer.

3. The method according to claim 1, characterized in that, The fourth step in the pruning process is as follows: to ensure the training network The module structure remains unchanged, and the pruned network... At least one channel must be reserved for each layer.

4. The method according to claim 1, characterized in that, The aforementioned use of knowledge distillation to restore the pruned network The following steps address the decrease in accuracy caused by over-pruning: Step 1: Determine the training network With pruning network Aligned Each module uses variables. and variables Record the feature maps output by these modules; Indicates training network Feature map output by the second activation layer; Represents a pruned network The feature map output by the 4th activation layer; Step 2: Based on the feature map variables and Calculate the similarity index of feature maps ; ； in Indicates the first Similarity index of feature maps of layers; Represents the L2 norm; Step 3: Use hyperparameters Balanced pruning network With training network The loss of hyperparameters The relationship between balancing pruning losses and similarity indicators; ； ； ； in, Represents a pruned network The output, Indicates training network The output, Labels representing the dataset Denotes cross-entropy, where express In distribution The probability, express In distribution The probability, Represents all possible values, To represent a certain value; Step 4: Based on the loss function and the garbage dataset, train the pruned network using stochastic gradient descent. That is, the target network .

Images