A deep learning-based method for classifying junk images

Abstract

The application discloses a kind of based on deep learning's rubbish image classification method, this method specifically includes the following steps: step one, propose a kind of based on transfer learning's EfficientNet network, give an RGB image img as input, using pre-trained EfficientNet on ImageNet extraction main feature F a , step two, the feature F a Extracted is further input to CA module and spatial location information is important attention.Step three, the feature obtained in the previous step is input into MSCAM module for channel and scale information focus.Step five, the feature obtained in the previous step is sequentially passed through 1x1 convolution, global average pooling layer, fully connected layer and softmax activation function to obtain the final output of the network.In addition, more than 30,000 rubbish images are collected for network training, including more than 90 types of rubbish.In order to solve the long-tail distribution phenomenon existing in the data set used in training, Focal Loss loss function is used in experimental training to alleviate the long-tail distribution problem.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to an application of a garbage image classification method based on deep learning in actual life. BACKGROUND

[0002] With the continuous advancement of urbanization, the ecological environment is facing severe challenges, specifically manifested as a sharp increase in urban household waste production and saturation of waste plants. How to effectively manage waste has become a social topic. With the application of deep learning and artificial intelligence in various industries, it has become possible to use intelligent methods to solve the problem of waste classification. Using deep learning methods to identify garbage images and apply them to actual life has also become a research hotspot.

[0003] When Gary Thung and Mindy Yang proposed the TrashNet dataset, they used SVM and Faster R-CNN for training, and the final results were only 63.0% and 68.3%, which were not very ideal. Rahmi Arda Ara et al. trained Densenet121, DenseNet169, MobileNet, Xception-v4 network models using transfer learning on the TrashNet dataset, and selected appropriate optimizers and corresponding training strategies, finally obtaining accuracies of 95%, 95%, 84%, and 94%, respectively. However, these studies all train neural networks based on the TrashNet dataset, which contains very limited types of garbage images, only 6 types of garbage images. In order to solve the problem of training in more data, Dong Ziyuan et al. designed a GCNet network, which achieved an accuracy of 96% on the Huawei garbage dataset. Gao Ming et al. improved the EfficientNetB5 network model, which achieved an accuracy of 97.17% on the Huawei garbage dataset. However, these studies did not take into account the long-tail distribution phenomenon of the Huawei garbage dataset. SUMMARY

[0004] The purpose of the present application is to provide a garbage image classification method based on deep learning, which can extract features while paying attention to the spatial position information and scale information of objects in the image, and is very lightweight.

[0005] To solve this technical problem, the technical solution of the present application is: a garbage image classification method based on deep learning, comprising,

[0006] obtaining an image dataset captured by a camera module;

[0007] preprocessing the image dataset;

[0008] The image dataset is imported into a neural network model to obtain a first predicted data result, the first predicted data result is compared with a correct result to obtain a performance index, the neural network model is trained according to the performance index until the performance index approaches convergence, and the trained neural network model is obtained, the neural network model is used for classifying and judging the image dataset input therein, judging which category the image dataset belongs to, and the neural network model contains a judgment standard for which category an article in the image dataset belongs to;

[0009] According to the trained neural network model, a garbage image classification small program is built, actual garbage images are obtained, and the classification results of the actual garbage images are obtained by combining the garbage classification small program.

[0010] In an embodiment, the preprocessing includes:

[0011] The garbage image dataset is resized, randomly cropped, randomly rotated and flipped, and randomly erased, and the size of the image in the garbage image dataset is adjusted to a resolution of 224x224.

[0012] In an embodiment, the neural network model includes the following processing steps:

[0013] Step one, an EfficientNet network based on transfer learning is proposed, and an RGB image img is given as input, and the pre-trained EfficientNet on ImageNet is used to extract main features F a :

[0014] F a =E(img)

[0015] Wherein, E represents a pre-trained EfficientNet network;

[0016] Step two, the extracted feature F a is further input to the CA module for deeper feature extraction to obtain the feature F ca :

[0017]

[0018] Wherein, CA represents a CA module, represents cross multiplication operation;

[0019] Step three, the feature F ca obtained in step two is input to the MSCAM module to obtain more rich features to obtain the feature F mscam :

[0020]

[0021] wherein MSCAM represents the MSCAM module;

[0022] Step four, taking the feature F mscam as input, a 1x1 convolution is used to extract the last feature F b :

[0023] F b = Conv1(F mscam )

[0024] wherein Conv1 represents a convolution operation;

[0025] Step five, taking the last feature F b obtained in step four as input, a global average pooling, a fully connected layer and a softmax activation function are used in sequence to obtain the result F of the final neural network:

[0026] F = softmax(FC(GAP(F b ))).

[0027] In an embodiment, the CA module specifically works as follows:

[0028] Step one, performing a decomposition coding operation on the feature F a input to the CA module, and performing a pooling operation along the height and width directions of the feature map respectively to obtain features z h and z w ;

[0029] Step two, after the features z h and z w obtained in step one are spliced, a 1x1 convolution operation is performed to obtain a feature f:

[0030] f = Conv([z h , z w ])

[0031] wherein [] represents a splicing operation, and Conv represents a convolution operation;

[0032] Step three, features f h and f w are obtained by splitting the feature f according to the inverse operation of the splicing operation described in step two;

[0033] Step four, a convolution operation and a sigmoid activation function are respectively applied to the features f h and f w obtained in step three to obtain weight matrices g h and g w :

[0034] g h = σ(F h (f h ))

[0035] g w = σ(F w (f w ))

[0036] where F h and F w represent convolution operation, and σ represents sigmoid activation function;

[0037] Step five, cross-multiplication operation is performed on g h and g w obtained in step four to obtain the final output f ca of the CA module:

[0038]

[0039] In an embodiment, the MSCAM module specifically works as follows:

[0040] Step one, global average pooling operation and point convolution operation are respectively performed on the feature F ca in claim 3 to obtain features f gap and f pw :

[0041] f gap = GAP(F ca )

[0042] f pw = PWConv(F ca )

[0043] where GAP represents global average pooling, and PWConv represents point convolution;

[0044] Step two, point convolution is performed twice on the feature f gap in step one to obtain a feature f up , and point convolution is performed once on f pw in step one to obtain a feature f down :

[0045] f up = PWConv(PWConv(f gap ))

[0046] f down = PWConv(f pw )

[0047] Step three, point convolution is performed on f up and fdown The final output f of MSCAM is obtained by sigmoid function after broadcast addition operation mscam :

[0048]

[0049] wherein, sigma represents a sigmoid activation function, represents broadcast addition.

[0050] In an embodiment, the above neural network model is implemented on a PyTorch platform, and experiments are performed using an Nvidia RTX3090Ti graphics card.

[0051] In an embodiment, the network is trained using a Focal Loss loss function:

[0052] FocalLoss(p t )=-α t (1-p t )γlog(p t )

[0053]

[0054] The parameters are set as alpha = [0.96, 1.26, 1.05, 0.16] and gamma = 1.3.

[0055] In an embodiment, the neural network model of claim 3 is used to analyze the effectiveness of classification from the perspective of accuracy and F1 score:

[0056] The calculation formula of accuracy is:

[0057]

[0058] The calculation formula of F1 score is:

[0059]

[0060] Wherein, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

[0061] In an embodiment, the garbage classification app is used to judge and identify garbage images, when the garbage classification app judges the garbage category, the result is displayed on the page in the form of a pop-up window, and corresponding points are given according to the use of the app, and certain coupons are used to purchase specified goods by exchanging points.

[0062] The beneficial effects of the present application are:

[0063] The application provides a deep learning-based garbage image classification method, which pays more attention to spatial position information of objects in images and scale features of the objects and pays more attention to fine-grained features while extracting features.

[0064] A large number of experiments on a garbage data set show that the proposed neural network has better accuracy than other neural networks; the method has more significant advantages in accuracy.

[0065] Thus, the above-mentioned purposes of the application are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is the neural network of the application.

[0067] Figure 2 is a spatial attention CA module structure diagram in the application.

[0068] Figure 3 is a channel attention MSCAM module structure diagram in the application.

[0069] Figure 4 is a visual comparison diagram of the application and the prior art. DETAILED DESCRIPTION

[0070] In order to further explain the technical scheme of the application, the application will be described in detail below through specific embodiments.

[0071] Step one, an EfficientNet network based on transfer learning is proposed, a given RGB image img is taken as input, and a pre-trained EfficientNet on ImageNet is used to extract main features F a :

[0072] F a = E(img)

[0073] Wherein, E represents the pre-trained EfficientNet network.

[0074] Step two, the extracted features F a are further input to the CA module for deeper feature extraction to obtain features F ca :

[0075]

[0076] Wherein, CA represents the CA module, represents cross multiplication operation.

[0077] Step three, the features F ca obtained in step two are input to the MSCAM module to obtain more rich features to obtain features Fmscam :

[0078]

[0079] wherein MSCAM represents the MSCAM module.

[0080] Step four, taking the last feature F mscam as input, a 1x1 convolution is used to extract the last feature F b :

[0081] F b = Conv1(F mscam )

[0082] wherein Conv1 represents a convolution operation.

[0083] Step five, taking the last feature F b as input, a global average pooling, a fully connected layer and a softmax activation function are used in sequence to obtain the final neural network result F:

[0084] F = softmax(FC(GAP(F b )))

[0085] The CA module works in the following specific manner:

[0086] Step one, performing a decomposition coding operation on the feature F a input to the CA module, respectively along the height and width directions of the feature map to obtain features z h and z w .

[0087] Step two, after the features z h and z w obtained in step one are spliced, a 1x1 convolution operation is performed to obtain a feature f:

[0088] f = Cony([z h , z w ])

[0089] [] represents a splicing operation, and Conv represents a convolution operation.

[0090] Step three, features f h and f w are obtained by splitting the feature f according to the inverse operation of the splicing operation described in step two.

[0091] Step four, applying a convolution operation and a sigmoid activation function to the features f h and f w obtained in step three respectively to obtain a weight matrix g hand g w :

[0092] g h = σ(F h (f h ))

[0093] g w = σ(F w (f w ))

[0094] where F h and F w denote convolution operations, and σ denotes a sigmoid activation function.

[0095] Step five, cross-multiplication is performed on g h and g w obtained in step four to obtain the final output f ca of the CA module.

[0096]

[0097] The MSCAM module specifically works as follows:

[0098] Step one, global average pooling and point convolution are respectively performed on the feature F ca in claim 3 to obtain features f gap and f pw :

[0099] f gap = GAP(F ca )

[0100] f pw = PWConv(F ca )

[0101] where GAP denotes global average pooling, and PWConv denotes point convolution.

[0102] Step two, point convolution is performed twice on the feature f gap in step one to obtain a feature f up , and point convolution is performed once on f pw in step one to obtain a feature f down :

[0103] f up = PWConv(PWConv(f gap ))

[0104] f down = PWConv(f pw )

[0105] Step three, f up and f down After the broadcast addition operation, the final output f of MSCAM is obtained through the sigmoid function mscam :

[0106]

[0107] wherein, sigma represents a sigmoid activation function, represents broadcast addition.

[0108] In this embodiment, the data set and implementation details are performed in sequence, and the details are as follows:

[0109] (1) Data set

[0110] We selected the TrashNet garbage data set and the Huawei data set in combination, and added more than 50 kinds of garbage images, a total of 95 classes, 4 large categories, and more than 30,000 images. Because the data set has a long-tail distribution phenomenon, in order to solve this problem, the test set ensures that there are 400 images for each class, and the remaining images are used as the training set, and the images of the test set and the training set are mutually exclusive.

[0111] (2) Implementation details

[0112] During training, data augmentation methods such as random cropping, random rotation and flipping, random erasing, etc. are used, the initial learning rate is 0.0002, 50 iterations are trained, and the learning rate is decayed to 0.97 times of the original every 3 iterations, and the RAdam optimizer is used for training. In Focal Loss, alpha is set to [0.96, 1.26, 1.05, 0.16] according to the inverse of the sample frequency of each class, and gamma is set to 1.3. The experiment uses NVIDIA GeForce RTX 3090Ti for computing power support.

[0113] The effectiveness of the two components CA and MSCAM proposed in this embodiment is verified through ablation experiments. EfficientNet-CSM is the final neural network of the application, EfficientNet-CA represents only the CA module is added, EfficientNet-CSM represents only the MSCAM module is added, and EfficientNet represents the original network. From Table 1, it can be proved that CA and MSCAM improve the accuracy of the network to a certain extent, and the combination brings greater accuracy.

[0114] Table 1: Proof of effectiveness of CA and MSCAM

[0115] Network Model Accuracy (%) Parameter Amount (M) EfficientNet 91.68 5.3 EfficientNet-CA 92.31 5.32 EfficientNet-MSCAM 92.68 5.4 EfficientNet-CSM 93.5 5.42

[0116] The embodiment verifies that the neural network in the application has higher accuracy than other neural networks through a comparison experiment, and it can be proved from Table 2 that the neural network in the application has more significant accuracy.

[0117] Table 2: Comparison of experimental results of each model

[0118] Network Model Accuracy (%) Parameter Amount (M) ShuffleNet 88.67 3.5 MobileNetV3 90.92 5.4 MobileViT 92.01 5.6 GhostNet 89.88 5.1 EfficientNet 91.68 5.3 EfficientNet-CSM 93.5 5.42

[0119] In order to solve the long-tail distribution problem in the data set, a Focal Loss loss function is introduced for training, in order to prove the effectiveness of the loss function, the cross entropy loss function with the highest usage rate is compared, Table 3 A represents the cross entropy loss function, B represents the Focal Loss loss function, the angle analysis of the F1 score of each class, the Focal Loss can relieve the long-tail distribution problem better than the cross entropy loss function.

[0120] Table 3: Comparison of loss functions

[0121]

Claims

1. A deep learning-based garbage image classification method, characterized in that, The application relates to a garbage image classification method based on deep learning. An image dataset captured by a camera module is acquired; The image dataset is preprocessed; The image dataset is imported into a neural network model to obtain a primary predicted data result, the primary predicted data result is compared with a correct result to obtain a performance index, the neural network model is trained according to the performance index until the performance index tends to converge, and the trained neural network model is obtained, the neural network model is used for classifying and judging the image dataset input into the neural network model, the image dataset belongs to which category is judged, and the neural network model contains a judgment standard for belonging of an article in the image dataset to which category; A garbage classification applet is built according to the trained neural network model, actual garbage images are acquired, and the classification result of the actual garbage images is obtained in combination with the garbage classification applet; The neural network model comprises the following processing steps: Step one, given an RGB image img as input, extract features F using EfficientNet pre-trained on ImageNet a : F a = E(img) Wherein, E represents a pre-trained EfficientNet network; Step two, the extracted feature F a Further input to the CA module for extraction to get the feature F ca : wherein CA represents a CA module, represents a cross multiplication operation; Step three, the feature F obtained in step two is input to the MSCAM module to obtain the feature F ca Input to the MSCAM module to obtain the feature F mscam : Wherein, MSCAM represents an MSCAM module; Step four, the feature F macam As input, the last feature F is extracted by 1x1 convolution b : F b = Conv1(F mscam ) Wherein, Conv1 represents a convolution operation; Step five, the last feature F obtained from step four b As input, the global average pooling, fully connected layer, softmax activation function are adopted in turn to obtain the final neural network result F: F = softmax(FC(GAP(F b ))) ; The specific working mode of the MSCAM module is as follows: For feature F ca respectively, a global average pooling operation and a point convolution operation are performed to obtain feature f gap and f pw : f gap = GAP(F ca ) f pw = PWConv(F ca ) Wherein, GAP represents global average pooling, and PWConv represents point convolution; For the feature f gap Two point convolutions are done to obtain the feature f up For the feature f pw One point convolution is done to obtain the feature f down : f up = PWConv(PWConv(f gap )) f down = PWConv(f pw ) The f up and f down Broadcast addition operation is performed on the f mscam sigmoid function to obtain the final output f where σ denotes a sigmoid activation function, denotes a broadcast addition.

2. The method of claim 1, wherein the method is based on deep learning. The preprocessing comprises the following steps: The garbage image dataset is resized, randomly cropped, randomly rotated and flipped, and randomly erased, and the image size in the garbage image dataset is adjusted to 224*224 resolution.

3. The garbage image classification method based on deep learning according to claim 1, wherein, The specific working mode of the CA module is as follows: Step one, the features F input to the CA module a Decomposition coding operation is performed, and pooling operation is performed along the height and width directions of the feature map respectively to obtain features z h and z w ; Step two, concatenate the feature z h and z w After concatenation, a 1x1 convolution operation is applied to get the feature f: f = Conv([z h ,z w ]) Wherein, [] represents a splicing operation, and Conv represents a convolution operation; Step three, split the feature f by the inverse operation of the concatenation operation described in step two to get feature f h and feature f w ; Step four, the feature f obtained in step three is applied to a convolution operation and a sigmoid activation function to obtain a weight matrix g h and the feature f w respectively to obtain a weight matrix g h and g w : g h = σ (F h (f h )) g w = σ (F w (f w )) where F h and F w denotes a convolution operation, and σ denotes a sigmoid activation function. Step five, the g h and g w cross-multiplication operation to obtain the final output f of the CA module ca :

4. The method of claim 1, wherein the method is based on deep learning. The neural network model is realized on a PyTorch platform and is experimented by using an Nvidia RTX3090Ti graphics card.

5. The method of claim 1, wherein the method is based on deep learning. The neural network model is analyzed for classification effectiveness by using accuracy and F1 score: The calculation formula of the accuracy is as follows: Wherein, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

6. The method of claim 1, wherein the method is based on deep learning. The garbage classification applet is used for judging and identifying garbage images, when the garbage classification applet judges the garbage category, the result is displayed on a page in the form of a pop-up window, and corresponding points are given according to the use of the applet, and certain coupons are used for purchasing specified goods by using the points.

Images