Garbage classification collection and transportation supervision method based on improved YOLOv3 network

Abstract

The present invention discloses a method for monitoring garbage classification collection and transportation based on an improved YOLOv3 network, including collecting videos of garbage collection and transportation operations and uploading them to a host computer, converting video files into image sequences, scaling all images to the same size and sequentially inputting them into a garbage classification collection and transportation supervision network for detection, and then outputting pictures with garbage bin position marking frames, garbage bin states, and the probability of the states in the host computer; the garbage classification collection and transportation supervision network replaces all standard 3×3 convolutions in the YOLOv3 network with deep separable convolutions, and adds a triple attention module to the output ends of three scales on the backbone network. The present invention uses deep separable convolutions to reduce the number of network parameters and the amount of calculation, improves the detection speed of the model, and introduces Triplet Attention to solve the problem that the YOLOv3 network has poor position judgment accuracy and poor performance in detecting relatively dense targets.

Description

Technical Field

[0001] The present invention relates to the field of garbage classification supervision, and in particular to a garbage classification collection and transportation supervision method based on an improved YOLOv3 network. Background Art

[0002] With the development of the economy, the advancement of urbanization and the improvement of living standards, the amount of garbage produced has increased rapidly, which has increased the pressure on the urban garbage collection, transportation and treatment systems. Garbage classification refers to a series of activities that classify garbage for storage, classified placement and classified transportation according to certain regulations or standards so that it becomes a public resource. Garbage classification can be divided into three steps: front-end classification, mid-end collection and transportation, and back-end recycling. After visiting a certain city's environmental sanitation and domestic solid waste disposal guarantee center, it was learned that the mixing rate during garbage collection and transportation is extremely high, and there is a phenomenon of "front-end classification and back-end mixing". Mixed collection and transportation will dampen the enthusiasm of citizens for garbage classification, increase the difficulty of subsequent treatment, and at the same time reduce the effect of garbage classification.

[0003] At present, the main problems in the garbage collection process are as follows: (1) The number of garbage bins is excessive during loading. Generally, only two garbage bins can be hung on the garbage collection truck at a time and then the garbage in the bins can be poured into the collection truck. This is to ensure that the garbage in the garbage bins does not leak onto the ground and to ensure the cleanliness of the work site; secondly, it is to ensure the safety of the collection workers to avoid potential safety hazards. However, in actual work sites, operators often mount three garbage bins in order to speed up the process, which is illegal. (2) The lid of the garbage bin is not fully opened. During the garbage collection operation, in order to ensure that the garbage in the bin can be cleaned, the lid of the garbage bin is required to be opened before the operation. However, in actual work sites, operators often do not manually open the lid of the garbage bin to save trouble, thereby reducing the efficiency of garbage collection. (3) The color of the garbage bin (corresponding to the garbage category) does not match the garbage category collected by the garbage collection truck. In order to achieve classified garbage collection, different types of garbage after classification will be loaded into garbage bins of corresponding colors and transported by different garbage collection trucks. This requires that the color of the garbage bin (i.e. the type of garbage) must match the garbage collection truck. In the actual collection and transportation process, mismatched garbage bins are often mistakenly transported, resulting in mixed loading and transportation of garbage, which reduces the effectiveness of garbage sorting. (4) Obvious surface garbage is collected and transported without being placed correctly. Different types of garbage are packed in garbage bags of corresponding colors. Whether some garbage is placed correctly can be determined by the color of the surface garbage bags. If some garbage bags that are placed incorrectly are not manually sorted and directly loaded into the collection truck, it will also cause mixed loading and transportation of garbage.

[0004] The removal process is affected by factors such as the operating environment and time, making it difficult to supervise. At present, it is difficult to achieve effective real-time supervision with the existing conditions and facilities. If it relies on manual monitoring, it is difficult to form effective real-time supervision during special operating times due to its limitations and subjectivity. Using industrial cameras on garbage trucks to collect garbage removal images in real time, and applying image processing and other technologies for analysis, using computers instead of manual real-time monitoring can effectively improve the efficiency of supervision during the removal process, effectively reduce the occurrence of irregular removal and mixed garbage, and improve the garbage classification effect from the middle end of garbage classification, which can meet the real-time and effectiveness of detection. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a garbage classification collection and transportation supervision method based on an improved YOLOv3 network, which is used to simultaneously complete the position marking and status identification of garbage bins during garbage collection and transportation.

[0006] In order to solve the above technical problems, the present invention provides a method for supervising garbage classification collection and transportation based on an improved YOLOv3 network, comprising: collecting videos of garbage collection and transportation operations and uploading them to a host computer, converting the video files into image sequences in the host computer, scaling all images to 416×416 pixels and then sequentially inputting them into a garbage classification collection and transportation supervision network (improved YOLOv3 network) for detection, and then outputting pictures with a garbage bin position mark box, a garbage bin state, and the probability of the state in the host computer;

[0007] The garbage classification collection and transportation supervision network replaces all standard 3×3 convolutions in the YOLOv3 network with depthwise separable convolutions, and adds a triple attention module to the output ends of the three scales of 52×52, 26×26 and 13×13 on the backbone network of the YOLOv3 network. The feature maps of 52×52 and 26×26 are respectively passed through the triple attention module and used as the input of the feature extraction and detection network of the next scale, while the feature map of 13×13 is directly input into the detection network after passing through the triple attention module.

[0008] As an improvement of the garbage classification collection and transportation supervision method based on the improved YOLOv3 network of the present invention:

[0009] The training and testing process of the garbage classification collection and transportation supervision network is as follows: construct training sets, validation sets and test sets. In the training phase, the total number of training rounds is set to 100, and the batch size in the training is The size is 16, the initial learning rate and the final learning rate are 0.001 and 0.00001 respectively, the learning rate decreases proportionally at 80 epochs and 90 epochs, the momentum is 0.9, the weight decay is 0.0002, and multi-scale training is set; then the training set is input into the garbage classification collection and transportation supervision network in groups of 16 photos, the loss value is calculated using the loss function, and the gradient descent method is used to implement back propagation so that the model parameters are continuously iterated in the direction of minimizing the loss function value; after each round of training, the validation set pictures are also input into the garbage classification collection and transportation supervision network in groups of 16 photos for testing to measure the training status of the model. After 100 rounds of training, the trained garbage classification collection and transportation supervision network is obtained; in the testing stage, the test set is input into the trained garbage classification collection and transportation supervision network to obtain the garbage bin position marking box, garbage bin state and state probability in each image of the test set, and then the recall rate, precision rate and average precision mean MAP@0.5 of the model are statistically analyzed to measure the generalization ability of the model, thereby obtaining the garbage classification collection and transportation supervision network that can be used online.

[0010] As a further improvement of the garbage classification collection and transportation supervision method based on the improved YOLOv3 network of the present invention:

[0011] The process of constructing the training set, validation set and test set is:

[0012] Videos of garbage collection and transportation operations are collected, and key frames in the videos are extracted as original images. The original images are manually annotated with the status of the trash cans. Each original image is uniformly scaled to 416×416 pixels and subjected to data augmentation and expansion. 40% of the images after data augmentation and expansion are randomly selected for weather simulation processing. The images after weather simulation processing and the images without weather simulation processing are combined and divided into the training set, validation set and test set in a ratio of 7:2:1.

[0013] As a further improvement of the garbage classification collection and transportation supervision method based on the improved YOLOv3 network of the present invention:

[0014] The data enhancement and expansion processing includes Gaussian blur, horizontal flip, GridMask, brightness adjustment and random rotation of -20° to +20°;

[0015] The weather simulation process includes simulating four weather types: daytime rain, daytime fog, nighttime fog and nighttime rain;

[0016] The trash can states include four states: full with lid open, not fully overflowing, full with lid closed, and not fully overflowing with lid closed.

[0017] As a further improvement of the garbage classification collection and transportation supervision method based on the improved YOLOv3 network of the present invention:

[0018] The loss function is L = L box +L cls +L obj

[0019] Positioning loss function L box for:

[0020] Confidence error L obj for:

[0021]

[0022] Among them, λ noobj refers to the penalty coefficient when there is no target, λ noobj =0.5; C i Refers to the confidence of the true box of the i-th grid; Refers to the confidence of the prediction box of the i-th grid; Indicates whether there is a target in the jth bounding box of the i-th grid. If not, it is 1, otherwise it is 0;

[0023] L cls refers to the classification error and is defined as:

[0024]

[0025] Among them, p i (c) refers to the conditional probability that the real box of the i-th grid contains a c-type trash can, Refers to the conditional probability that the prediction box of the i-th grid contains a c-type trash can;

[0026] Among them, the confidence error and classification error both use the two-class cross entropy loss function

[0027]

[0028] Among them, m i is the sample label. If the sample is a positive example, it is 1, otherwise it is 0. Predict the probability that the sample is a positive example for the model.

[0029] The beneficial effects of the present invention are mainly reflected in:

[0030] 1. The present invention replaces the standard convolution in the YOLOv3 network with a depth-separable convolution, which reduces the number of parameters and the amount of calculation of the YOLOv3 network and improves the detection speed of the model;

[0031] 2. The present invention introduces Triplet Attention into the backbone network of the YOLOv3 network, automatically learns cross-dimensional interactions without increasing parameters, improves effective feature channel weights, and enhances feature extraction capabilities, solving the problem of poor position judgment accuracy of the YOLOv3 network and poor performance in detecting densely packed targets;

[0032] 3. Ablation experiments show that the model performance of the present invention has been significantly improved; in addition, compared with the Faster-CNN network and MobileNetSSDv2 network, the average accuracy is higher, which meets the requirements of real-time supervision and environmental complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0034] Figure 1 It is a schematic diagram of the YOLOv3 network model;

[0035] Figure 2 To show the sample of the garbage classification collection and transportation supervision data set established;

[0036] Figure 3 A schematic diagram of the garbage classification collection and transportation supervision network of the present invention;

[0037] Figure 4 Schematic diagram of the Triplet Attention mechanism module;

[0038] Figure 5 Schematic diagram of depth-wise separable convolution;

[0039] Figure 6 It is a schematic diagram of the detection results of the garbage classification collection and transportation supervision network of the present invention;

[0040] Figure 7 It is the curve of each indicator changing with the number of iterations during the training process;

[0041] Figure 8 This is a schematic diagram of the structure of the YOLO-tiny network. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto:

[0043] Embodiment 1: A method for monitoring garbage classification, collection and transportation based on an improved YOLOv3 network, including the following process:

[0044] Step 1: Create a garbage classification collection and transportation dataset

[0045] The dataset is derived from videos shot by industrial cameras mounted on garbage trucks in a certain city. Key frames in the videos (i.e., video frames containing garbage bins) are extracted as original images to construct the dataset. There are 4,147 original images in total.

[0046] The original image is manually labeled with the trash can status using LabelImg software to obtain the corresponding label file. The trash can status is divided into four categories: open-full, open-empty, closed-full, and closed-empty. Figure 2 As shown in the figure, since there are multiple trash cans in each image, and each trash can has its own status, each image may contain one or more annotations of the trash can status. The 4147 original images contain a total of 8840 annotations, and then each original image is uniformly scaled to 416×416 pixels.

[0047] However, in the original data set, the number of trash can status images of each category is unevenly distributed, which will affect the training process. Therefore, the present invention uses methods such as Gaussian blur, horizontal flip, GridMask, brightness adjustment and random rotation (-20°~+20°) to perform data enhancement and expansion processing on the scaled 416×416 image. For images with multiple annotations in the same image, when performing data enhancement, in order not to increase the number of annotations of other categories in this process, all non-annotated categories are blocked with black frames. For images with a small number of category samples, two or more data enhancement methods are used at the same time, such as horizontally flipping the Gaussian blurred image. After data enhancement and expansion processing, the number of trash can status images of each category tends to be balanced, including 4288 images with open lids overflowing, 3414 images with open lids not overflowing, 3116 images with closed lids overflowing, and 4254 images with closed lids not overflowing.

[0048] In order to meet the requirements of environmental complexity recognition, OpenCV software is used to simulate weather conditions such as daytime rain, daytime fog, nighttime fog, and nighttime rain, and weather simulation is performed on the images after data augmentation and expansion. 40% of the samples in the images after data augmentation and expansion are randomly selected, that is, 3052 images are randomly simulated as images under four weather types: daytime rain, daytime fog, nighttime fog, and nighttime rain. Each of the four weather types accounts for 10%, that is, 763 images.

[0049] Finally, the 3052 images processed by weather simulation and the 4578 images without weather simulation (i.e. the remaining 60% of the images after data augmentation and expansion) were divided into training set, validation set and test set in a ratio of 7:2:1, as shown in Table 1.

[0050] Table 1: Number of training sets, validation sets, and test sets

[0051] Training set Validation set Test Set total 5345 1527 758 7630

[0052] Step 2: Improve the YOLOv3 network to build a garbage classification and collection supervision network

[0053] The garbage classification collection and transportation supervision network of the present invention is built based on the YOLOv3 network. By improving the backbone network, reducing the number of model parameters, extracting multi-scale features, and improving network performance, the details are as follows:

[0054] Step 2.1: Build a basic YOLOv3 network

[0055] The YOLOv3 network is a target detection network that can identify the target category and location in the image. In the process of garbage collection and transportation supervision, this network can be used to realize the location marking of garbage bins and the detection of garbage bin status.

[0056] The YOLOv3 model structure consists of two parts: classification network and detection network. Figure 1 As shown in the figure, the classification network is the backbone network in the model. The Darknet53 network composed of a residual network outputs three feature maps of 52×52, 26×26 and 13×13 sizes as the input of the detection network, which can extract richer and deeper features and classify the image more accurately. The detection network uses a feature pyramid for fine-grained feature detection, detects the three input feature maps at three scales, and outputs multi-scale detection results. The YOLOv3 network outputs three feature maps in total. The first feature map is downsampled 32 times, the second feature map is downsampled 16 times, and the third feature map is downsampled 8 times, which are used to detect targets of different scales.

[0057] DBL is the smallest component in the YOLOv3 network, which consists of three parts: convolution, batch normalization and leaky ReLU activation function; res unit draws on the residual structure in the Resnet network to make the network deeper; resn is composed of a DBL component and n res units, which is a large component in YOLOv3, in which each DBL plays a role of downsampling.

[0058] The YOLOv3 network also includes Concat, which is tensor concatenation, which expands the dimensions of two tensors; add, which is tensor addition, where tensors are added directly without expanding the dimensions.

[0059] Step 2.2: Improve the basic YOLOv3 network to build a garbage classification and collection supervision network

[0060] The garbage classification collection and transportation supervision network (i.e. the improved YOLOv3 network) replaces all standard 3×3 convolutions in the YOLOv3 network with depthwise separable convolutions, thereby reducing the number of model parameters and improving the detection speed. At the same time, a triplet attention module is added after the three different scale outputs of the backbone network to improve the model recognition accuracy. Figure 3 shown.

[0061] Step 2.2.1: Application of Depthwise Separable Convolution

[0062] Depthwise separable convolution splits the standard convolution into point-by-point convolution of depthwise convolution. Standard convolution is to perform convolution operation on each channel of input data with a specific convolution kernel, and add the convolution results of each channel. Depthwise convolution splits the convolution kernel into single channels, and performs convolution operation on each channel, so as to obtain a feature map with the same number of channels as the input feature map. Then perform point-by-point convolution on it to achieve dimensionality increase and dimensionality reduction of the output feature map. Depthwise separable convolution can greatly reduce model parameters and reduce computational complexity. The calculation process of depthwise separable convolution and standard convolution is as follows: Figure 5 shown.

[0063] Let the convolution kernel be D k ×D k ×M, the number of convolution kernels is N. The number of parameters of standard convolution is P std :

[0064] P std =D k ×D k ×M×N

[0065] Among them, D k is the size of the convolution kernel, and M is the number of convolution kernel channels;

[0066] The size of the output feature map is D w ×D h , each convolution kernel must be D w ×D h times, then the amount of computation for standard convolution is O std :

[0067] O std =D k ×D k ×M×N×Dw ×D h

[0068] The number of parameters P of depthwise separable convolution dsc :

[0069] P dsc =D k ×D k ×M×N+M×N

[0070] The computational cost of depthwise separable convolution is O dsc :

[0071] O dsc =D k ×D k ×M×D w ×D h +M×N×D w ×D h

[0072] Therefore, replacing the standard convolution with a depth-separable convolution will theoretically reduce the number of parameters:

[0073]

[0074] The amount of calculation is theoretically reduced:

[0075]

[0076] In summary, replacing the 3×3 standard convolution in the basic YOLOv3 network with a depthwise separable convolution will reduce the amount of computation and parameters to about 1 / 9 of the original. Depthwise separable convolution greatly reduces the amount of parameters and computation, thereby improving the detection rate.

[0077] Step 2.2.2: Application of triple attention module

[0078] The structure of the Triplet Attention module is as follows Figure 4 As shown in the figure, compared with other attention mechanisms, the triplet attention module has the following advantages: it can retain spatial and channel information, avoiding information loss; it can interact across dimensions, eliminating the indirect correspondence between channels and weights; the computational overhead can be ignored, and the number of parameters is almost not increased.

[0079] The Triplet Attention module mainly consists of three branches. The first branch captures the interaction between channel C and spatial H dimensions. The input features are arranged to become W×H×C dimensional features, and then rearranged to C×W×H dimensional features after Z-Pool, 7×7 convolution and Sigmoid activation function in H dimension. The second branch is similar to the first branch and is used to capture the interaction between channel C and spatial W dimension. The third branch is used to calculate channel attention. The input features are processed by Z-Pool, convolution and Sigmoid activation function to generate spatial attention weights. Finally, the output features of the three branches are added and averaged. Two branches are used to capture the cross-channel interaction between channel C dimension and spatial dimension W and H respectively, and the remaining branch is the traditional calculation of spatial attention weights.

[0080] The present invention adds a triple attention module to the output ends of the three scales of 52×52, 26×26 and 13×13 on the YOLOv3 backbone network (i.e., the Darknet53 network), respectively. The feature maps of 52×52 and 26×26 are respectively used as the input of the feature extraction and detection network of the next scale after passing through the triple attention module, and the feature map of 13×13 is directly input into the detection network after passing through the triple attention module, that is, the 52×52 feature map for small target detection output by the Darknet-53 network is input into the resn module for medium-scale feature extraction after passing through the first triple attention module, and the feature map for larger target detection is extracted, and is used as the input of the medium tensor fusion (contact) for small-scale target detection; the 26×26 feature map for medium-scale target detection output by the Darknet-53 network is output through the second triple attention module. The output of the Darknet-53 network after the Triplet Attention module is input into the resn module for large-scale feature extraction and is also used as the input of the tensor fusion (Contact) for medium-scale object detection. The 13×13 feature map output by the Darknet-53 network for large-scale object detection is directly input into the detection network after passing through the third Triplet Attention module.

[0081] Triplet Attention can extract semantic dependencies between different dimensions, eliminate the indirect correspondence between channels and weights, and achieve the effect of improving accuracy with very little computational overhead without reducing the dimension. It automatically learns cross-dimensional interactions without increasing parameters, improves effective feature channel weights, and enhances feature extraction capabilities, thereby improving the shortcomings of traditional YOLOv3, such as poor position accuracy and poor performance in detecting densely packed targets.

[0082] Step 3: Model training and testing

[0083] Step 3.1 Establish the loss function

[0084] The loss function of the basic YOLOv3 network constructed in step 3.1 is as follows:

[0085] L=L box +L cls +L obj

[0086] L box refers to the positioning loss function, which is defined as:

[0087]

[0088] In the formula, S 2 refers to the number of grids (S′S); B refers to the number of bounding boxes of each grid; Indicates whether there is a target in the jth bounding box of the i-th grid. If yes, it is 1, otherwise it is 0;

[0089]

[0090] The IoU regression effect used in YOLOv3 is not obvious enough, and only focuses on the overlapping area between the predicted box and the real box. In some cases, it is difficult to express the effect of the frame regression. Therefore, the garbage classification collection and transportation supervision network (improved YOLOv3 model) of the present invention uses GIoU (Generalized Intersection over Union) instead of the original IoU to measure the size of the coordinate error, which is defined as:

[0091]

[0092] Where A c It refers to the minimum area of ​​the region that contains both the predicted and true bounding boxes, U refers to the intersection area of ​​the predicted and true bounding boxes, and IoU (Intersection Over Union) refers to the intersection ratio of the predicted and true bounding boxes.

[0093] Compared with IoU, GIoU has a wider range. IoU is between 0 and 1, while GIoU has a symmetrical range between -1 and 1, which can better measure the gap between non-overlapping frames and focus on the non-overlapping areas of the two. Therefore, it has stronger encoding capabilities. GIOU replaces the original IOU algorithm and can improve the learning ability of the network. The improved positioning loss function is defined as follows:

[0094]

[0095] L obj refers to the confidence error, which is defined as:

[0096]

[0097] In the formula, λ noobj Refers to the penalty coefficient when there is no target, where λ noobj =0.5; C i Refers to the confidence of the true box of the i-th grid; Refers to the confidence of the prediction box of the i-th grid; Indicates whether there is a target in the jth bounding box of the ith grid. If not, it is 1, otherwise it is 0.

[0098] L cls refers to the classification error and is defined as:

[0099]

[0100] In the formula, p i (c) refers to the conditional probability that the real box of the i-th grid contains a c-type trash can, Refers to the conditional probability that the prediction box of the i-th grid contains a c-type trash can.

[0101] The confidence error and classification error both use the binary cross entropy loss function The definition is as follows:

[0102]

[0103] Among them, m i is the sample label. If the sample is a positive example, it is 1, otherwise it is 0. Predict the probability that the sample is a positive example for the model.

[0104] Step 3.2 Training process

[0105] The training set and validation set are input into the garbage classification collection and transportation supervision network (i.e. the improved YOLOv3 network).

[0106] During the training phase, the total number of training rounds was set to 100, and the batch size during training was 16; the initial and final learning rates were 0.001 and 0.00001, and the learning rate was proportionally reduced at 80 epochs and 90 epochs; the momentum was 0.9, the weight decay was 0.0002, and multi-scale training was set.

[0107] After setting the above hyperparameters, the garbage classification collection and transportation supervision network (i.e., the improved YOLOv3 network) is trained. A total of 100 rounds of training are performed. During the training process, the training set is read into the improved YOLOv3 network in the form of 16 photos as a group and the calculation is performed in the order of the pictures. Each picture may be used twice or more. During the training process, the loss value is calculated according to the loss function established in step 3.1, and the gradient descent method is used to implement back propagation so that the model parameters are continuously iterated in the direction of minimizing the loss function value. After each round of training, the validation set pictures are also tested in groups of 16 pictures to measure the training status of the model, such as determining whether the model is overfitting. After 100 rounds of training, the trained garbage classification collection and transportation supervision network (i.e., the improved YOLOv3 network) is obtained.

[0108] Step 3.3 Testing process

[0109] The test set is input into the trained garbage classification collection and transportation supervision network (i.e. the improved YOLOv3 network) to obtain the garbage bin status (and the probability of each status) and the box with the garbage bin location mark in the test set image. Then the recall rate, precision rate and average precision mean MAP@0.5 of the model are statistically analyzed to measure the generalization ability of the model.

[0110] (1) Precision refers to the proportion of positive samples in the positive examples determined by the classifier.

[0111]

[0112] (2) Recall refers to the proportion of samples predicted as positive to the total positive samples.

[0113]

[0114] Among them, TP refers to the number of positive examples predicted correctly, FP refers to the number of positive examples predicted incorrectly, and FN refers to the number of negative examples predicted incorrectly.

[0115] (3) mAP is the arithmetic mean of APs of all categories. AP is the average precision, which is the area under the PR curve for each category. mAP@0.5 means that if IoU ≥ 0.5, the prediction is correct.

[0116] After testing, the trained model has a detection speed of 24.19ms / it and a mAP of 95.3%, which meets the requirements of real-time detection and high accuracy. After saving the trained model, a garbage classification collection and transportation supervision network (i.e., the improved YOLOv3 network) that can be used online is obtained. Some test results are shown in the following example: Figure 6 As shown, Figure 6There are two trash cans, one of which is classified as not full when opened, with a corresponding probability of 72%, and the other is classified as full when opened, with a corresponding probability of 98%.

[0117] Step 4: Use the online garbage classification collection and transportation supervision network

[0118] In actual use, the online garbage classification and collection supervision network obtained in step 3.3 is converted into ONNX format, and the model can be integrated into the required computer or system by calling C++ and OpenCV libraries.

[0119] In actual production, the industrial camera on the garbage truck shoots the video during operation and uploads it to the host computer, where the video file is converted into an image sequence and all images are scaled to 416×416 pixels to increase the recognition speed. The 416×416 images are then input into the garbage classification collection and transportation supervision network for detection. In actual use, no manual sorting is required. Whether there are garbage bins in the image or not, they can be input into the garbage classification collection and transportation supervision network. After detection, the garbage bin position mark and garbage bin status in the image (including one of the four states of open lid overflowing, open lid not overflowing, closed lid overflowing, and closed lid not overflowing and the probability of each state) can be obtained, and the results are stored and displayed.

[0120] experiment:

[0121] 1. Experimental environment

[0122] The hardware environment of this experiment is as follows: the processor is Intel(R) Core(TM) i5-9300H CPU@2.40GHz, the memory is 8GB, the graphics card is Nvidia GTX 1650, the operating system is 64-bit, and the disk size is 1T.

[0123] The software environment is: operating system is Windows 10, deep learning framework is Pytorch 1.10.1, development environment is PyCharm, development language is Python, package manager and environment manager are Anaconda, and GPU acceleration libraries are CUDA 11.3 and cuDNN 8.2.1.

[0124] 2. The evaluation parameters of the results in the experiment include: mAP, detection speed, number of parameters, precision, recall and F1 value:

[0125] The F1 value (F1-score) is calculated based on the precision and recall rate, and its formula is as follows:

[0126]

[0127] The detection speed refers to the average detection speed of a single photo. Several photos are input into the network for detection, the total detection time is calculated, and finally the average detection time required for a single photo is calculated.

[0128] The number of parameters is calculated by the program when training and loading the network structure, which refers to the number of parameters in the network.

[0129] The data sets used in the experiment all adopt the training set, validation set and test set shown in Table 1 in Example 1, totaling 7630 images.

[0130] 3. Ablation experiment

[0131] Ablation experiments refer to comparative experiments with the garbage classification collection and transportation supervision network of the present invention and YOLOv3-TA, YOLOv3-DSC and YOLOv3. YOLOv3-TA is a network based on YOLOv3. According to step 2.2.2 of Example 1, a triple attention (Triplet Attention) module is added after the three scale outputs of the YOLOv3 backbone network respectively. YOLOv3-DSC is a network based on YOLOv3. According to step 2.2.1 of Example 1, the 3×3 standard convolutions in the backbone network are replaced with depthwise separable convolutions. The results of the ablation experiment are shown in Table 2. During the training process, the curves of Train_loss and mAP of YOLOv3-TA, YOLOv3-DSC and YOLOv3 as the number of network iterations are shown in the following table. Figure 7 shown.

[0132] Table 2 Ablation experiment test results

[0133]

[0134] *TA stands for Triplet Attention mechanism

[0135] *DSC stands for Depth-wise Separable Convolution, deep separable convolution

[0136] Through ablation experiments, it can be found that the garbage classification collection and transportation supervision network is superior to the original YOLOv3 in terms of speed, number of parameters and mAP. After YOLOv3 was reconstructed into YOLOv3-DSC using deep separable convolution, the number of model parameters was greatly reduced from the original 6.154×10 7 is reduced to 1.770×10 7 At the same time, the detection speed has also been greatly improved, and the detection time for each image has been shortened by 49.05% compared with the original.

[0137] Adding the Triplet Attention mechanism on the basis of the depthwise separable convolution reduces the speed from 23.27ms / it to 24.19ms / it, but can increase mAP from 92.1% to 95.3%, an increase of 3.2%, but the number of parameters only increases by 0.96%, or 0.017×10 7 The experimental results show that adding Triplet Attention to the YOLOv3 backbone network can significantly improve the performance of the model without increasing parameters and sacrificing a small amount of speed. The above ablation experiment proves the effectiveness of the two improved schemes on this dataset.

[0138] 4. Comparative experiment

[0139] The improved YOLOv3 network is compared with other networks. The networks involved in the comparative experiment in the present invention are YOLOv3, YOLOv3-SSP, YOLO-tiny, MobileNetSSDv2 and Fast-CNN, and their definitions and descriptions are as follows:

[0140] (1) YOLOv3-SSP refers to the YOLOv3 network with a spatial pyramid pooling structure (SSP) added to the end of the YOLOv3 Darknet-53 network.

[0141] (2) YOLO-tiny refers to a lightweight YOLO network with a shallower depth and only two feature scale predictions, such as Figure 8 shown.

[0142] (3) MobileNetSSDv2 refers to the network structure proposed in the paper "Inverted Residuals and LinearBottlenecks:Mobile Networks for Classification, Detection and Segmentation".

[0143] (4)Fast-CNN refers to the network structure proposed by Ross Girshick in his paper "Fast R-CNN".

[0144] The comparative experimental results are shown in Table 3 and Table 4.

[0145] Table 3 Comparative experiment (YOLO series)

[0146]

[0147] Table 4 Comparative experiments (other mainstream networks)

[0148]

[0149]

[0150] It can be seen from Table 3 and Table 4 that the network proposed in the present invention has a high accuracy rate, and the mAP can reach 95.3%. At the same time, the detection speed is fast, which can reach 27.38ms / it, and the model size is greatly reduced. The above experiments prove the excellent performance of the garbage classification collection and transportation supervision method based on the improved YOLOv3 network proposed in the present invention.

[0151] Finally, it should be noted that the above examples are only some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there are many variations. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A method for monitoring garbage classification, collection and transportation based on an improved YOLOv3 network, characterized in that : Collect videos of garbage collection and transportation operations and upload them to the host computer. Convert the video files into image sequences in the host computer, scale all images to 416×416 pixels, and then input them into the garbage classification and collection supervision network for detection. Then output images with garbage bin location mark boxes, garbage bin status, and status probability in the host computer; The garbage classification collection and transportation supervision network replaces all standard 3×3 convolutions in the YOLOv3 network with depthwise separable convolutions, and adds a triple attention module to the output ends of the three scales of 52×52, 26×26 and 13×13 on the backbone network of the YOLOv3 network. The feature maps of 52×52 and 26×26 are respectively passed through the triple attention module and used as the input of the feature extraction and detection network of the next scale, while the feature map of 13×13 is directly input into the detection network after passing through the triple attention module; The depthwise separable convolution splits the standard convolution into point-by-point convolution of the depthwise convolution; the standard convolution is to perform a convolution operation on each channel of the input data with a specific convolution kernel, and add the convolution results of each channel; the depthwise convolution splits the convolution kernel into a single channel form, performs a convolution operation on each channel, thereby obtaining a feature map with the same number of channels as the input feature map, and then performs point-by-point convolution on it to achieve dimensionality increase and dimensionality reduction of the output feature map; The triple attention module mainly includes three branches. The first branch captures the interaction between channel C and spatial H dimensions. The input features are arranged to become W×H×C dimensional features, and then rearranged to C×W×H dimensional features after Z-Pool, 7×7 convolution and Sigmoid activation function operations on the H dimension; the second branch is similar to the first branch and is used to capture the interaction between channel C and spatial W dimensions; the third branch is used to calculate channel attention, and the input features are subjected to Z-Pool, convolution and Sigmoid activation function to generate spatial attention weights; finally, the output features of the three branches are added and averaged, two of which are used to capture the cross-channel interaction between channel C dimension and spatial dimension W and H, and the remaining branch is the traditional calculation of spatial attention weights.

2. According to claim 1, a method for monitoring garbage classification, collection and transportation based on an improved YOLOv3 network is characterized in that: The training and testing process of the garbage classification collection and transportation supervision network is as follows: construct a training set, a validation set and a test set. In the training stage, the total number of training rounds is set to 100, the batch size in the training is 16, the initial learning rate and the final learning rate are 0.001 and 0.00001 respectively, the learning rate is proportionally reduced at 80 epochs and 90 epochs, the momentum is 0.9, the weight decay is 0.0002, and multi-scale training is set; then the training set is input into the garbage classification collection and transportation supervision network in groups of 16 photos, the loss value is calculated using the loss function, and the gradient descent method is used to realize back propagation so that the model parameters are continuously iterated in the direction of minimizing the loss function value; after each round of training, the validation set pictures are also input into the garbage classification collection and transportation supervision network in groups of 16 photos for testing to measure the training status of the model. After 100 rounds of training, a trained garbage classification collection and transportation supervision network is obtained; During the testing phase, the test set is input into the trained garbage classification and collection supervision network to obtain the garbage bin position marking box, garbage bin status and the probability of the status in each image of the test set. Then, the recall rate, precision rate and average precision mean MAP@0.5 of the model are statistically analyzed to measure the generalization ability of the model, thereby obtaining the garbage classification and collection supervision network that can be used online.

3. According to claim 2, a method for monitoring garbage classification, collection and transportation based on an improved YOLOv3 network is characterized in that: The process of constructing the training set, validation set and test set is: Videos of garbage collection and transportation operations are collected, and key frames in the videos are extracted as original images. The original images are manually annotated with the status of the trash cans. Each original image is uniformly scaled to 416×416 pixels and subjected to data augmentation and expansion. 40% of the images after data augmentation and expansion are randomly selected for weather simulation processing. The images after weather simulation processing and the images without weather simulation processing are combined and divided into the training set, validation set and test set in a ratio of 7:2:

1.

4. The method for monitoring garbage classification, collection and transportation based on an improved YOLOv3 network according to claim 3 is characterized in that: The data enhancement and expansion processing includes Gaussian blur, horizontal flip, GridMask, brightness adjustment and random rotation of -20° to +20°; The weather simulation process includes simulating four weather types: daytime rain, daytime fog, nighttime fog and nighttime rain; The trash can states include four states: full with lid open, not fully overflowing, full with lid closed, and not fully overflowing with lid closed.

5. The method for monitoring garbage classification, collection and transportation based on an improved YOLOv3 network according to claim 4, characterized in that: The loss function is L = L box +L cls +L obj Positioning loss function L box for: Confidence error L obj for: Among them, λ noobj refers to the penalty coefficient when there is no target, λ noobj =0.5; C i Refers to the confidence of the true box of the i-th grid; Refers to the confidence of the prediction box of the i-th grid; Indicates whether there is a target in the jth bounding box of the i-th grid. If not, it is 1, otherwise it is 0; L cls refers to the classification error, which is defined as: Among them, p i (c) refers to the conditional probability that the real box of the i-th grid contains a c-type trash can, Refers to the conditional probability that the prediction box of the i-th grid contains a c-type trash can; Among them, the confidence error and classification error both use the two-class cross entropy loss function Among them, m i is the sample label. If the sample is a positive example, it is 1, otherwise it is 0. Predict the probability that the sample is a positive example for the model.

Images