Garbage detection method and device

Abstract

The present application discloses a garbage detection method, which includes: obtaining an image to be identified, inputting the image to be identified into a trained garbage detection model; the garbage detection model can perform feature extraction on the image to be identified, and obtain multiple feature maps of different scales; the garbage detection model performs garbage detection on multiple feature maps respectively, and obtains garbage detection results and result confidences corresponding to each of the multiple feature maps; the garbage detection model outputs the garbage detection result of the image to be identified according to the garbage detection results and result confidences corresponding to each of the multiple feature maps. It can be seen that the present application uses the garbage detection model to identify garbage in the image, and does not need to manually identify garbage from the image as in the prior art, thereby avoiding the problems of missing identification, low efficiency, time-consuming and labor-intensive due to operational errors in the process of manually identifying garbage, improving the efficiency and accuracy of identifying garbage from the image, and thus improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of deep learning visual algorithms, and in particular to a garbage detection method and device. Background Art

[0002] In addition to pursuing economic development, modern society also focuses on the living environment of human beings. In urban life, untimely garbage disposal not only affects the city appearance, but also reduces people's quality of life and affects people's living conditions. How to deal with garbage in a timely manner is the primary problem to be solved by urban management departments and sanitation departments.

[0003] At present, urban garbage supervision and treatment require inspections by patrol personnel and the personnel of the sanitation department are responsible for the area. For large piles of garbage, the garbage inspected needs to be photographed and uploaded to the processing background in the form of pictures. The processing personnel will then command and dispatch the regional person in charge, who will then assign the task to the processing personnel, who will then process the task and feed back the results to the processing background. This process is time-consuming and the presence of multiple transfers can easily lead to a backlog of processing tasks, which not only consumes a lot of manpower and material resources but also leads to citizen complaints due to untimely processing, adding a lot of pressure to municipal and financial management, and greatly reducing the citizens' sense of happiness in life.

[0004] With the popularization of urban surveillance, cameras are almost everywhere in the city, and some urban management work also applies to these cameras. How to combine these surveillance cameras with the treatment of urban garbage and apply deep learning technology and image processing technology to urban garbage detection and management is an important step for smart cities. Summary of the invention

[0005] The present application provides a garbage detection method, which can improve the efficiency and accuracy of identifying garbage from images, thereby improving user experience.

[0006] In a first aspect, the present application provides a garbage detection method, the method comprising:

[0007] Obtain an image to be identified, and input the image to be identified into a trained garbage detection model;

[0008] The garbage detection model extracts features from the image to be identified to obtain a plurality of feature maps of different scales;

[0009] The garbage detection model performs garbage detection on the multiple feature maps respectively to obtain garbage detection results and result confidences corresponding to the multiple feature maps respectively;

[0010] The garbage detection model outputs the garbage detection result of the image to be identified according to the garbage detection results corresponding to each of the multiple feature maps and the result confidence.

[0011] Optionally, the garbage detection result includes garbage location information and garbage category; the garbage location information includes an image to be identified with a garbage area and coordinate information of the garbage.

[0012] Optionally, the garbage detection model is generated in a manner including:

[0013] Obtain a training sample data set; the training sample data set includes several groups of training sample data, wherein each group of training sample data includes a training sample image and a garbage location information label and a garbage category label corresponding to the training sample image;

[0014] Input the training sample image into a preset network model to obtain the predicted garbage location information, predicted garbage category label and predicted confidence corresponding to the training sample image;

[0015] Determine the total loss value according to the predicted garbage location information, predicted garbage category label, and prediction confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image;

[0016] The model parameters of the network model are adjusted according to the total loss value until the network model meets the training conditions, and the network model is used as the garbage detection model.

[0017] Optionally, determining the total loss value according to the predicted garbage location information, predicted garbage category label, and prediction confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image, includes:

[0018] Determine a border loss value, a category loss value, and a confidence loss value according to the predicted garbage location information, the predicted garbage category label, and the predicted confidence corresponding to the training sample, and the garbage location information label and the garbage category label corresponding to the training sample image;

[0019] A total loss value is determined according to the border loss value, the category loss value and the confidence loss value.

[0020] Optionally, obtaining a training sample data set includes:

[0021] Obtaining the original collected image;

[0022] Performing correction processing on the original collected image to obtain a corrected image;

[0023] Obtaining garbage location information labels and garbage category labels corresponding to the corrected image;

[0024] The corrected image is used as a training sample image in the training sample data, and the garbage position information label and garbage category label corresponding to the corrected image are used as the garbage position information label and garbage category label corresponding to the training sample image.

[0025] Optionally, after the step of obtaining the training sample data set, the method further includes:

[0026] Performing data enhancement processing on the training sample images in the training sample data set to obtain data augmented images;

[0027] The data augmented image and the garbage location information label and garbage category label corresponding to the data augmented image are used as a set of training sample data in the training sample data set;

[0028] The data enhancement processing method includes at least one of the following: flip transformation, random pruning, color jitter, translation transformation, scale transformation, contrast transformation, noise disturbance, rotation transformation / reflection transformation.

[0029] Optionally, after the step of obtaining the training sample data set, the method further includes:

[0030] Splicing any multiple training sample images in the training sample data set to obtain a spliced ​​training sample image;

[0031] The spliced ​​training sample image and the garbage location information labels and garbage category labels corresponding to the multiple training sample images are used as a set of training sample data in the training sample data set.

[0032] Optionally, the method further includes:

[0033] If the garbage detection result of the image to be identified includes garbage location information and garbage category, garbage cleaning prompt information is output.

[0034] Optionally, the method further includes:

[0035] If the verification result of the garbage detection result of the image to be identified is a negative sample, the garbage detection model is trained by using the image to be identified and the corrected garbage detection result of the image to be identified.

[0036] In a second aspect, the present application provides a garbage detection device, the device comprising:

[0037] An input unit, used for acquiring an image to be identified and inputting the image to be identified into a trained garbage detection model;

[0038] A first acquisition unit, configured to extract features of the image to be identified by the garbage detection model to obtain feature maps of three different scales;

[0039] A second acquisition unit is used to control the garbage detection model to perform garbage detection on the three feature maps respectively, and obtain garbage detection results and result confidences corresponding to the three feature maps respectively;

[0040] The third acquisition unit is used to control the garbage detection model to output the garbage detection result of the image to be identified according to the garbage detection results and result confidences corresponding to the three feature maps.

[0041] Optionally, the garbage detection result includes garbage location information and garbage category; the garbage location information includes an image to be identified with a garbage area and coordinate information of the garbage.

[0042] Optionally, the device further comprises a model generating unit, configured to:

[0043] Obtain a training sample data set; the training sample data set includes several groups of training sample data, wherein each group of training sample data includes a training sample image and a garbage location information label and a garbage category label corresponding to the training sample image;

[0044] Input the training sample image into a preset network model to obtain the predicted garbage location information, predicted garbage category label and predicted confidence corresponding to the training sample image;

[0045] Determine the total loss value according to the predicted garbage location information, predicted garbage category label, and prediction confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image;

[0046] The model parameters of the network model are adjusted according to the total loss value until the network model meets the training conditions, and the network model is used as the garbage detection model.

[0047] Optionally, the model generating unit is specifically used to:

[0048] Determine a border loss value, a category loss value, and a confidence loss value according to the predicted garbage location information, the predicted garbage category label, and the predicted confidence corresponding to the training sample, and the garbage location information label and the garbage category label corresponding to the training sample image;

[0049] A total loss value is determined according to the border loss value, the category loss value and the confidence loss value.

[0050] Optionally, the model generating unit is specifically used to:

[0051] Obtaining the original collected image;

[0052] Performing correction processing on the original collected image to obtain a corrected image;

[0053] Obtaining garbage location information labels and garbage category labels corresponding to the corrected image;

[0054] The corrected image is used as a training sample image in the training sample data, and the garbage position information label and garbage category label corresponding to the corrected image are used as the garbage position information label and garbage category label corresponding to the training sample image.

[0055] Optionally, the model generating unit is further used for:

[0056] Performing data enhancement processing on the training sample images in the training sample data set to obtain data augmented images;

[0057] The data augmented image and the garbage location information label and garbage category label corresponding to the data augmented image are used as a set of training sample data in the training sample data set;

[0058] The data enhancement processing method includes at least one of the following: flip transformation, random pruning, color jitter, translation transformation, scale transformation, contrast transformation, noise disturbance, rotation transformation / reflection transformation.

[0059] Optionally, the model generating unit is further used for:

[0060] Splicing any multiple training sample images in the training sample data set to obtain a spliced ​​training sample image;

[0061] The spliced ​​training sample image and the garbage location information labels and garbage category labels corresponding to the multiple training sample images are used as a set of training sample data in the training sample data set.

[0062] Optionally, the device further comprises a prompting unit:

[0063] If the garbage detection result of the image to be identified includes garbage location information and garbage category, garbage cleaning prompt information is output.

[0064] Optionally, the device further comprises a retraining unit:

[0065] If the verification result of the garbage detection result of the image to be identified is a negative sample, the garbage detection model is trained by using the image to be identified and the corrected garbage detection result of the image to be identified.

[0066] In a third aspect, the present application provides a readable medium comprising execution instructions. When a processor of an electronic device executes the execution instructions, the electronic device executes any method described in the first aspect.

[0067] In a fourth aspect, the present application provides an electronic device, comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes any method described in the first aspect.

[0068] It can be seen from the above technical scheme that the present application provides a garbage detection method. In this embodiment, the image to be identified can be first obtained, and the image to be identified can be input into the trained garbage detection model; then, the garbage detection model can perform feature extraction on the image to be identified to obtain multiple feature maps of different scales; then, the garbage detection model performs garbage detection on the multiple feature maps respectively to obtain the garbage detection results and result confidences corresponding to the multiple feature maps; finally, the garbage detection model outputs the garbage detection results of the image to be identified according to the garbage detection results and result confidences corresponding to the multiple feature maps. It can be seen that the present application can use the garbage detection model to identify garbage in the image, without the need to manually identify garbage from the image as in the prior art, thereby avoiding the problems of missing identification, low efficiency, time-consuming and labor-intensive due to operational errors in the process of manually identifying garbage, improving the efficiency and accuracy of identifying garbage from the image, and thus improving the user experience.

[0069] The further effects of the above-mentioned non-conventional preferred manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0071] Figure 1 A flowchart of a garbage detection method provided in one embodiment of the present application;

[0072] Figure 2 A flowchart of a garbage detection method provided in one embodiment of the present application;

[0073] Figure 3 A schematic diagram of the structure of a garbage detection device provided in one embodiment of the present application;

[0074] Figure 4 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0076] At present, urban garbage supervision and treatment requires patrol inspections, and the personnel of the sanitation department are responsible for the area. For large piles of garbage, the garbage found during the inspection needs to be photographed and uploaded to the processing background in the form of pictures. The processing personnel will then command and dispatch the regional person in charge, and the regional person in charge will then assign the task to the processing personnel, who will then process and feedback the results to the processing personnel. This process is time-consuming, and there are multiple transfers that easily cause the processing tasks to pile up, which not only consumes a lot of manpower and material resources, but also leads to citizen complaints due to untimely processing, adding a lot of pressure to municipal management and financial management, and greatly reducing the happiness of citizens' lives. Therefore, a new garbage detection method is urgently needed.

[0077] The present application provides a garbage detection method. In this embodiment, an image to be identified can be first obtained, and the image to be identified can be input into a trained garbage detection model; then, the garbage detection model can perform feature extraction on the image to be identified to obtain multiple feature maps of different scales; then, the garbage detection model performs garbage detection on the multiple feature maps respectively to obtain the garbage detection results and result confidences corresponding to the multiple feature maps; finally, the garbage detection model outputs the garbage detection results of the image to be identified according to the garbage detection results and result confidences corresponding to the multiple feature maps. It can be seen that the present application can use the garbage detection model to identify garbage in the image, without the need to manually identify garbage from the image as in the prior art, thereby avoiding the problem of missing identification, low efficiency, time-consuming and labor-intensive due to operational errors in the process of manually identifying garbage, improving the efficiency and accuracy of identifying garbage from the image, and thus improving the user experience. It should be noted that the embodiments of the present application can be applied to electronic devices (such as mobile phones, tablets, computers, etc.) or servers. It should be noted that in addition to the above-mentioned methods, other implementation methods can also be used, which are not limited here.

[0078] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0079] See also Figure 1 , shows a garbage detection method in an embodiment of the present application. In this embodiment, the method may include the following steps:

[0080] S101: Acquire an image to be identified, and input the image to be identified into a trained garbage detection model.

[0081] In this embodiment, an image to be identified may be obtained first, wherein the image to be identified may be understood as an image that needs to be identified as garbage. In one implementation, the image to be identified may be acquired by a camera, for example, in real time, such as once every 10 seconds. It is understood that the camera may be fixed to capture a target area (i.e., an area that needs to be monitored for garbage), or may rotate the camera to capture multiple target areas.

[0082] After acquiring the image to be identified, the image to be identified can be input into a trained garbage detection model, wherein the garbage detection model can be trained based on a training sample data set; wherein the training sample data set can include several groups of training sample data, wherein each group of training sample data includes a training sample image and a garbage detection result corresponding to the training sample image.

[0083] S102: The garbage detection model extracts features from the image to be identified to obtain a plurality of feature maps of different scales.

[0084] In this embodiment, after the garbage detection model obtains the image to be identified, it can perform feature extraction on the image to be identified to obtain multiple feature maps of different scales.

[0085] It should be emphasized that an image usually contains various objects, some large and some small. It is ideal to detect objects of all sizes at once. Therefore, the network must be able to detect objects of different sizes. The deeper the network, the smaller the feature map, so the smaller the object, the more difficult it is to detect. To address this problem, the garbage detection model can use features of multiple scales to detect garbage. For example, in one implementation, the garbage detection model can be a YOLOv3 model, and YOLOv3 can detect garbage through feature maps of three different scales. In this way, the garbage detection model can detect more fine-grained features. The garbage detection model can use the FPN (FeaturePyramid Network) structure to correspond to the different accuracies of multiple scales, and perform garbage detection on feature maps (i.e., feature maps) of different depths. The feature map of the current layer will upsample and utilize the feature map of the future layer (i.e., the next layer). In this way, low-order features can be combined with high-order features to improve the accuracy of garbage detection.

[0086] S103: The junk detection model performs junk detection on the multiple feature maps respectively to obtain junk detection results and result confidences corresponding to the multiple feature maps respectively.

[0087] After obtaining multiple feature maps, the garbage detection model can perform garbage detection on the multiple feature maps respectively to obtain the garbage detection results and result confidences corresponding to the multiple feature maps, that is, the garbage detection results corresponding to each feature map and the result confidences of the garbage detection results. It should be noted that the higher the result confidence of the garbage detection result, the higher the credibility of the garbage detection result, that is, the higher the true rate.

[0088] The garbage detection result may include garbage location information and garbage category. The garbage location information may include an image to be identified with a garbage area and the coordinate information of the garbage. For example, the image to be identified with a garbage area may be an image to be identified with a marked box in the image area with garbage in the image, that is, the garbage area is marked in the image to be identified by the marked box, and the coordinate information of the garbage may be understood as the coordinate information of the marked box. For example, when the marked box is a rectangular box, the coordinate information of the garbage may be the coordinate information of at least two diagonal points of the rectangular box, such as the coordinate information of the upper left corner and the lower right corner of the rectangular box, or the coordinate information of the upper right corner and the lower left corner of the rectangular box. The garbage category may be understood as the type of garbage, for example, it may include waste paper, plastic, glass, metal, cloth, kitchen waste, hazardous waste, etc. The result confidence may include the confidence corresponding to the garbage location information and the confidence corresponding to the garbage category.

[0089] S104: The junk detection model outputs the junk detection result of the image to be identified according to the junk detection results corresponding to each of the plurality of feature maps and the confidence level of the result.

[0090] After the garbage detection model obtains the garbage detection results and result confidences corresponding to the multiple feature maps, the garbage detection model outputs the garbage detection result of the image to be identified based on the garbage detection results and result confidences corresponding to the multiple feature maps. In one implementation, the garbage detection result with the highest result confidence among the garbage detection results and result confidences corresponding to the multiple feature maps can be used as the garbage detection result of the image to be identified.

[0091] It can be seen from the above technical scheme that the present application provides a garbage detection method. In this embodiment, the image to be identified can be first obtained, and the image to be identified can be input into the trained garbage detection model; then, the garbage detection model can perform feature extraction on the image to be identified to obtain multiple feature maps of different scales; then, the garbage detection model performs garbage detection on the multiple feature maps respectively to obtain the garbage detection results and result confidences corresponding to the multiple feature maps; finally, the garbage detection model outputs the garbage detection results of the image to be identified according to the garbage detection results and result confidences corresponding to the multiple feature maps. It can be seen that the present application can use the garbage detection model to identify garbage in the image, without the need to manually identify garbage from the image as in the prior art, thereby avoiding the problems of missing identification, low efficiency, time-consuming and labor-intensive due to operational errors in the process of manually identifying garbage, improving the efficiency and accuracy of identifying garbage from the image, and thus improving the user experience.

[0092] Next, the method for generating the garbage detection model will be introduced. Specifically, in this embodiment, the method for generating the garbage detection model includes the following steps:

[0093] Step a: Obtain a training sample dataset.

[0094] In this embodiment, the training sample data set may include several groups of training sample data, wherein each group of training sample data includes a training sample image and a garbage location information label and a garbage category label corresponding to the training sample image.

[0095] As an example, the original collected images can be obtained first. For example, pictures containing garbage (garbage cans, garbage bags, cardboard (including cartons), plastic bottles, beer bottles) can be intercepted from videos shot by multiple fixed cameras, and these pictures can be aggregated to generate a data set.

[0096] Then, the original collected image can be corrected to obtain a corrected image. Because the process of collecting and intercepting pictures may cause problems such as picture distortion, deformation, blur, etc., it is necessary to screen and correct the data. For the problem of picture deformation, the original collected image can be processed by graphic imagery algorithms to ensure the quality of the data set and avoid the model from reaching the optimal level due to data problems.

[0097] Next, the garbage location information label and garbage category label corresponding to the corrected image can be obtained. Furthermore, the corrected image can be used as a training sample image in the training sample data, and the garbage location information label and garbage category label corresponding to the corrected image can be used as the garbage location information label and garbage category label corresponding to the training sample image. For example, the garbage category contained in each image in the original collected image data set can be selected with a rectangular frame, and the garbage category can be marked to obtain the garbage type and garbage coordinate information (the upper left corner and lower right corner coordinate information of the rectangular frame) contained in each image, forming a labeled data set, wherein the labeled files in the labeled data set correspond one-to-one to the original collected images in the original collected image set.

[0098] In one implementation, after the step of obtaining the training sample data set, the method further includes: performing data enhancement processing on the training sample images in the training sample data set to obtain data augmented images; and using the data augmented images and the garbage location information labels and garbage category labels corresponding to the data augmented images as a set of training sample data in the training sample data set.

[0099] The data enhancement processing method includes at least one of the following: flip transformation, random pruning, color jitter, translation transformation, scale transformation, contrast transformation, noise disturbance, rotation transformation / reflection transformation.

[0100] It should be noted that after the training sample images are labeled, data enhancement is performed on the training sample images to increase the number of training sample images and improve the recognition performance and generalization ability of the model.

[0101] In one implementation, after the step of obtaining the training sample data set, the method further includes: splicing any multiple training sample images in the training sample data set to obtain a spliced ​​training sample image; and using the spliced ​​training sample image and the garbage location information labels and garbage category labels corresponding to the multiple training sample images as a set of training sample data in the training sample data set.

[0102] Specifically, in order to enrich the background of the detected object, multiple training sample images can be fused at the same time in this implementation. For example, four training sample images in the training sample data set can be spliced ​​to obtain one training sample image. This is equivalent to increasing the number of batches in the training process in disguise, which can significantly reduce the demand for large mini_batch.

[0103] Step b: Input the training sample image into a preset network model to obtain the predicted garbage location information, predicted garbage category label and predicted confidence corresponding to the training sample image.

[0104] In this embodiment, one or more training sample images may be sequentially input into a preset network model to obtain predicted garbage location information, predicted garbage category labels, and predicted confidence corresponding to each training sample image.

[0105] It should be noted that in one implementation of this embodiment, the preset network model can be a YOLO V3 model. YOLOv3 adjusts the network structure based on YOLOv1 and YOLOv2, uses multi-scale features for object detection, and replaces softmax with Logistic for object classification. YOLOv3 has no fully connected layer and pooling layer, can correspond to input images of any size, is mainly composed of 75 convolutional layers, and adds a resnet residual module to the network, which is conducive to solving the gradient problem of deep networks. The Resnet residual network is equivalent to adding a shortcut path to the original CNN network structure. The learning process changes from directly learning features to adding certain features on the basis of previously learned features to obtain better features. In this way, a complex feature H(x), which was previously learned independently layer by layer, now becomes such a model H(x)=F(x)+x, where x is the feature at the beginning of the shortcut, and F(x) is the filling and addition of x to become a residual. Therefore, the goal of learning changes from learning complete information to learning residuals, which greatly reduces the difficulty of learning high-quality features. The Softmax layer in YOLOv3 is replaced by a 1x1 convolution layer + logistic activation function structure, which can correspond to multi-label objects. When YOLOv3 predicts the pre-selected box bbox, it uses logistic regression. Each pre-selected box contains five elements bbox(x, y, w, h, c). The first four represent the size and coordinate position of the pre-selected box, and the last value is the confidence; Pr(object)*IOU(bbox, object) = σ(t0), where Pr(object)*IOU(bbox, object) is the confidence of the predicted border, which is the value of the confidence after the prediction parameter t0 is transformed by σ, and t0 is the network output confidence. σ is the sigmoid function.

[0106] Step c: determining a total loss value according to the predicted garbage location information, predicted garbage category label, and prediction confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image;

[0107] In this embodiment, the border loss value (lbox (i.e., the error of bbox)), the category loss value (lcls), and the confidence loss value (lobj) can be determined based on the predicted garbage location information, the predicted garbage category label, the predicted confidence corresponding to the training sample, and the garbage location information label and the garbage category label corresponding to the training sample image. Specifically, the border loss value can be determined based on the predicted garbage location information and the garbage location information label, the category loss value can be determined based on the predicted garbage category label and the garbage category label, and the confidence loss value can be determined based on the predicted confidence. Among them, the calculation method of the border loss value (lbox), the category loss value (lcls), and the confidence loss value (lobj) is as follows:

[0108]

[0109]

[0110]

[0111]

[0112] Among them, S represents the prediction grid scale, B represents the prediction coordinates, Indicates that if there is a target at the predicted coordinates at i, j, its value is 1, otherwise it is 0; Indicates that if there is no target at the predicted coordinates at i, j, its value is 1, otherwise it is 0; c represents the confidence, p represents the category probability; λ coord ,

[0113] λ class , nobj , opj is a parameter that exists to increase the loss of bounding boxes and reduce the loss of confidence predictions for boxes that do not contain objects; x i ,y i represents the center point coordinates, w i ,h i Represents width and height respectively; Respectively represent the expected center point coordinates, Represent the expected width and height respectively, is the expected confidence. Then, the total loss value can be determined according to the border loss value, the category loss value and the confidence loss value. For example, the sum of the border loss value, the category loss value and the confidence loss value can be used as the total loss value, that is, loss = lbox + lobj + lcls, where loss is the total loss value.

[0114] Step d: adjusting the model parameters of the network model according to the total loss value until the network model meets the training conditions, and using the network model as the garbage detection model.

[0115] The training condition is that the number of iterative training of the model meets a preset number, such as 500 times, or the model parameters of the model have converged.

[0116] In addition, it should be noted that, in this embodiment, the network model training model and adjusting parameters to obtain the optimal weight model are as follows: the data set is divided into three parts: training set, validation set, and test set. The training set is used for model training based on the YOLOv3 network, and multiple weight models can be obtained by sending the training set to the YOLOv3 model for iterative training. The validation set is used to adjust the model parameters in time during model training. Since the weights of the network model are randomly initialized at the beginning of training, if a larger learning rate is selected at this time, the instability (oscillation) of the model may be caused. In the process of training the model, the gradual warmup method is used to adjust the learning rate. Starting from the initial small learning rate, each step increases a little bit until the relatively large learning rate set initially is reached. The learning rate set initially is used for training, which can make the learning rate in the first few epoches or some steps of the training smaller. Under the preheated small learning rate, the model can slowly tend to be stable. After the model is relatively stable, the pre-set learning rate is selected for training, so that the model convergence speed becomes faster and the model effect is better. During the training process, a corresponding weight file will be generated based on the network learning effect, and garbage can be detected in untrained images based on the weight file. The weight file is equivalent to saving the learned features. When used, the features in the weight file are used to match untrained images to obtain detection results.

[0117] The training is based on the coco pre-trained model, which can quickly reduce the model learning loss and improve the model training efficiency. Verification while training can adjust the model parameters in time and reduce the time cost caused by late result verification. The test set is used to test the model training effect after the model training is completed, and is used for the prediction of the optimal weight model.

[0118] In one implementation, after S104, the method further includes: if the garbage detection result of the image to be identified includes garbage location information and garbage category, outputting garbage cleaning prompt information.

[0119] In this embodiment, if the garbage detection result of the image to be identified includes garbage location information and garbage category, it means that the corresponding area in the image to be identified includes garbage. At this time, garbage cleaning prompt information can be output, such as voice reminder or pop-up cleaning message in the terminal interface to prompt the user to clean up the garbage. In this way, this embodiment outputs garbage cleaning prompt information to prompt the user to clean up the garbage by determining that the garbage detection result of the image to be identified includes garbage location information and garbage category. It can be combined with the social governance platform, and the governance linkage realizes the function of automatic abnormal reporting and early warning, which plays a significant role in the city appearance, people's livelihood improvement and other aspects.

[0120] Because garbage time reporting takes a certain period of time, the continued existence of garbage will update the garbage existence time, and whether the garbage has been processed can be determined based on the duration. Therefore, in this embodiment, the garbage cleaning prompt information can be output only when garbage is continuously detected in the same area for a continuous period of time (for example, within ten minutes).

[0121] In one implementation, after S104, the method further includes:

[0122] If the verification result of the garbage detection result of the image to be identified is a negative sample, that is, the garbage monitoring result is inconsistent with the actual garbage detection result of the image to be identified, the verification result is a negative sample, indicating that the garbage detection model has identified errors. The image to be identified and the corrected garbage detection result of the image to be identified can be used to train the garbage detection model.

[0123] As an example, Figure 2 As shown, the video shot under the fixed camera can be transmitted back to the social governance platform server in real time, and the trained model can be used to detect the transmitted video in real time. The detected garbage is marked with a rectangular frame in the video and the category is marked. The social governance platform server will intervene in the detection results. If the social governance platform server detects that the detection result is wrong, the negative sample of the detection error can be collected. When the collected negative samples exceed a certain number or there are new foreign matter (garbage) categories, self-optimization will automatically start, that is, the collected negative samples will be used to continue model training for the garbage detection model. At this time, the training sample data in the data set includes two sources, one is the prepared data, and the other is the pictures automatically collected by the system. In this way, as time goes by, the more negative samples of feedback obtained by this application, the more times the garbage detection model is trained online, the more the effect of the model will be gradually optimized, and the accuracy will gradually increase.

[0124] In summary, compared with the prior art, this application has the following main benefits:

[0125] (1) This application uses video shot by a fixed camera as the data source, which reduces human intervention and can accurately and real-time detect the current situation under the camera.

[0126] (2) This application can be applied to garbage detection in social governance platforms. After garbage is detected, it will be manually disposed of on the platform. This application will collect images of detected objects and the results of manual disposal, and convert the results of manual disposal into labels corresponding to the training model. When the number of collected object images reaches the specified threshold, this application will automatically start the model training process. As time goes by, the present invention obtains more manual feedback, the number of online training iterations also increases, the effect of the model will gradually be optimized, and the accuracy will gradually increase.

[0127] (3) This application takes garbage management as its main purpose. It is based on YOLOv3 real-time detection. The model has high stability and good detection and recognition performance. It can eliminate the influence of external factors and achieve the purpose of "precise management".

[0128] (4) Combined with the social governance platform, the governance linkage realizes the function of automatic reporting and early warning of abnormalities, which plays a significant role in improving the city appearance and people's livelihood.

[0129] like Figure 3 As shown, a specific embodiment of a garbage detection device described in the present application is shown. The device described in this embodiment is a physical device for executing the method described in the above embodiment. Its technical solution is essentially consistent with the above embodiment. The device described in this embodiment includes:

[0130] An input unit 301 is used to obtain an image to be identified and input the image to be identified into a trained garbage detection model;

[0131] A first acquisition unit 302 is used for the garbage detection model to extract features from the image to be identified to obtain feature maps of three different scales;

[0132] A second acquisition unit 303 is used to control the junk detection model to perform junk detection on the three feature maps respectively, and obtain junk detection results and result confidences corresponding to the three feature maps respectively;

[0133] The third acquisition unit 304 is used to control the junk detection model to output the junk detection result of the image to be identified according to the junk detection results and result confidences corresponding to the three feature maps.

[0134] Optionally, the garbage detection result includes garbage location information and garbage category; the garbage location information includes an image to be identified with a garbage area and coordinate information of the garbage.

[0135] Optionally, the device further comprises a model generating unit, configured to:

[0136] Obtain a training sample data set; the training sample data set includes several groups of training sample data, wherein each group of training sample data includes a training sample image and a garbage location information label and a garbage category label corresponding to the training sample image;

[0137] Input the training sample image into a preset network model to obtain the predicted garbage location information, predicted garbage category label and predicted confidence corresponding to the training sample image;

[0138] Determine the total loss value according to the predicted garbage location information, predicted garbage category label, and prediction confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image;

[0139] The model parameters of the network model are adjusted according to the total loss value until the network model meets the training conditions, and the network model is used as the garbage detection model.

[0140] Optionally, the model generating unit is specifically used to:

[0141] Determine a border loss value, a category loss value, and a confidence loss value according to the predicted garbage location information, the predicted garbage category label, and the predicted confidence corresponding to the training sample, and the garbage location information label and the garbage category label corresponding to the training sample image;

[0142] A total loss value is determined according to the border loss value, the category loss value and the confidence loss value.

[0143] Optionally, the model generating unit is specifically used to:

[0144] Obtaining the original collected image;

[0145] Performing correction processing on the original collected image to obtain a corrected image;

[0146] Obtaining garbage location information labels and garbage category labels corresponding to the corrected image;

[0147] The corrected image is used as a training sample image in the training sample data, and the garbage position information label and garbage category label corresponding to the corrected image are used as the garbage position information label and garbage category label corresponding to the training sample image.

[0148] Optionally, the model generating unit is further used for:

[0149] Performing data enhancement processing on the training sample images in the training sample data set to obtain data augmented images;

[0150] The data augmented image and the garbage location information label and garbage category label corresponding to the data augmented image are used as a set of training sample data in the training sample data set;

[0151] The data enhancement processing method includes at least one of the following: flip transformation, random pruning, color jitter, translation transformation, scale transformation, contrast transformation, noise disturbance, rotation transformation / reflection transformation.

[0152] Optionally, the model generating unit is further used for:

[0153] Splicing any multiple training sample images in the training sample data set to obtain a spliced ​​training sample image;

[0154] The spliced ​​training sample image and the garbage location information labels and garbage category labels corresponding to the multiple training sample images are used as a set of training sample data in the training sample data set.

[0155] Optionally, the device further comprises a prompting unit:

[0156] If the garbage detection result of the image to be identified includes garbage location information and garbage category, garbage cleaning prompt information is output.

[0157] Optionally, the device further comprises a retraining unit:

[0158] If the verification result of the garbage detection result of the image to be identified is a negative sample, the garbage detection model is trained by using the image to be identified and the corrected garbage detection result of the image to be identified.

[0159] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0160] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0161] The memory is used to store execution instructions. Specifically, the execution instructions are computer programs that can be executed. The memory may include internal memory and non-volatile memory, and provides execution instructions and data to the processor.

[0162] In one possible implementation, the processor reads the corresponding execution instructions from the non-volatile memory into the memory and then runs them, and can also obtain the corresponding execution instructions from other devices to form a garbage detection device at the logical level. The processor executes the execution instructions stored in the memory to implement the garbage detection method provided in any embodiment of the present application through the executed execution instructions.

[0163] The above application Figure 1 The method performed by the garbage detection device provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0164] The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0165] An embodiment of the present application also proposes a readable storage medium, which stores execution instructions. When the stored execution instructions are executed by a processor of an electronic device, the electronic device can execute the garbage detection method provided in any embodiment of the present application, and is specifically used to execute the method described above for garbage detection.

[0166] The electronic device described in the above embodiments may be a computer.

[0167] Those skilled in the art should understand that the embodiments of the present application can be provided as methods or computer program products. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware.

[0168] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0170] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A garbage detection method, characterized in that: The method comprises: Obtaining an image to be identified, and inputting the image to be identified into a trained garbage detection model; The garbage detection model extracts features from the image to be identified to obtain a plurality of feature maps of different scales; The garbage detection model performs garbage detection on the multiple feature maps respectively to obtain garbage detection results and result confidences corresponding to the multiple feature maps respectively; The junk detection model outputs the junk detection result of the image to be identified according to the junk detection results corresponding to each of the multiple feature graphs and the result confidence; The generation method of the garbage detection model includes: Obtain a training sample data set; the training sample data set includes several groups of training sample data, wherein each group of training sample data includes a training sample image and a garbage location information label and a garbage category label corresponding to the training sample image; Input the training sample image into a preset network model to obtain the predicted garbage location information, predicted garbage category label and predicted confidence corresponding to the training sample image; Determine the total loss value according to the predicted garbage location information, predicted garbage category label, and prediction confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image; Adjusting the model parameters of the network model according to the total loss value until the network model meets the training conditions, and using the network model as the garbage detection model; The determining of the total loss value according to the predicted garbage location information, predicted garbage category label, and predicted confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image, includes: According to the predicted garbage location information, predicted garbage category label, predicted confidence corresponding to the training sample, and the garbage location information label and garbage category label corresponding to the training sample image, the border loss value lbox, the category loss value lcls, and the confidence loss value lobj are determined: Among them, S represents the prediction grid scale, B represents the prediction coordinates, Indicates that if there is a target at the predicted coordinates at i, j, its value is 1, otherwise it is 0; Indicates that if there is no target at the predicted coordinates at i, j, its value is 1, otherwise it is 0; c represents the confidence, p represents the category probability; λ coord , class , nobj , opj is a parameter that exists to increase the loss of bounding boxes and reduce the loss of confidence predictions for boxes that do not contain objects; x i ,y i represents the center point coordinates, w i ,h i Represents width and height respectively; , Respectively represent the expected center point coordinates, , Represent the expected width and height respectively, is the expected confidence; According to the border loss value lbox, the category loss value lcls and the confidence loss value lobj, the total loss value loss is determined. 。 2. The method according to claim 1, characterized in that The garbage detection result includes garbage location information and garbage category; the garbage location information includes an image to be identified with a garbage area marked and coordinate information of the garbage.

3. The method according to claim 1, characterized in that The step of obtaining a training sample data set includes: Obtaining the original collected image; Performing correction processing on the original collected image to obtain a corrected image; Obtaining garbage location information labels and garbage category labels corresponding to the corrected image; The corrected image is used as a training sample image in the training sample data, and the garbage position information label and garbage category label corresponding to the corrected image are used as the garbage position information label and garbage category label corresponding to the training sample image.

4. The method according to claim 1, characterized in that: After the step of obtaining the training sample data set, the method further includes: Performing data enhancement processing on the training sample images in the training sample data set to obtain data augmented images; The data augmented image and the garbage location information label and garbage category label corresponding to the data augmented image are used as a set of training sample data in the training sample data set; The data enhancement processing method includes at least one of the following: flip transformation, random pruning, color jitter, translation transformation, scale transformation, contrast transformation, noise disturbance, rotation transformation / reflection transformation.

5. The method according to claim 1, characterized in that: After the step of obtaining the training sample data set, the method further includes: Splicing any multiple training sample images in the training sample data set to obtain a spliced ​​training sample image; The spliced ​​training sample image and the garbage location information labels and garbage category labels corresponding to the multiple training sample images are used as a set of training sample data in the training sample data set.

6. The method according to claim 1, characterized in that The method further comprises: If the garbage detection result of the image to be identified includes garbage location information and garbage category, garbage cleaning prompt information is output.

7. The method according to claim 1, characterized in that The method further comprises: If the verification result of the garbage detection result of the image to be identified is a negative sample, the garbage detection model is trained by using the image to be identified and the corrected garbage detection result of the image to be identified.

Images