A method for detecting exposed garbage and overflowing garbage based on deep learning

Abstract

The present invention provides an exposed garbage and overflowing garbage detection method based on deep learning, comprising: training a first target detection model for detecting garbage, and inputting a target image into the first target detection model to obtain a first target detection result; when it is determined that there is garbage according to the first target detection result, determining whether the garbage is exposed garbage according to the positional relationship between the garbage and the garbage bin; when it is determined that the garbage is not exposed garbage, inputting the target sub-image into a pre-trained second target detection model, and determining whether the garbage is overflowing garbage according to the second target detection result; by training the first target detection model for detecting garbage and the second target detection model for detecting overflowing garbage based on deep learning, the accuracy of garbage type recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a method for detecting exposed garbage and overflowing garbage based on deep learning. Background Art

[0002] In recent years, local governments have been working hard to promote the construction of civilized cities nationwide. Whether it is urban roads or commercial areas with large traffic, or in various residential areas, a large number of trash cans have been added. As people's requirements for living environment gradually increase, it is particularly important to be able to find garbage in time, especially exposed garbage or garbage that has overflowed from the trash can, and deal with it in time.

[0003] The current method for detecting garbage performs region detection on the acquired original image to obtain the target area in the original image, and then performs target detection on the target area to obtain the location information and type information of the garbage. Although the garbage can be detected to a certain extent, due to the similarity of the garbage itself, it is difficult to distinguish whether the garbage is exposed on the ground or overflowing from the trash can. It is difficult to stably identify the specific situation of the garbage by relying on a single target detection algorithm. Summary of the invention

[0004] The present invention provides an exposed garbage and overflowing garbage detection method based on deep learning. Based on deep learning, a first target detection model for detecting garbage and a second target detection model for detecting overflowing garbage are trained to improve the accuracy of garbage type recognition.

[0005] A method for detecting exposed garbage and overflowing garbage based on deep learning, characterized by comprising:

[0006] Step 1: Train a first target detection model for detecting garbage, and input a target image into the first target detection model to obtain a first target detection result;

[0007] Step 2: When it is determined that there is garbage according to the first target detection result, the target image is marked as garbage, and a target sub-image is determined according to the garbage marking result;

[0008] Step 3: Detect whether there is a trash can in the target sub-image. If not, determine that the trash is exposed trash; if so, determine whether the trash is exposed trash based on the positional relationship between the trash and the trash can;

[0009] Step 4: When it is determined that the garbage is not exposed garbage, the target sub-image is input into a pre-trained second target detection model, and based on the second target detection result, it is determined whether the garbage is overflowing garbage.

[0010] Preferably, in step 1, training a first target detection model for detecting garbage includes:

[0011] Acquire sample images captured by the same camera as the target image, divide the images into a first sample image set and a second sample image set, perform a first annotation on the first sample image set, and perform a second annotation on the second sample image set;

[0012] The detection model is trained using the first sample image set and the second sample image set until the probability that the output result of the detection model conforms to the first annotation and the second annotation is greater than a preset probability, and the training of the detection model is completed to obtain a first target detection model.

[0013] Preferably, in step 2, the step of determining whether there is garbage according to the first target detection result includes:

[0014] Inputting the target image into the first target detection model to obtain a first target detection result;

[0015] Obtaining a marked feature in the first target detection result, and determining whether the marked feature is a first mark;

[0016] If so, it indicates that there is garbage in the target image;

[0017] Otherwise, it indicates that there is no garbage in the target image.

[0018] Preferably, in step 2, after determining that garbage exists according to the first target detection result, the step further includes correcting the target image, specifically:

[0019] Inputting the target image into the first target detection model to obtain an initial detection result;

[0020] Performing feature extraction on the target image to obtain multiple groups of image feature maps;

[0021] Inputting the multiple groups of image feature maps into the first object detection model in sequence to obtain multiple groups of image detection results;

[0022] Determining whether the initial detection result is consistent with the detection results of multiple sets of images;

[0023] If yes, do not modify the target image;

[0024] Otherwise, the target image features corresponding to the inconsistent image detection results are obtained, and the target image features are corrected based on the initial detection results to obtain a corrected target image.

[0025] Preferably, correcting the target image feature based on the initial detection result to obtain a corrected target image includes:

[0026] Obtaining obvious image features corresponding to the target detection results, and extracting obvious attribute features from the obvious image features that are consistent with the attributes of the target image features;

[0027] Based on the difference between the obvious attribute feature and the target image feature, determining the correction part and the correction scheme of the target image feature, and correcting the correction part by using the correction scheme to obtain a correction feature map;

[0028] The corrected feature map and other groups of image feature maps are combined to obtain a corrected target image.

[0029] Preferably, in step 2, determining the target sub-image according to the junk labeling result includes:

[0030] According to the garbage marking result, determining the number and location of garbage areas in the target image;

[0031] If the number of regions is 1, the area of ​​the search region is determined based on the area of ​​the regional location of the garbage;

[0032] Taking the area position as the center, obtaining the image features of the first image area of ​​the search area, and obtaining the sub-image area corresponding to the image features within the preset image feature range;

[0033] Reducing a portion of the first image region where no sub-image region exists within a preset range to obtain a second image region;

[0034] Determine that the sub-image area exists at a boundary of the second image area, and based on a proportion of the sub-image area in the boundary, expand the boundary to obtain a third image area;

[0035] Determining a target sub-image based on the third image area;

[0036] If the number of regions is greater than 1, two adjacent garbage locations are connected in pairs with the garbage locations as the boundary to obtain a fourth image region;

[0037] Based on image features of an image area near the outside of the fourth image area, acquiring a target external image area corresponding to the image features within the preset image feature range;

[0038] For the connection relationship between the target garbage contained in the target external image region, the positions where the target garbage is located are reconnected according to the image features of the target external image region to obtain a new target external image region;

[0039] Based on the new target external image region, updating the fourth image region to obtain a fifth image region;

[0040] Expanding the fifth image region according to a preset expansion method to obtain a sixth image region;

[0041] Based on the sixth image area, a target sub-image is determined.

[0042] Preferably, the fifth image region is expanded according to a preset expansion method to obtain a sixth image region including:

[0043] Determining an expansion area based on the area of ​​the fifth image area;

[0044] Determining expansion ratios in various directions of the fifth image region based on image features of each piece of garbage in the fifth image region;

[0045] Based on the expansion area and the expansion ratio, the fifth image region is expanded to obtain a sixth image region.

[0046] Preferably, in step 3, determining whether the garbage is exposed garbage according to the positional relationship between the garbage and the garbage bin includes:

[0047] Segmenting the target sub-image according to the positions of the garbage and the garbage bin to obtain a plurality of pixel sets;

[0048] Extracting pixel positions from the pixel set, and selecting a first pixel position belonging to the garbage and a second pixel position belonging to the garbage bin from the pixel positions, and determining an image distance between the garbage and the garbage bin according to a positional relationship between the first pixel position and the second pixel position;

[0049] If the image distance is greater than a preset distance, determining that the garbage is exposed garbage;

[0050] Otherwise, obtaining the position relationship between the garbage and the garbage bin;

[0051] The method for obtaining the positional relationship between the garbage and the garbage bin is as follows:

[0052] Based on the relative positions of the garbage and the garbage bin, determining a method for establishing a first coordinate system for the garbage and a method for establishing a second coordinate system for the garbage bin;

[0053] Selecting a first center position and a second center position of the first pixel point position and the second pixel point position as coordinate origins respectively, and establishing a first coordinate system and a second coordinate system by using the first coordinate system establishing method and the second coordinate system establishing method;

[0054] Obtain the first highest point, the first lowest point, the first leftmost point, and the first rightmost point of the trash can in the first coordinate system, and the second highest point, the second lowest point, the second leftmost point, and the second rightmost point of the trash in the second coordinate system;

[0055] Determine a coordinate relationship set between each of the first highest point, the first lowest point, the first leftmost point, the first rightmost point, the second highest point, the second lowest point, the second leftmost point, and the second rightmost point in the first coordinate system and the second coordinate system;

[0056] Inputting the coordinate relationship set into a pre-trained image position and orientation recognition model to obtain the orientation relationship between the garbage and the garbage bin;

[0057] Determine the actual distance between the garbage and the garbage bin based on the orientation relationship and the image distance;

[0058] Determining whether the actual distance is greater than a preset actual distance;

[0059] If so, it is determined that the waste is exposed waste;

[0060] Otherwise, it is determined that the garbage is not exposure garbage.

[0061] Preferably, in step 4, the steps of pre-training the second target detection model are as follows:

[0062] Obtain overflowing garbage images from a garbage image database, and classify the overflowing garbage images according to overflow types to obtain multiple sets of overflowing garbage images;

[0063] Obtaining an average image feature in each group of overflowing garbage image sets, and searching the garbage image database for a comparison garbage image set whose difference from the average image feature is within a preset feature difference range;

[0064] Based on each set of overflowing garbage image sets and comparison garbage image sets, multiple sets of sample image sets are established, and a detection image and a comparison image in each set of sample image sets are defined;

[0065] Setting data collection accuracy for each set of sample images based on average similarity between image features of the detection image and image features of the comparison image in each set of sample images;

[0066] According to the data collection accuracy, point cloud data collection is performed on the images in each group of sample image sets to obtain point cloud data;

[0067] and setting a label for each group of sample image sets based on the image type in each group of sample image sets to obtain a labeling result;

[0068] Based on the image features of each set of sample images and the data acquisition accuracy, the image analysis rules of the preset detection model are set to obtain a preliminary detection model;

[0069] Based on the point cloud data and the annotation results, the preliminary detection model is trained to obtain a complete detection model;

[0070] Randomly selecting a preset number of images to be detected from the overflowing garbage image and the comparison garbage image set, inputting them into the complete detection model, and obtaining the accuracy of the detection result;

[0071] If the accuracy rate is greater than a preset probability, determining that the complete detection model is a second target detection model;

[0072] Otherwise, the image analysis rules and model parameters are adjusted until the accuracy is greater than the preset probability, and the latest detection model obtained is used as the second target detection model.

[0073] Preferably, in step 4, determining whether the garbage is overflowing garbage according to the second target detection result includes:

[0074] Inputting the target sub-image into a pre-trained second target detection model, and outputting a plurality of output labels for the target sub-image;

[0075] Based on a preset label type, the multiple output labels are divided into two categories to obtain label types of the multiple output labels, and the proportion of the label types that are target types is determined, and the proportion is used as the second target detection result;

[0076] If the proportion of the second target detection results is greater than a preset proportion, determining that the garbage is overflowing garbage;

[0077] Otherwise, it is determined that the garbage is not overflowing garbage.

[0078] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0079] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0081] Figure 1 Flow chart of a method for detecting exposed garbage and overflowing garbage based on deep learning in an embodiment of the present invention;

[0082] Figure 2 This is a flow chart of a first target detection model trained to detect garbage in an embodiment of the present invention;

[0083] Figure 3 The figure is a flow chart of correcting the target image in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0085] Example 1

[0086] The embodiment of the present invention provides a method for detecting exposed garbage and overflowing garbage based on deep learning, such as Figure 1 As shown, including:

[0087] Step 1: Train a first target detection model for detecting garbage, and input a target image into the first target detection model to obtain a first target detection result;

[0088] Step 2: When it is determined that there is garbage according to the first target detection result, the target image is marked as garbage, and a target sub-image is determined according to the garbage marking result;

[0089] Step 3: Detect whether there is a trash can in the target sub-image. If not, determine that the trash is exposed trash; if so, determine whether the trash is exposed trash based on the positional relationship between the trash and the trash can;

[0090] Step 4: When it is determined that the garbage is not exposed garbage, the target sub-image is input into a pre-trained second target detection model, and based on the second target detection result, it is determined whether the garbage is overflowing garbage.

[0091] In this embodiment, the first target detection model is trained by a deep learning network and is used to detect whether there is garbage in an image.

[0092] In this embodiment, the target sub-image is an image in the target image that contains the garbage annotation and its surrounding area.

[0093] In this embodiment, if the distance and shape of the garbage from the garbage bin meet the preset requirements, it is exposed garbage.

[0094] In this embodiment, the second object detection model is trained by a deep learning network and is used to detect whether the garbage in the image is overflowing garbage.

[0095] The beneficial effect of the above design scheme is: through deep learning-based training, a first target detection model for detecting garbage and a second target detection model for detecting overflowing garbage are obtained, thereby ensuring the accuracy of identifying garbage and its garbage type. When it is determined that there is garbage in the target image, the area where the garbage is located is further determined, thereby improving the accuracy of identifying the garbage type.

[0096] Example 2

[0097] Based on Example 1, the embodiment of the present invention provides a method for detecting exposed garbage and overflowing garbage based on deep learning. In step 1, a first target detection model for detecting garbage is trained, including:

[0098] Step 1-1: obtaining sample images captured by the same camera as the target image, and dividing the images into a first sample image set and a second sample image set, and performing a first annotation on the first sample image set and a second annotation on the second sample image set;

[0099] Step 1-2: Use the first sample image set and the second sample image set to train the detection model until the probability that the output result of the detection model conforms to the first annotation and the second annotation is greater than a preset probability, completing the training of the detection model and obtaining a first target detection model.

[0100] In this embodiment, the images in the first sample image set are images containing garbage.

[0101] In this embodiment, the images in the second sample image set are images in which no garbage is present and objects similar to garbage are present to varying degrees.

[0102] In this embodiment, the first label is a label for garbage, and the second label is a label for an object similar to garbage.

[0103] The beneficial effects of the above design scheme are: by obtaining sample images under the same camera as the target image, the effectiveness of the selection of training samples for the detection model is guaranteed, the detection of the target image by the first target detection model is guaranteed to be more targeted, and the recognition accuracy is guaranteed; by selecting a second sample image set in which there is no garbage and objects similar to garbage exist to varying degrees in the image, the accuracy of garbage recognition by the trained first target detection model is guaranteed.

[0104] Example 3

[0105] Based on Example 1, the embodiment of the present invention provides a method for detecting exposed garbage and overflowing garbage based on deep learning. In step 2, the step of determining the presence of garbage according to the first target detection result includes:

[0106] Inputting the target image into the first target detection model to obtain a first target detection result;

[0107] Obtaining a marked feature in the first target detection result, and determining whether the marked feature is a first mark;

[0108] If so, it indicates that there is garbage in the target image;

[0109] Otherwise, it indicates that there is no garbage in the target image.

[0110] In this embodiment, the first annotation indicates that there are junk features in the image.

[0111] The beneficial effect of the above design scheme is: by judging based on the labeled features in the first target detection result, it is determined whether there is garbage in the target image, and the result of garbage detection on the target image is obtained.

[0112] Example 4

[0113] Based on Example 1, the embodiment of the present invention provides an exposed garbage and overflowing garbage detection method based on deep learning. In step 2, after determining that there is garbage according to the first target detection result, it also includes correcting the target image, specifically:

[0114] Step 2-1: input the target image into the first target detection model to obtain an initial detection result;

[0115] Step 2-2: extracting features from the target image to obtain multiple sets of image feature maps;

[0116] Step 2-3: inputting the multiple groups of image feature maps into the first object detection model in sequence to obtain multiple groups of image detection results;

[0117] Step 2-4: Determine whether the initial detection result is consistent with the detection results of multiple groups of images;

[0118] If yes, no correction is performed on the target image;

[0119] Otherwise, the target image features corresponding to the inconsistent image detection results are obtained, and the target image features are corrected based on the initial detection results to obtain a corrected target image.

[0120] In this embodiment, the multiple groups of image feature maps may be, for example, feature maps of color, contour, shape, etc.

[0121] In this embodiment, correcting the target image feature based on the initial detection result specifically involves correcting the target image feature according to the image characteristics of the initial detection result.

[0122] The beneficial effect of the above design scheme is that when it is determined that there is garbage in the target image, multiple groups of features of the target image are further detected and corrected, providing an accurate data basis for subsequent further image detection of the target image.

[0123] Example 5

[0124] Based on Example 4, the embodiment of the present invention provides an exposed garbage and overflowing garbage detection method based on deep learning, which corrects the target image feature based on the initial detection result to obtain a corrected target image, including:

[0125] Obtaining obvious image features corresponding to the target detection results, and extracting obvious attribute features from the obvious image features that are consistent with the attributes of the target image features;

[0126] Based on the difference between the obvious attribute feature and the target image feature, determining the correction part and the correction scheme of the target image feature, and correcting the correction part by using the correction scheme to obtain a correction feature map;

[0127] The corrected feature map and other groups of image feature maps are combined to obtain a corrected target image.

[0128] The beneficial effect of the above design scheme is: by correcting the target image, the recognition accuracy of the target image in the detection model is guaranteed, and an accurate data basis is provided for subsequent further image detection of the target image.

[0129] Example 6

[0130] Based on Example 1, the embodiment of the present invention provides a method for detecting exposed garbage and overflowing garbage based on deep learning. In step 2, according to the garbage labeling result, a target sub-image is determined, including:

[0131] According to the garbage marking result, determining the number and location of garbage areas in the target image;

[0132] If the number of regions is 1, the area of ​​the search region is determined based on the area of ​​the regional location of the garbage;

[0133] Taking the area position as the center, obtaining the image features of the first image area of ​​the search area, and obtaining the sub-image area corresponding to the image features within the preset image feature range;

[0134] Reducing a portion of the first image region where no sub-image region exists within a preset range to obtain a second image region;

[0135] Determine that the sub-image area exists at a boundary of the second image area, and based on a proportion of the sub-image area in the boundary, expand the boundary to obtain a third image area;

[0136] Determining a target sub-image based on the third image area;

[0137] If the number of regions is greater than 1, two adjacent garbage locations are connected in pairs with the garbage locations as the boundary to obtain a fourth image region;

[0138] Based on image features of an image area near the outside of the fourth image area, acquiring a target external image area corresponding to the image features within the preset image feature range;

[0139] For the connection relationship between the target garbage contained in the target external image region, the positions where the target garbage is located are reconnected according to the image features of the target external image region to obtain a new target external image region;

[0140] Based on the new target external image region, updating the fourth image region to obtain a fifth image region;

[0141] Expanding the fifth image region according to a preset expansion method to obtain a sixth image region;

[0142] Based on the sixth image area, a target sub-image is determined.

[0143] In this embodiment, the preset image feature range is roughly set in advance according to the features of the trash can, and the preset image feature range represents image features that are likely to represent the trash can.

[0144] In this embodiment, the larger the area of ​​the regional location is, the larger the area of ​​the search region is determined to be.

[0145] In this embodiment, the greater the proportion of the image area in the boundary, the greater the boundary expansion range, and the larger the obtained third image area.

[0146] In this embodiment, the difference in the connection relationship between the locations where the marked garbage is located directly affects the determination of the range of the target external image area.

[0147] The beneficial effects of the above design scheme are: by adopting different methods to determine the target sub-image according to the different amounts of garbage in the target image, the range accuracy of the target sub-image is guaranteed, and at the same time, the optimal image basis is provided for the subsequent identification of the type of garbage, wherein, when the number of regions is 1, the location of the garbage is taken as the center, and the expansion range in each direction is determined according to the surrounding image features; when the number of regions is greater than 1, the image region composed of each garbage is first determined, and then expanded to finally obtain the target sub-image, ensuring that the area of ​​the target sub-image is minimized as much as possible under the premise that the target sub-image can contain the image features for subsequent analysis, thereby improving the subsequent analysis efficiency and analysis accuracy of the target sub-image, and when it is determined that there is garbage in the target image, the area where the garbage is located is further determined to improve the accuracy of garbage type identification.

[0148] Example 7

[0149] Based on Example 6, the embodiment of the present invention provides a method for detecting exposed garbage and overflowing garbage based on deep learning, wherein the fifth image area is expanded according to a preset expansion method, and the sixth image area includes:

[0150] Determining an expansion area based on the area of ​​the fifth image area;

[0151] Determining expansion ratios in various directions of the fifth image region based on image features of each piece of garbage in the fifth image region;

[0152] Based on the expansion area and the expansion ratio, the fifth image region is expanded to obtain a sixth image region.

[0153] The beneficial effects of the above-mentioned design scheme are as follows: by determining the expansion area based on the area of ​​the fifth image area, the rationality of the determined area of ​​the sixth image area is ensured; based on the image features of each garbage in the fifth image area, the expansion ratio in each direction of the fifth image area is determined to ensure the comprehensiveness of the image features contained in the determined sixth image area.

[0154] Example 8

[0155] Based on Example 1, the embodiment of the present invention provides a method for detecting exposed garbage and overflowing garbage based on deep learning. In step 3, determining whether the garbage is exposed garbage according to the positional relationship between the garbage and the garbage bin includes:

[0156] Segmenting the target sub-image according to the positions of the garbage and the garbage bin to obtain a plurality of pixel sets;

[0157] Extracting pixel positions from a pixel set, and selecting a first pixel position belonging to the garbage and a second pixel position belonging to the garbage bin from the pixel positions, and determining an image distance between the garbage and the garbage bin according to a positional relationship between the first pixel position and the second pixel position;

[0158] If the image distance is greater than a preset distance, determining that the garbage is exposed garbage;

[0159] Otherwise, obtaining the position relationship between the garbage and the garbage bin;

[0160] The method for obtaining the positional relationship between the garbage and the garbage bin is as follows:

[0161] Based on the relative positions of the garbage and the garbage bin, determining a method for establishing a first coordinate system for the garbage and a method for establishing a second coordinate system for the garbage bin;

[0162] Selecting a first center position and a second center position of the first pixel point position and the second pixel point position as coordinate origins respectively, and establishing a first coordinate system and a second coordinate system by using the first coordinate system establishing method and the second coordinate system establishing method;

[0163] Obtain the first highest point, the first lowest point, the first leftmost point, and the first rightmost point of the trash can in the first coordinate system, and the second highest point, the second lowest point, the second leftmost point, and the second rightmost point of the trash in the second coordinate system;

[0164] Determine a coordinate relationship set between each of the first highest point, the first lowest point, the first leftmost point, the first rightmost point, the second highest point, the second lowest point, the second leftmost point, and the second rightmost point in the first coordinate system and the second coordinate system;

[0165] Inputting the coordinate relationship set into a preset trained image position and orientation recognition model to obtain the orientation relationship between the garbage and the garbage bin;

[0166] Determine the actual distance between the garbage and the garbage bin based on the orientation relationship and the image distance;

[0167] Determining whether the actual distance is greater than a preset actual distance;

[0168] If so, it is determined that the waste is exposed waste;

[0169] Otherwise, it is determined that the garbage is not exposure garbage.

[0170] In this embodiment, the relative positions of the garbage and the garbage can are up, down, left and right. If the garbage is in the lower left position of the garbage can, the first coordinate is established in such a way that the x-axis is to the right, the y-axis is upward, and the z-axis is vertically upward; the second coordinate is established in such a way that the x-axis is to the left, the y-axis is downward, and the z-axis is vertically downward.

[0171] In this embodiment, the target sub-image is a two-dimensional image, and the determined image distance cannot represent the actual distance between the garbage and the trash can. However, when the image distance is greater than the preset distance, even if it cannot represent the accurate actual distance, it can be roughly determined that the range of the actual distance meets the characteristics of exposing the garbage.

[0172] In this embodiment, the pre-trained image position and orientation recognition model is trained based on the relationship features and orientation relationship between points of the first object and the second object in the sample image.

[0173] The beneficial effect of the above design scheme is: by analyzing the image distance, orientation relationship and actual distance between the garbage and the garbage bin in the target sub-image, whether the garbage is exposed garbage is finally determined based on the actual distance. At the same time, when determining the orientation relationship between the garbage and the garbage bin, a dual coordinate system is established, and an image position and orientation recognition model is pre-trained to determine the orientation relationship between the garbage and the garbage bin under the two-dimensional target sub-image, thereby ensuring the accuracy of the actual distance between the garbage and the garbage bin and improving the accuracy of identifying exposed garbage.

[0174] Example 9

[0175] Based on Example 1, the embodiment of the present invention provides a method for detecting exposed garbage and overflowing garbage based on deep learning. In step 4, the steps of the pre-trained second target detection model are as follows:

[0176] Obtain overflowing garbage images from a garbage image database, and classify the overflowing garbage images according to overflow types to obtain multiple sets of overflowing garbage images;

[0177] Obtaining an average image feature in each group of overflowing garbage image sets, and searching the garbage image database for a comparison garbage image set whose difference from the average image feature is within a preset feature difference range;

[0178] Based on each set of overflowing garbage image sets and comparison garbage image sets, multiple sets of sample image sets are established, and a detection image and a comparison image in each set of sample image sets are defined;

[0179] Setting data collection accuracy for each set of sample images based on average similarity between image features of the detection image and image features of the comparison image in each set of sample images;

[0180] According to the data collection accuracy, point cloud data collection is performed on the images in each group of sample image sets to obtain point cloud data;

[0181] and setting a label for each group of sample image sets based on the image type in each group of sample image sets to obtain a labeling result;

[0182] Based on the image features of each set of sample images and the data acquisition accuracy, the image analysis rules of the preset detection model are set to obtain a preliminary detection model;

[0183] Based on the point cloud data and the annotation results, the preliminary detection model is trained to obtain a complete detection model;

[0184] Randomly selecting a preset number of images to be detected from the overflowing garbage image and the comparison garbage image set, inputting them into the complete detection model, and obtaining the accuracy of the detection result;

[0185] If the accuracy rate is greater than a preset probability, determining that the complete detection model is a second target detection model;

[0186] Otherwise, the image analysis rules and model parameters are adjusted until the accuracy is greater than the preset probability, and the latest detection model obtained is used as the second target detection model.

[0187] In this embodiment, the greater the average similarity is, the greater the difficulty in identifying overflowing garbage in the set of sample images, and therefore, the greater the data collection accuracy setting should be.

[0188] In this embodiment, each group of sample image sets includes two types of images, one is a detection image, ie, an overflowing garbage image, and the other is a comparison image, ie, a non-overflowing garbage image.

[0189] In this embodiment, the average image feature in each group of overflowing garbage image sets is obtained by averaging the image features of all overflowing garbage images in the group of overflowing garbage image sets to represent the overall image feature of the group of overflowing garbage image sets.

[0190] In this embodiment, the image features of the detection image represent the features of all the detection images in the set of sample images, and the image features of the comparison image represent the image features of all the comparison images in the set of sample images.

[0191] In this embodiment, the point cloud data includes geometric position information, color information, and reflection intensity information of the data.

[0192] In this embodiment, the image analysis rules are determined by the data acquisition accuracy.

[0193] In this embodiment, the marking results include overflowing garbage and non-overflowing garbage.

[0194] In this embodiment, the overflow types include garbage overflowing above the garbage bin and garbage overflowing around the garbage bin.

[0195] In this embodiment, the adjustment of the model parameters includes adjustment of the output layer weight, the number of hidden layers, the hidden layer weight, the bias, etc.

[0196] In this embodiment, adjusting the image analysis rules includes: determining the adjustment amount for the data acquisition accuracy based on the difference between the accuracy rate and the preset probability, obtaining new image analysis rules according to the adjusted data acquisition accuracy, and adjusting the image analysis rules according to the difference between the accuracy rate and the preset probability, so as to further improve the detection accuracy of the second target detection model.

[0197] The beneficial effects of the above design scheme are: by determining the second target detection model according to the classification of overflowing garbage images and the difficulty of identifying overflowing garbage images, the recognition accuracy of the second target detection model is ensured, and the detection effect of the second target detection model is verified and adjusted to further ensure the recognition accuracy of the second target detection model.

[0198] Example 10

[0199] Based on Example 1, the embodiment of the present invention provides an exposed garbage and overflowing garbage detection method based on deep learning. In step 4, the target sub-image is input into a pre-trained second target detection model, and according to the second target detection result, it is determined whether the garbage is overflowing garbage, including:

[0200] Inputting the target sub-image into a pre-trained second target detection model, and outputting a plurality of output labels for the target sub-image;

[0201] Based on a preset label type, the multiple output labels are divided into two categories to obtain label types of the multiple output labels, and the proportion of the label types that are target types is determined, and the proportion is used as the second target detection result;

[0202] If the proportion of the second target detection results is greater than a preset proportion, determining that the garbage is overflowing garbage;

[0203] Otherwise, it is determined that the garbage is not overflowing garbage.

[0204] In this embodiment, the output labels include two types, target labels and non-target labels.

[0205] In this embodiment, the multiple output labels correspond to the detection results of various parts of the target sub-image respectively.

[0206] The beneficial effect of the above design scheme is: by using the second target detection model to detect various parts of the target sub-image, the second target detection result is comprehensively determined according to the results of multiple output labels, thereby ensuring the accuracy of the second target detection result.

[0207] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for detecting exposed garbage and overflowing garbage based on deep learning, characterized in that: include: Step 1: Train a first target detection model for detecting garbage, and input a target image into the first target detection model to obtain a first target detection result; Step 2: When it is determined that there is garbage according to the first target detection result, the target image is marked as garbage, and a target sub-image is determined according to the garbage marking result; Step 3: Detect whether there is a trash can in the target sub-image, and if not, determine that the trash is exposed trash; If so, determine whether the garbage is exposed garbage based on the positional relationship between the garbage and the garbage bin; Step 4: when it is determined that the garbage is not exposed garbage, the target sub-image is input into a pre-trained second target detection model, and according to the second target detection result, it is determined whether the garbage is overflowing garbage; In step 2, after determining that garbage exists according to the first target detection result, the step further includes correcting the target image, specifically: Inputting the target image into the first target detection model to obtain an initial detection result; Performing feature extraction on the target image to obtain multiple groups of image feature maps; Inputting the multiple groups of image feature maps into the first object detection model in sequence to obtain multiple groups of image detection results; Determining whether the initial detection result is consistent with the detection results of multiple sets of images; If yes, no correction is performed on the target image; Otherwise, the target image features corresponding to the inconsistent image detection results are obtained, and the target image features are corrected based on the initial detection results to obtain a corrected target image.

2. According to claim 1, a method for detecting exposed garbage and overflowing garbage based on deep learning is characterized in that: In step 1, a first object detection model for detecting garbage is trained, including: Acquire sample images captured by the same camera as the target image, divide the images into a first sample image set and a second sample image set, perform a first annotation on the first sample image set, and perform a second annotation on the second sample image set; The detection model is trained using the first sample image set and the second sample image set until the probability that the output result of the detection model conforms to the first annotation and the second annotation is greater than a preset probability, and the training of the detection model is completed to obtain a first target detection model.

3. According to claim 1, a method for detecting exposed garbage and overflowing garbage based on deep learning is characterized in that: In step 2, the step of determining whether there is garbage according to the first target detection result includes: Inputting the target image into the first target detection model to obtain a first target detection result; Obtaining a marked feature in the first target detection result, and determining whether the marked feature is a first mark; If so, it indicates that there is garbage in the target image; Otherwise, it indicates that there is no garbage in the target image.

4. The method for detecting exposed garbage and overflowing garbage based on deep learning according to claim 1, characterized in that: Correcting the target image feature based on the initial detection result to obtain a corrected target image includes: Obtaining obvious image features corresponding to the target detection results, and extracting obvious attribute features from the obvious image features that are consistent with the attributes of the target image features; Based on the difference between the obvious attribute feature and the target image feature, determining a correction part and a correction scheme of the target image feature, and correcting the correction part using the correction scheme to obtain a correction feature map; The corrected feature map and other groups of image feature maps are combined to obtain a corrected target image.

5. The method for detecting exposed garbage and overflowing garbage based on deep learning according to claim 1, characterized in that: In step 2, the target sub-image is determined according to the garbage labeling result, including: According to the garbage marking result, determining the number and location of garbage areas in the target image; If the number of regions is 1, the area of ​​the search region is determined based on the area of ​​the regional location of the garbage; Taking the area position as the center, obtaining the image features of the first image area of ​​the search area, and obtaining the sub-image area corresponding to the image features within the preset image feature range; Reducing a portion of the first image region where no sub-image region exists within a preset range to obtain a second image region; Determine that the sub-image area exists at a boundary of the second image area, and based on a proportion of the sub-image area in the boundary, expand the boundary to obtain a third image area; Determining a target sub-image based on the third image area; If the number of regions is greater than 1, two adjacent garbage locations are connected in pairs with the garbage locations as the boundary to obtain a fourth image region; Based on image features of an image area near the outside of the fourth image area, acquiring a target external image area corresponding to the image features within the preset image feature range; For the connection relationship between the target garbage contained in the target external image region, the positions where the target garbage is located are reconnected according to the image features of the target external image region to obtain a new target external image region; Based on the new target external image region, updating the fourth image region to obtain a fifth image region; Expanding the fifth image region according to a preset expansion method to obtain a sixth image region; Based on the sixth image area, a target sub-image is determined.

6. The method for detecting exposed garbage and overflowing garbage based on deep learning according to claim 5, characterized in that: The fifth image region is expanded according to a preset expansion method to obtain a sixth image region including: Determining an expansion area based on the area of ​​the fifth image area; Based on the image features of each piece of garbage in the fifth image area, determining the expansion ratio in each direction of the fifth image area; Based on the expansion area and the expansion ratio, the fifth image region is expanded to obtain a sixth image region.

7. The method for detecting exposed garbage and overflowing garbage based on deep learning according to claim 1, characterized in that: In step 3, determining whether the garbage is exposed garbage based on the positional relationship between the garbage and the garbage bin includes: Segmenting the target sub-image according to the positions of the garbage and the garbage bin to obtain a plurality of pixel sets; Extracting pixel positions from the pixel set, and selecting a first pixel position belonging to the garbage and a second pixel position belonging to the garbage bin from the pixel positions, and determining an image distance between the garbage and the garbage bin according to a positional relationship between the first pixel position and the second pixel position; If the image distance is greater than a preset distance, determining that the garbage is exposed garbage; Otherwise, obtaining the position relationship between the garbage and the garbage bin; The method for obtaining the positional relationship between the garbage and the garbage bin is as follows: Based on the relative positions of the garbage and the garbage bin, determining a method for establishing a first coordinate system for the garbage and a method for establishing a second coordinate system for the garbage bin; Selecting a first center position and a second center position of the first pixel point position and the second pixel point position as coordinate origins respectively, and establishing a first coordinate system and a second coordinate system by using the first coordinate system establishing method and the second coordinate system establishing method; Obtain the first highest point, the first lowest point, the first leftmost point, and the first rightmost point of the trash can in the first coordinate system, and the second highest point, the second lowest point, the second leftmost point, and the second rightmost point of the trash in the second coordinate system; Determine a coordinate relationship set between each of the first highest point, the first lowest point, the first leftmost point, the first rightmost point, the second highest point, the second lowest point, the second leftmost point, and the second rightmost point in the first coordinate system and the second coordinate system; Inputting the coordinate relationship set into a pre-trained image position and orientation recognition model to obtain the orientation relationship between the garbage and the garbage bin; Determine the actual distance between the garbage and the garbage bin based on the orientation relationship and the image distance; Determining whether the actual distance is greater than a preset actual distance; If so, it is determined that the waste is exposed waste; Otherwise, it is determined that the garbage is not exposure garbage.

8. The method for detecting exposed garbage and overflowing garbage based on deep learning according to claim 1, characterized in that: In step 4, the steps of the pre-trained second target detection model are as follows: Obtain overflowing garbage images from a garbage image database, and classify the overflowing garbage images according to overflow types to obtain multiple sets of overflowing garbage images; Obtaining an average image feature in each group of overflowing garbage image sets, and searching the garbage image database for a comparison garbage image set whose difference from the average image feature is within a preset feature difference range; Based on each set of overflowing garbage image sets and comparison garbage image sets, multiple sets of sample image sets are established, and a detection image and a comparison image in each set of sample image sets are defined; Setting data collection accuracy for each set of sample images based on average similarity between image features of the detection image and image features of the comparison image in each set of sample images; According to the data collection accuracy, point cloud data collection is performed on the images in each group of sample image sets to obtain point cloud data; and setting a label for each group of sample image sets based on the image type in each group of sample image sets to obtain a labeling result; Based on the image features of each set of sample images and the data acquisition accuracy, the image analysis rules of the preset detection model are set to obtain a preliminary detection model; Based on the point cloud data and the annotation results, the preliminary detection model is trained to obtain a complete detection model; Randomly selecting a preset number of images to be detected from the overflowing garbage image and the comparison garbage image set, inputting them into the complete detection model, and obtaining the accuracy of the detection result; If the accuracy rate is greater than a preset probability, determining that the complete detection model is a second target detection model; Otherwise, the image analysis rules and model parameters are adjusted until the accuracy is greater than the preset probability, and the latest detection model obtained is used as the second target detection model.

9. The method for detecting exposed garbage and overflowing garbage based on deep learning according to claim 1, characterized in that: In step 4, the target sub-image is input into a pre-trained second target detection model, and according to the second target detection result, it is determined whether the garbage is overflowing garbage, including: Inputting the target sub-image into a pre-trained second target detection model, and outputting a plurality of output labels for the target sub-image; Based on a preset label type, the multiple output labels are divided into two categories to obtain label types of the multiple output labels, and the proportion of the label types that are target types is determined, and the proportion is used as the second target detection result; If the proportion of the second target detection results is greater than a preset proportion, determining that the garbage is overflowing garbage; Otherwise, it is determined that the garbage is not overflowing garbage.

Images