A method and device for detecting overflow of trash can based on vision

Abstract

The present invention discloses a vision-based trash can overflow detection method and device, the method comprising: training to obtain a target detection model for detecting the location of trash cans; using the trained teacher model to assign pseudo labels to unlabeled data, using real labeled data and pseudo labeled data to train a student model, and training a trash overflow classification model through continuous iteration; inputting the monitoring image to be detected into the target detection model to obtain the location coordinates of each trash can, extracting the feature information of each image according to the location coordinates, and inputting the feature information into the trash overflow classification model to determine whether it is overflowing. The present application proposes a new trash can overflow detection method through the above scheme, first using a target detection algorithm to detect the location of the trash can and then using a classification network to determine whether the trash can is overflowing. Compared with the traditional single detection algorithm, this method can not only achieve higher accuracy, but also is suitable for new types of trash classification bins.

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence, and in particular relates to a method for detecting overflow of a garbage can. Background Art

[0002] With the progress of social civilization, people have higher and higher requirements for urban environment. In the past, it was necessary to monitor whether the garbage bins in the city were overflowing. This not only required a large amount of manual labor, but also led to untimely supervision. If the supervision was not timely, garbage overflow might occur, which would not only affect the city appearance but also pollute the environment. Nowadays, in the context of smart cities, artificial intelligence technology can be used to automatically judge whether the garbage bins are overflowing based on vision, which can not only achieve real-time supervision, but also greatly save the cost of manual supervision.

[0003] Traditional methods for determining whether a trash can is overflowing are simply done through target detection. Considering the different heights and angles of camera installations in real scenes, the method of determining whether the trash is overflowing only through target detection is only applicable to situations where the trash can is obviously overflowing. For most situations where the trash can is visible but not overflowing, false alarms will be generated, and the trash will be considered overflowing and require manual processing. Especially for new types of trash sorting bins, the features of overflowing trash are not obvious. At this time, relying solely on target detection to determine whether it is overflowing requires a large number of samples and annotations as training support, which undoubtedly increases labor costs. Summary of the invention

[0004] A brief summary of embodiments of the present invention is provided below to provide a basic understanding of certain aspects of the present invention. It should be understood that the following summary is not an exhaustive summary of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to provide certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0005] According to one aspect of the present application, a vision-based trash can overflow detection method is provided, which includes:

[0006] Step S1: training a target detection model for detecting the location of a trash can;

[0007] Step S2: Use the trained teacher model to assign pseudo labels to the unlabeled data, use the data with real labels and pseudo label data to train the student model, and through continuous iteration, train the garbage overflow classification model;

[0008] Step S3: Input the monitoring image to be detected into the target detection model of step S1, obtain the position coordinates of each trash can, obtain the center coordinates and width and height of the trash can according to the position coordinates, and obtain the feature information of each image according to the center coordinates of the trash can, input the feature information into the garbage overflow classification model of step S2, and judge whether it is overflowing. The monitoring image to be detected is a trash can image taken by the camera on site.

[0009] Preferably, the characteristic information of each image is the region of interest (ROI) of each image. In machine vision and image processing, the region of interest (ROI) is a region to be processed that is outlined in the form of a box, circle, ellipse, irregular polygon, etc. from the processed image, which is a prior art and will not be described here.

[0010] The above-mentioned garbage overflow classification model is a semi-supervised training module. This method can train a high-precision model without a large amount of manual labeling, which can greatly reduce labor costs.

[0011] Furthermore, in step S1, the target detection model is implemented based on the YOLO algorithm to detect the location of the trash can.

[0012] Furthermore, the step S2 comprises:

[0013] Step S21: Prepare the trash bin dataset D1{(x1,y1),(x2,y2),...,(x n ,y n )}and Where D1 is the data with the real overflow label, which includes the garbage bin overflow label and the garbage bin not overflow label. n Represents the nth labeled sample, the corresponding label is y n , D2 is a massive amount of unlabeled data, Represents the mth unlabeled sample.

[0014] Step S22: Use the data set D1 to train the teacher model T by continuously reducing the cross entropy loss until the model converges. Its network loss function is:

[0015]

[0016] Among them, x n represents the nth sample, x i represents the i-th sample, i = {1, 2, ..., n};

[0017] The network loss function is the basis for judging whether the model has converged. The smaller the calculated value of the loss function, the better the convergence.

[0018] Step S23: Use the teacher model T to generate classification probability and pseudo label for each sample of the unlabeled dataset D2, and filter out samples with classification probability less than the threshold according to the threshold h to generate the dataset D3:

[0019]

[0020] in, For sample The corresponding pseudo label, p i The corresponding classification probability is the output result of each sample input teacher model, and the threshold h can generally be set to 0.5.

[0021] Step S24: performing data enhancement on data sets D1 and D3;

[0022] Step S25: Use the data sets D1 and D3 enhanced in step S24 to train the student model S, by continuously reducing the cross entropy loss until the model converges. The loss function of the network is:

[0023]

[0024] Among them, x i represents the i-th sample;

[0025] Step S26: Take the converged student model S as the teacher model and repeat steps S23 to S25;

[0026] Step S27: After n rounds of iterations, the final garbage overflow classification model S is obtained n . n is the number of times the network converges.

[0027] When in use, the trash can pictures taken by the camera are input into the target detection model to obtain the position coordinates of each trash can, and the center coordinates and width and height of the trash can are obtained according to the position coordinates. The region of interest roi of each picture is obtained according to the center coordinates of the trash can, and the roi is input into the garbage overflow classification model S n , determine whether it is overflowing.

[0028] According to another aspect of the present application, a vision-based trash can overflow detection device is provided, comprising:

[0029] A target detection model acquisition module, which is used to train and obtain a target detection model for detecting the location of a trash can;

[0030] A garbage overflow classification model training module, which uses a trained teacher model to assign pseudo labels to unlabeled data, and uses data with real labels and pseudo label data to train a student model. Through continuous iteration, a garbage overflow classification model is trained;

[0031] The execution detection module is used to input the monitoring image to be detected into the target detection model trained by the target detection model acquisition module, obtain the location coordinates of each trash can, extract the feature information of each image according to the location coordinates, and input the feature information into the garbage overflow classification model trained by the garbage overflow classification model training module to determine whether it is overflowing.

[0032] Furthermore, the garbage overflow classification model training module performs the following operations:

[0033] Step S21: Prepare the trash bin dataset D1{(x1,y1),(x2,y2),...,(x n ,y n )}and Where D1 is the data with the real overflow label, x n Represents the nth labeled sample, the corresponding label is y n , D2 is a massive amount of unlabeled data, Represents the mth unlabeled sample.

[0034] Step S22: Use the data set D1 to train the teacher model T by continuously reducing the cross entropy loss until the model converges. The loss function of the network is:

[0035]

[0036] Step S23: Use the teacher model T to generate classification probability and pseudo label for each sample of the unlabeled dataset D2, and filter out samples with classification probability less than the threshold according to the threshold h to generate the dataset D3:

[0037]

[0038] in, For sample The corresponding pseudo label, p i is the corresponding classification probability.

[0039] Step S24: performing data enhancement on data sets D1 and D3;

[0040] Step S25: Use data sets D1 and D3 to train the student model S by continuously reducing the cross entropy loss until the model converges. The loss function of the network is:

[0041]

[0042] Step S26: Take the converged student model S as the teacher model and repeat steps S23 to S25;

[0043] Step S27: After n rounds of iterations, the final garbage overflow classification model S is obtained n .

[0044] This application proposes a new method for detecting overflowing trash cans through the above scheme. The target detection algorithm is first used to detect the position of the trash can, and then the garbage overflow classification model is used to judge whether the trash can is overflowing. Compared with the traditional single detection algorithm, this method can not only achieve higher accuracy, but also is applicable to new types of garbage classification trash cans. In addition, the method first detects the position of the trash can with the target detection algorithm, and then uses the garbage overflow classification model (classification network) to judge whether the trash can is overflowing. In particular, its garbage overflow classification model is a semi-supervised model training method. This method can automatically generate accurate labels for massive amounts of unlabeled data, and improve the accuracy of the classification model through continuous iteration, thereby greatly improving the classification judgment accuracy of the garbage classification trash can. In summary, the method and device for detecting overflowing trash cans reduce the waste of human resources, handle garbage in a timely manner, and can simultaneously achieve accurate classification detection, ensure the resource value and economic value of garbage, strive to make the best use of materials, and reduce the amount of garbage processed and the use of processing equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention may be better understood by referring to the following description in conjunction with the accompanying drawings, wherein the same or similar reference numerals are used throughout the drawings to represent the same or similar components. The accompanying drawings, together with the following detailed description, are incorporated in and form a part of this specification and are used to further illustrate the preferred embodiments of the present invention and to explain the principles and advantages of the present invention.

[0046] In the figure:

[0047] Figure 1 This is a schematic diagram of a method for classifying overflowing garbage in Embodiment 1 of the present invention;

[0048] Figure 2 Schematic diagram of the training process of the garbage overflow classification model in Example 1 of the present invention;

[0049] Figure 3 This is a flow chart of judging overflow in garbage overflow classification in Embodiment 1 of the present invention;

[0050] Figure 4 Schematic diagram of a vision-based trash can overflow detection device in Example 2 of the present invention. DETAILED DESCRIPTION

[0051] Embodiments of the present invention will be described below with reference to the accompanying drawings. It should be noted that for the purpose of clarity, the drawings and descriptions omit representations and descriptions of components and processes that are not related to the present invention and are known to those skilled in the art.

[0052] Example 1

[0053] As a specific example, see Figure 1 The visual-based trash can overflow detection method of the present invention comprises:

[0054] Step S1: training a target detection model for detecting the location of a trash can;

[0055] Step S2: Use the trained teacher model to assign pseudo labels to the unlabeled data, use the data with real labels and pseudo label data to train the student model, and through continuous iteration, train the garbage overflow classification model;

[0056] Step S3: Input the surveillance image to be detected into the target detection model of step S1 to obtain the position coordinates of each trash can, obtain the center coordinates and width and height of the trash can according to the position coordinates, and obtain the feature information of each image according to the center coordinates of the trash can, and input the feature information into the garbage overflow classification model of step S2 to determine whether it is overflowing.

[0057] It should be noted that, in step S1, the YOLO algorithm is used to train the target detection model for detecting the location of the trash can.

[0058] like Figure 2 As shown, in step S2, the training process of the garbage overflow classification model includes steps S21-S27, which are described below respectively.

[0059] Step S21: Prepare the trash bin dataset D1{(x1,y1),(x2,y2),...,(x n ,y n )}and Where D1 is the data with the real overflow label, x n Represents the nth labeled sample, the corresponding label is y n , D2 is a massive amount of unlabeled data, Represents the mth unlabeled sample.

[0060] Step S22: Use the data set D1 to train the teacher model T by continuously reducing the cross entropy loss until the model converges. The loss function of the network is:

[0061]

[0062] Among them, xn represents the nth sample, x i represents the i-th sample, i = {1, 2, ..., n}.

[0063] Step S23: Use the teacher model T to generate classification probability and pseudo label for each sample of the unlabeled data set D2, and filter out samples with classification probability less than the threshold according to the threshold h to generate the data set D3: where the classification probability is calculated by the following formula:

[0064]

[0065] in, For sample The corresponding pseudo label, p i is the corresponding classification probability.

[0066] Step S24: perform data enhancement on data sets D1 and D3 to increase the data volume of the data sets; data enhancement is a prior art, and its purpose is to increase the data volume. There are many ways to do this. For image data enhancement, it can be achieved through geometric transformation and color transformation, which will not be described in detail here.

[0067] Step S25: Use the data sets D1 and D3 enhanced in step S24 to train the student model S, by continuously reducing the cross entropy loss until the model converges. The loss function of the network is:

[0068]

[0069] Step S26: Take the converged student model S as the teacher model and repeat steps S23 to S25.

[0070] Step S27: After n rounds of iterations, the final garbage overflow classification model S is obtained n .

[0071] As attached Figure 3 As shown, we get the target detection model and garbage overflow classification model S n Then, in step S3, the trash can images taken by the camera are input into the target detection model to obtain the position coordinates of each trash can, the center coordinates and width and height of the trash can are obtained according to the position coordinates, and the region of interest roi of each trash can image is obtained according to the center coordinates of the trash can, and the region of interest roi is input into the garbage overflow classification model S n , determine whether it is overflowing.

[0072] The present application proposes a new method for detecting overflowing trash cans through the above scheme. The method first uses a target detection algorithm to detect the location of the trash can, and then uses a classification network to determine whether the trash can is overflowing. In particular, the garbage overflow classification model of the present invention is a semi-supervised model training method, which can automatically generate accurate labels for massive unlabeled data and improve the accuracy of the classification model through continuous iteration.

[0073] Compared with the prior art, the garbage overflow classification model of the present invention is a semi-supervised training module, which uses a trained teacher model to assign pseudo labels to unlabeled data, and uses data with real labels and pseudo-label data to train a student model. Through continuous iteration, the final classification model is trained. This method does not require a large amount of manual labeling to train a high-precision model, which can greatly reduce labor costs.

[0074] Example 2

[0075] This embodiment provides a visual-based garbage can overflow detection device, see Figure 3 , which includes:

[0076] A target detection model acquisition module 1, which is used to train and obtain a target detection model for detecting the location of a trash can;

[0077] Garbage overflow classification model training module 2, the garbage overflow classification model training module uses the trained teacher model to assign pseudo labels to unlabeled data, uses the data with real labels and the pseudo label data to train the student model, and trains the garbage overflow classification model through continuous iteration;

[0078] Execution detection module 3, which is used to input the monitoring image to be detected into the target detection model trained by the target detection model acquisition module, obtain the location coordinates of each trash can, extract the feature information of each image according to the location coordinates, and input the feature information into the garbage overflow classification model trained by the garbage overflow classification model training module to determine whether it is overflowing.

[0079] Among them, the garbage overflow classification model training module executes steps S1 to S8 in Example 1.

[0080] In addition, compared with the direct detection process of the prior art, the present invention adopts a process of first detection and then classification: first use the target detection algorithm to detect the position of the trash can and then use the classification network to determine whether the trash can is overflowing. Compared with the traditional single detection algorithm, this method can not only achieve higher accuracy, but also is suitable for new types of garbage classification trash cans.

[0081] In addition, the method of the present invention is not limited to being executed in the time sequence described in the specification, and may also be executed in other time sequences, in parallel or independently. Therefore, the execution order of the method described in this specification does not limit the technical scope of the present invention.

[0082] Although the present invention has been disclosed above by describing specific embodiments of the present invention, it should be understood that all the above embodiments and examples are exemplary rather than restrictive. Those skilled in the art may design various modifications, improvements or equivalents of the present invention within the spirit and scope of the appended claims. These modifications, improvements or equivalents should also be considered to be included in the protection scope of the present invention.

Claims

1. A vision-based trash can overflow detection method, characterized in that: include: Step S1: training a target detection model for detecting the location of a trash can; Step S2: Use the trained teacher model to assign pseudo labels to the unlabeled data, use the data with real labels and pseudo label data to train the student model, and through continuous iteration, train the garbage overflow classification model; Step S3: Input the monitored image to be detected into the target detection model of step S1 to obtain the position coordinates of each trash can, obtain the center coordinates and width and height of the trash can according to the position coordinates, and obtain the feature information of each image according to the center coordinates of the trash can, and input the feature information into the garbage overflow classification model of step S2 to determine whether it is overflowing; The step S2 comprises: Step S21: Prepare the trash bin dataset D1{(x1,y1),(x2,y2),...,(x n ,y n )}and Where D1 is the data with the real overflow label, x n Represents the nth labeled sample, the corresponding label is y n , D2 is a massive amount of unlabeled data, Represents the mth unlabeled sample; Step S22: Use the data set D1 to train the teacher model T by continuously reducing the cross entropy loss until the model converges. The loss function of the network is: Among them, x n represents the nth labeled sample, x i represents the i-th labeled sample, i = {1, 2, ..., n}; T is the teacher model, f(x i ,T) represents the i-th labeled sample x i The prediction results obtained after inputting the teacher model, is the network loss function for the i-th sample of the teacher model; Step S23: Use the teacher model T to generate classification probability and pseudo label for each sample of the unlabeled dataset D2, and filter out samples with classification probability less than the threshold according to the threshold h to generate the dataset D3: in, For sample The corresponding pseudo label, p i is the corresponding classification probability; Yes Input to the teacher model T to obtain its classification probability value p i , and then get its corresponding label Step S24: performing data enhancement on data sets D1 and D3; Step S25: Use the data sets D1 and D3 enhanced in step S24 to train the student model S, by continuously reducing the cross entropy loss until the model converges. The loss function of the network is: is the network loss function for the i-th sample of the student model, are pseudo-labeled samples for the student model The network loss function of Step S26: Take the converged student model S as the teacher model and repeat steps S23 to S25; Step S27: After n rounds of iterations, the final garbage overflow classification model S is obtained n .

2. The visual-based trash can overflow detection method according to claim 1, characterized in that: In step S1, the target detection model is implemented based on the YOLO algorithm and is used to detect the location of the trash can.

3. A visual-based trash can overflow detection device, characterized in that: include: A target detection model acquisition module, which is used to train and obtain a target detection model for detecting the location of a trash can; A garbage overflow classification model training module, which uses a trained teacher model to assign pseudo labels to unlabeled data, and uses data with real labels and pseudo label data to train a student model. Through continuous iteration, a garbage overflow classification model is trained; An execution detection module is used to input the monitoring image to be detected into the target detection model trained by the target detection model acquisition module, obtain the position coordinates of each trash can, extract the feature information of each image according to the position coordinates, and input the feature information into the garbage overflow classification model trained by the garbage overflow classification model training module to determine whether it is overflowing; The garbage overflow classification model training module performs the following operations: Step S21: Prepare the trash bin dataset D1{(x1,y1),(x2,y2),...,(x n ,y n )}and Where D1 is the data with the real overflow label, x n Represents the nth labeled sample, the corresponding label is y n , D2 is a massive amount of unlabeled data, Represents the mth unlabeled sample; Step S22: Use the data set D1 to train the teacher model T by continuously reducing the cross entropy loss until the model converges. The loss function of the network is: Among them, x n represents the nth sample, x i represents the i-th sample, i = {1, 2, ..., n}; T is the teacher model, f(x i ,T) represents the i-th sample x i The prediction results obtained after inputting the teacher model, is the network loss function for the i-th sample of the teacher model; Step S23: Use the teacher model T to generate classification probability and pseudo label for each sample of the unlabeled dataset D2, and filter out samples with classification probability less than the threshold according to the threshold h to generate the dataset D3: in, For sample The corresponding pseudo label, p i is the corresponding classification probability; Yes Input to the teacher model T to obtain its classification probability value p i , and then get its corresponding label Step S24: performing data enhancement on data sets D1 and D3; Step S25: Use data sets D1 and D3 to train the student model S by continuously reducing the cross entropy loss until the model converges. The loss function of the network is: is the network loss function for the i-th sample of the student model, are pseudo-labeled samples for the student model The network loss function of Step S26: Take the converged student model S as the teacher model and repeat steps S23 to S25; Step S27: After n rounds of iterations, the final garbage overflow classification model S is obtained n .

Images