A method and device for identifying and classifying delivered garbage based on images

Abstract

The application discloses a method and device for identifying and classifying delivered garbage based on images, wherein the method comprises the following steps: S1: obtaining and preprocessing images before and after delivery; S2: respectively identifying the preprocessed images before and after delivery, extracting the coordinate set features and category features of the regions where the objects on the image before delivery are located, and extracting the coordinate set features and category features of the regions where the objects on the image after delivery are located; S3: respectively extracting the features of the regions where the objects on the image before delivery and the image after delivery are located; and S4: comparing the features on the image before delivery and the image after delivery, and summarizing the newly added regions and categories. The application realizes the identification of the newly added garbage delivery regions and the newly added garbage categories, reduces the cost of manual garbage category labeling, eliminates the unstable output and non-uniform standards caused by manual labeling, and can be widely applied to garbage classification equipment.

Description

Technical Field

[0001] This invention relates to the field of intelligent waste sorting technology, and in particular to a method and apparatus for identifying and classifying waste based on images. Background Technology

[0002] Garbage classification generally refers to a series of activities involving the sorting, storage, disposal, and transportation of garbage according to certain regulations or standards, thereby transforming it into public resources. The purpose of classification is to increase the resource and economic value of garbage, striving to make the most of everything. Implementing garbage sorting and collection can reduce the amount of garbage to be processed and the need for processing equipment, lower processing costs, and reduce the consumption of land resources, thus providing social, economic, and ecological benefits.

[0003] Precise waste sorting and management can significantly improve the utilization rate of recyclable waste. This is because each type of waste has its own different processing technology and recycling value. The finer the granularity of waste sorting, the higher its value. However, the sorting cost will also be higher. How to achieve precise waste sorting while significantly reducing sorting costs is an urgent problem to be solved in this field.

[0004] Chinese invention patent CN110683240A discloses a waste sorting system based on image processing, which includes an image acquisition module that captures images of the waste to be disposed of when a user does so; an image processing module that performs cropping, noise reduction, and brightness adjustment on the waste images obtained from the image acquisition module; an image comparison module that compares the processed waste images with images of waste stored in a waste image database; and a waste sorting module that opens the lid of the corresponding waste bin according to the waste type obtained from the image comparison module and closes the lid after the user has disposed of the waste. Although waste sorting is achieved, there is a problem that when there are many waste images, the system cannot reasonably allocate and process the waste images, resulting in slow waste sorting efficiency.

[0005] Therefore, it is particularly important to explore efficient and low-cost waste recycling devices. Summary of the Invention

[0006] Currently, there is a lack of existing technologies to explore highly efficient and low-cost waste recycling devices.

[0007] To address the aforementioned technical problems, this application discloses a method for identifying and classifying garbage based on images, comprising the following steps: S1: acquiring and preprocessing images before and after garbage disposal; S2: identifying the preprocessed images before and after garbage disposal, extracting the coordinate set features and category features of the areas where items are located in the images before and after garbage disposal; S3: extracting the features of the areas where items are located in the images before and after garbage disposal; S4: comparing the features in the images before and after garbage disposal, and summarizing the newly added areas and categories.

[0008] According to an embodiment of this application, in step S1, the image preprocessing before and after delivery includes scaling, brightness adjustment and enhancement, angle adjustment, stretching transformation, linear transformation and normalization of the image.

[0009] According to the embodiments of this application, in step S2, the image instance segmentation method is used to identify the areas where the items are located in the images before and after delivery. Instance segmentation includes using the depth convolution method to obtain the distribution relationship of pixels, and then using the pattern to segment all complete object shapes in the image. A series of edge points of the object shape constitute a coordinate set, which completely describes the shape features of the object.

[0010] According to an embodiment of this application, step S3 further includes: compressing the feature information of the item location area in the pre-delivery image and the post-delivery image into a multi-dimensional matrix through a convolutional layer, thereby outputting the multi-dimensional feature matrix of the item location area in the pre-delivery image and the post-delivery image respectively.

[0011] According to embodiments of this application, the multidimensional feature matrix includes the RGB color information, brightness variation information, and texture information of the image.

[0012] According to an embodiment of this application, step S4 further includes the following steps:

[0013] S41: Calculate the similarity between the multidimensional feature matrices of the pre-delivery image and the post-delivery image. If the similarity is greater than the threshold, it is considered that the similarity is high and the area where the compared items are located in the post-delivery image is not a new area. If the similarity is less than the threshold, it is considered that the area where the compared items are located in the post-delivery image is a new area and the item category in the new area is a new category.

[0014] S42: If the delivered item does not exist in the image before delivery, then the area where the item is located in the image after delivery is the newly added area, and the item category is the newly added category.

[0015] S43: Summarize the newly added regions and categories obtained in steps S41 and S42 to obtain the final newly added regions and categories.

[0016] According to an embodiment of this application, the features in step S4 include the multidimensional feature matrix, coordinate set features, and category features.

[0017] According to embodiments of this application, the category features are based on the accurate classification of four common types of household waste: paper, metal, plastic, and textiles. The classification results output by the classifier from the image data provide support for the automated classification of household waste.

[0018] Another aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the image-based method for identifying and classifying garbage as described above.

[0019] This application also includes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the image-based method for identifying and classifying garbage as described above.

[0020] The advantages of the present invention over the prior art are:

[0021] 1. This invention utilizes deep learning for automatic image comparison and classification to identify newly added waste disposal areas and waste categories, achieving fine-grained waste category identification. Compared to traditional methods, this invention offers higher efficiency. Previous methods could only classify one image, while this invention can simultaneously compare two images, identify differences, and then classify those differences. This reduces the cost of manual waste category labeling, eliminates the instability and inconsistent standards caused by manual labeling, and improves the overall quality and efficiency of waste sorting.

[0022] 2. By extracting garbage features from images through convolutional kernels, the feature extraction is more thorough than that of traditional machine learning, which is more conducive to guiding classification work; data augmentation enriches sample features and improves model robustness; this method uses a lightweight convolutional neural network model built by ourselves, which has low computational cost and fast operation speed, and through a large number of experiments to optimize and adjust parameters, the accuracy is greatly improved.

[0023] 3. Compared with traditional methods, this invention achieves higher classification accuracy for these four types of household waste. Based on a neural convolutional network, this invention performs convolution operations on the pixel information of an image using convolution kernels to comprehensively extract image features. Then, by calculating the similarity of the feature matrices, it achieves accurate classification of the four common types of household waste, thus providing support for automated household waste classification. Attached Figure Description

[0024] Figure 1 This is a flowchart of a method for identifying and classifying garbage based on images according to the present invention.

[0025] Figure 2 This is a schematic diagram illustrating item category recognition in an example of the present invention;

[0026] Figure 3 This is a schematic diagram of the instance segmentation method in an example of the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0028] Image instance segmentation is a further refinement of object detection, separating the foreground and background of an object to achieve pixel-level object separation. Therefore, image instance segmentation is an improvement upon object detection. It has applications in various scenarios, including object detection, face detection, expression recognition, medical image processing and disease-aided diagnosis, video surveillance and object tracking, and shelf emptying detection in retail settings.

[0029] according to Figure 1 As shown, a method for identifying and classifying garbage based on images includes the following steps:

[0030] S1: Acquire images before and after delivery and perform preprocessing;

[0031] S2: Recognize the pre-delivery image and the post-delivery image after preprocessing, extract the coordinate set features and category features of the area where the item is located in the pre-delivery image, and extract the coordinate set features and category features of the area where the item is located in the post-delivery image;

[0032] S3: Extract the features of the area where the item is located in the images before and after delivery, respectively;

[0033] S4: Compare the features on the images before and after delivery, and summarize the newly added regions and categories.

[0034] In this embodiment, in step S1, the image preprocessing before and after delivery includes scaling, brightness adjustment and enhancement, angle adjustment, stretching transformation, linear transformation and normalization of the image.

[0035] In this embodiment, step S2 uses image instance segmentation to identify the areas where the items are located in the pre-delivery and post-delivery images, respectively; for example Figure 3As shown, objects A and B in the image before delivery are identified. An instance segmentation method is used, employing depthwise convolution to obtain the pattern of pixel distribution. This pattern is then used to segment all complete object shapes within the image. A series of edge points of each object shape constitutes a coordinate set, which fully describes the object's shape features. Similarly, the image after delivery, including objects A and B, as well as newly added items C and D, is identified. Again, an instance segmentation method is used, employing depthwise convolution to obtain the pattern of pixel distribution. This pattern is then used to segment all complete object shapes within the image after delivery. A series of edge points of each object shape constitutes a coordinate set, which fully describes the object's shape features.

[0036] In this embodiment, step S3 further includes: compressing the feature information of the item location area in the pre-delivery image and the post-delivery image into a multi-dimensional matrix through a convolutional layer, thereby outputting the multi-dimensional feature matrix of the item location area in the pre-delivery image and the post-delivery image respectively.

[0037] In this embodiment, the multidimensional feature matrix includes the image's RGB color information, brightness variation information, and texture information.

[0038] In this embodiment, step S4 further includes the following steps:

[0039] S41: Calculate the similarity between the multidimensional feature matrices of the image before delivery and the image after delivery. If the similarity is greater than the threshold, it is considered that the similarity is high and the area where the item is compared in the image after delivery is not a new area. If the similarity is less than the threshold, it is considered that the area where the item is compared in the image after delivery is a new area and the item category in the new area is a new category.

[0040] S42: If the delivered item does not exist in the image before delivery, then the area where the item is located in the image after delivery is the newly added area, and the item category is the newly added category.

[0041] S43: Summarize the newly added regions and categories obtained in steps S41 and S42 to obtain the final newly added regions and categories.

[0042] In this embodiment, the features in step S4 include multidimensional feature matrix, coordinate set features, and category features.

[0043] In this embodiment, the category features are based on the accurate classification of four common types of household waste: paper, metal, plastic and textile. The classification results output by the classifier from the image data support the automated classification of household waste.

[0044] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described method for identifying and classifying garbage based on images.

[0045] This application also includes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method for identifying and classifying garbage based on images.

[0046] like Figure 2 As shown, this application uses a deep learning-based image automatic comparison and classification method to automatically compare two images before and after delivery, and can identify the delivery area newly added in the later image compared to the earlier image (e.g., Figure 2 The system identifies the categories of items to be delivered in the newly added areas (categories of items C and D).

[0047] It is understood that this application describes an electronic device and a method for using the electronic device. The electronic device may include: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory and the processor operate with each other through the computer program. The processor can call logical instructions in the memory to implement a method for identifying and classifying garbage based on images, including the following steps: S1: acquiring and preprocessing images before and after garbage disposal; S2: identifying the preprocessed images before and after garbage disposal respectively, extracting the coordinate set features and category features of the area where the items are located in the image before garbage disposal, and extracting the coordinate set features and category features of the area where the items are located in the image after garbage disposal; S3: extracting the features of the areas where the items are located in the images before and after garbage disposal respectively; S4: comparing the features in the images before and after garbage disposal, and summarizing the newly added areas and categories. Furthermore, the logical instructions in the memory described above can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by the computer, the computer can execute the image-based method for identifying and classifying garbage provided by the above methods. The method includes: S1: acquiring images before and after garbage disposal and performing preprocessing; S2: identifying the preprocessed images before and after garbage disposal respectively, extracting the coordinate set features and category features of the area where the items are located in the image before garbage disposal, and extracting the coordinate set features and category features of the area where the items are located in the image after garbage disposal; S3: extracting the features of the areas where the items are located in the images before and after garbage disposal respectively; S4: comparing the features in the images before and after garbage disposal, and summarizing the newly added areas and newly added categories.

[0049] In summary, the technical solution of this application has the following beneficial effects:

[0050] 1. This invention utilizes deep learning for automatic image comparison and classification to identify newly added waste disposal areas and waste categories, achieving fine-grained waste category identification. Compared to traditional methods, this invention offers higher efficiency. Previous methods could only classify one image, while this invention can simultaneously compare two images, identify differences, and then classify those differences. This reduces the cost of manual waste category labeling, eliminates the instability and inconsistent standards caused by manual labeling, and improves the overall quality and efficiency of waste sorting.

[0051] 2. By extracting garbage features from images through convolutional kernels, the feature extraction is more thorough than that of traditional machine learning, which is more conducive to guiding classification work; data augmentation enriches sample features and improves model robustness; this method uses a lightweight convolutional neural network model built by ourselves, which has low computational cost and fast operation speed, and through a large number of experiments to optimize and adjust parameters, the accuracy is greatly improved.

[0052] 3. Compared with traditional methods, this invention achieves higher classification accuracy for these four types of household waste. Based on a neural convolutional network, this invention performs convolution operations on the pixel information of an image using convolution kernels to comprehensively extract image features. Then, by calculating the similarity of the feature matrices, it achieves accurate classification of the four common types of household waste, thus providing support for automated household waste classification.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying and classifying disposed waste based on images, characterized in that, Includes the following steps: S1: Acquire images before and after delivery and perform preprocessing; S2: Recognize the pre-delivery image and the post-delivery image after preprocessing, extract the coordinate set features and category features of the area where the item is located in the pre-delivery image, and extract the coordinate set features and category features of the area where the item is located in the post-delivery image; S3: Extract the features of the area where the item is located in the images before and after delivery, respectively; S4: Compare the features on the images before and after delivery, and summarize the newly added regions and categories; In step S2, the image instance segmentation method is used to identify the areas where the items are located in the images before and after delivery. The instance segmentation includes using the depth convolution method to obtain the distribution relationship of pixels, and then using the rule to segment all complete object shapes in the image. A series of edge points of the object shape constitute a coordinate set, which completely describes the shape features of the object. Step S3 further includes: compressing the feature information of the item location area in the pre-delivery image and the post-delivery image into a multi-dimensional matrix through a convolutional layer, thereby outputting the multi-dimensional feature matrix of the item location area in the pre-delivery image and the post-delivery image respectively; the multi-dimensional feature matrix includes the RGB color information, brightness variation information and texture information of the image; Step S4 further includes the following steps: S41: Calculate the similarity between the multidimensional feature matrices of the pre-delivery image and the post-delivery image. If the similarity is greater than a threshold, it is considered that the similarity is high and the area where the compared item is located in the post-delivery image is not a new area; if the similarity is less than the threshold, it is considered that the area where the compared item is located in the post-delivery image is a new area, and the item category in the new area is a new category; S42: If the item in the post-delivery image does not exist, then the area where the item is located in the post-delivery image is a new area, and the item category is a new category; S43: Summarize the new areas and new categories obtained in steps S41 and S42 to obtain the final new areas and new categories; The features in step S4 include the multidimensional feature matrix, coordinate set features, and category features.

2. The method for identifying and classifying delivered waste based on images according to claim 1, characterized in that, In step S1, the image preprocessing before and after delivery includes scaling, brightness adjustment and enhancement, angle adjustment, stretching transformation, linear transformation and normalization of the image.

3. The method for identifying and classifying delivered waste based on images according to claim 1, characterized in that, The category features are based on the accurate classification of four common types of household waste: paper, metal, plastic, and textiles. The classification results output by the classifier from the image data support the automated classification of household waste.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for identifying and classifying garbage based on images as described in any one of claims 1 to 3.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for identifying and classifying garbage based on images as described in any one of claims 1 to 3.

Images