Delivery garbage identification method, program product, electronic equipment and storage medium

Abstract

The invention provides a delivered garbage identification method, a program product, electronic equipment and a storage medium. The method comprises the steps that an image of a garbage can before garbage delivery and an image of the garbage after garbage delivery are collected; performing channel combination on the image before garbage delivery and the image after garbage delivery to obtain a channel combination image; segmenting the image after garbage delivery to obtain a garbage mask; the garbage mask comprises an overall garbage area mask and a garbage category mask; performing instance segmentation on the image after garbage delivery to obtain a garbage instance mask; inputting the channel combination image into a difference detection model to obtain a newly added garbage area mask; and on the basis of the overall garbage area mask, the garbage category mask, the garbage instance mask and the newly-added garbage area mask, identifying newly-delivered garbage to obtain an identification result. Through multi-model cooperation and time sequence image analysis, the accuracy and robustness of garbage classification are significantly improved.

Description

Technical Field

[0001] This application relates to the field of waste identification, and more specifically, to a waste identification method, program product, electronic device, and storage medium. Background Technology

[0002] Waste sorting is an important measure for environmental protection and resource recycling. Traditional methods rely on manual supervision, where staff are stationed at garbage bins to guide residents on proper sorting. However, this method is labor-intensive, has limited coverage, and is difficult to monitor continuously. Some solutions use artificial intelligence to identify the waste disposed of by users, but this method is prone to misjudgment due to changes in lighting, angle, and other factors, resulting in low accuracy. Summary of the Invention

[0003] The purpose of this application is to provide a method, program product, electronic device and storage medium for identifying garbage, in order to improve the above-mentioned problems and improve the accuracy of identification.

[0004] In a first aspect, embodiments of this application provide a method for identifying disposed waste, comprising: acquiring images of waste in a trash can before and after disposal; combining the images of waste before and after disposal into channels to obtain a combined channel image; segmenting the image of waste after disposal to obtain a waste mask; the waste mask includes an overall waste region mask and a waste category mask; performing instance segmentation on the image of waste after disposal to obtain a waste instance mask; inputting the combined channel image into a difference detection model to obtain a newly added waste region mask; and identifying newly disposed waste based on the overall waste region mask, the waste category mask, the waste instance mask, and the newly added waste region mask to obtain an identification result. In the aforementioned implementation process, multi-model collaboration and temporal image analysis significantly improved the accuracy and robustness of waste sorting. The difference detection model effectively avoided misjudgments caused by waste displacement or changes in lighting. The combined use of semantic segmentation and instance segmentation made waste category identification more refined, capable of distinguishing individual instances and calculating their proportions. This reduced the need for manual supervision and improved identification efficiency. Furthermore, by using overall waste area masks, waste category masks, waste instance masks, and newly added waste area masks, multiple pieces of information were used to identify newly delivered waste, considering more comprehensive factors and thus resulting in more accurate identification results.

[0005] Optionally, in this embodiment of the application, the images before and after garbage disposal are combined by channels to obtain a channel-combined image, including: taking the images before and after garbage disposal as three-channel image blocks respectively; the three channels include R channel, G channel and B channel; and superimposing the three-channel image blocks corresponding to the images before and after garbage disposal along the channel dimension to generate a channel-combined image.

[0006] In the above implementation process, by overlaying the images before and after delivery along the channel dimension, a six-channel combined image is generated, providing the most original and complete information source for subsequent deep learning models. The model can automatically learn the complex and nonlinear relationship between the scene before and after delivery through the combined channel image, rather than relying on manually preset difference thresholds. This enhances robustness to complex situations such as lighting fluctuations, shadow interference, and garbage deformation, thereby improving the accuracy and reliability of difference detection.

[0007] Optionally, in this embodiment of the application, segmenting the image after garbage disposal to obtain a garbage mask includes: using an image segmentation model to segment the image after garbage disposal to obtain an overall garbage region mask; the image segmentation model is used to identify garbage regions and non-garbage regions; using a semantic segmentation model to segment the image after garbage disposal to obtain a garbage category mask; the semantic segmentation model is used to identify the category of garbage in the image after garbage disposal.

[0008] In the above implementation process, the image segmentation model performs precise binary segmentation on the post-garbage disposal image, effectively eliminating complex background interference, such as the inner wall of the garbage can, reflections, or shadows, thus purely extracting the pixel regions occupied by all the garbage. Through a semantic segmentation model, fine-grained classification of garbage pixels in the image is achieved, accurately distinguishing multiple garbage categories. Pixel-level classification capability not only tells us "where the garbage is," but also clearly tells us "what kind of garbage it is," providing crucial data support for subsequent calculations of the precise proportion of various types of garbage and identification of incorrectly disposed garbage categories, greatly improving the precision and accuracy of garbage classification quality judgment.

[0009] Optionally, in this embodiment of the application, performing instance segmentation on the post-garbage disposal image to obtain a garbage instance mask includes: inputting the post-garbage disposal image into a pre-trained instance segmentation model, using the instance segmentation model to perform instance segmentation on the post-garbage disposal image to obtain a garbage instance mask; the instance segmentation model is used to identify and segment different garbage individuals in the post-garbage disposal image; the instance segmentation model includes a multi-scale image feature extraction network, an encoder, and a decoder; the decoder includes a mask attention module and a multi-level decoding layer, the mask attention module is used to update local features within the query mask region; the multi-level decoding layer is used to receive image features of the corresponding scale output by the multi-scale image feature extraction network and fuse the multi-scale image features.

[0010] In the above implementation process, the instance segmentation model achieves accurate identification and segmentation of each individual piece of waste in the image. The model's built-in multi-scale feature extraction network and fusion mechanism effectively enhance the ability to identify waste of different sizes, reducing the problem of missing small targets. The mask attention module, by focusing on local features, not only accelerates model convergence but also generates instance masks with more accurate boundaries. This pixel-level instance discrimination capability fundamentally avoids misclassifying multiple adjacent or adhered pieces of waste as a single entity, providing a crucial data foundation for accurately calculating newly added waste areas and analyzing the specific number and proportion of individuals under each category. This comprehensively improves the recognition accuracy and analytical precision of the entire waste classification quality assessment system in complex real-world scenarios.

[0011] Optionally, in this embodiment of the application, newly delivered waste is identified based on an overall waste area mask, a waste category mask, a waste instance mask, and a newly added waste area mask to obtain identification results. This includes: determining a preliminary newly added area based on the overall waste area mask and the newly added waste area mask; fusing the preliminary newly added area with the waste instance mask to identify at least one newly delivered waste instance within the preliminary newly added area; determining the waste category of the waste instance based on the waste category mask; and determining the proportion of waste of the same category based on the proportion of pixel information of waste instances of the same category in the preliminary newly added area.

[0012] In the above implementation process, multi-level data fusion and cross-validation of the overall waste area, newly added areas, instance segmentation, and semantic segmentation results significantly improve the accuracy and robustness of waste classification. Logical AND operations effectively eliminate interference from non-waste change areas, improving the purity of the analysis target; associated instance masks enable precise tracking of each individual waste item in this delivery; finally, pixel-level category and area statistics yield the proportion of each category. This reduces problems such as misclassifying residual waste as part of this delivery or misclassifying multiple sticky waste items.

[0013] Optionally, in this embodiment of the application, determining the initial new area based on the overall waste area mask and the newly added waste area mask includes: determining the initial new area based on the intersection of the overall waste area mask and the newly added waste area mask; wherein, if the intersection is empty, the entire newly added waste area mask is taken as the initial new area; the new waste area does not include the area corresponding to the original waste after transformation; the original waste refers to the waste originally contained in the waste bin before the waste was disposed of; the transformation includes at least one of displacement, deformation and flipping.

[0014] In the above implementation process, false new signals generated after the transformation of existing waste can be identified and eliminated: the difference detection model, through a deep learning mechanism, can distinguish between genuine new waste and morphological changes in existing waste. For displacement cases, the model analyzes instances to identify that it is a change in the position of the same object rather than the addition of a new object. For deformation cases, the model uses instance analysis and category analysis, among other invariant features, to determine that it is a change in the shape of an existing object. None of these transformation cases will be incorrectly identified as newly added waste areas.

[0015] Optionally, in this embodiment, the identification result includes the newly disposed waste category and the proportion of each waste category; after obtaining the identification result, the method further includes: matching each newly disposed waste category with the category of the waste bin according to the disposal mapping table; if the matching fails, it means that the waste disposal is incorrect, and obtaining the waste category and proportion of the incorrect waste disposal; the disposal mapping table is used to represent the waste bin category that each waste category needs to be disposed of in; the score of this new disposal is determined according to the proportion of incorrect waste disposal and the weight coefficient; the weight coefficient is dynamically adjusted according to the waste category and / or the category of the waste bin where the waste disposal is incorrect.

[0016] In the aforementioned implementation process, a refined and intelligent waste sorting quality evaluation system is constructed by introducing a delivery mapping table and dynamic weighting coefficients. The delivery mapping table provides a benchmark for judging accuracy and consistency, improving the correctness of sorting. The introduction of dynamic weighting coefficients enables the system to intelligently distinguish the severity of different delivery errors, reducing the limitations of a "one-size-fits-all" scoring method. More punitive scores are applied to delivery errors that are highly harmful or polluting, thereby guiding users to pay more attention to key sorting steps. This scoring mechanism, by quantitatively reflecting delivery quality and linking it with reward and punishment mechanisms, effectively promotes users' awareness of waste sorting and the standardization of their behavior, improving the accuracy of waste sorting and overall management efficiency from a results-oriented perspective.

[0017] Optionally, in this embodiment, before inputting the channel combination image into the difference detection model to obtain the mask of the newly added garbage area, the method further includes: acquiring paired images; the paired images include temporally consecutive pre-delivery sample images and post-delivery sample images; pixel annotation is performed on each pair of paired images to generate an annotated image carrying the ground truth mask of the newly added garbage area; wherein the first value of the pixel annotation represents the newly added garbage area in this delivery, and the second value represents the original garbage area; the original garbage area includes the unchanged area and the area corresponding to the original garbage after transformation; the initial difference detection model is trained using the annotated images through a loss function to obtain a trained difference detection model; wherein the loss function includes an invariance loss, which is used to constrain the output of the difference detection model to remain consistent for the same image pair under different illumination enhancements; the initial difference detection model includes a two-stream Siamese network and a cross-attention module; the two-stream Siamese network is used to extract image features of the pre-delivery sample image and the post-delivery sample image, respectively; the cross-attention module is used to calculate the difference between the paired images.

[0018] In the above implementation process, labeled images are used to train the initial difference detection model, improving its performance and robustness in complex real-world environments. Pixel-level labeled ground truth masks provide accurate supervision signals for model learning, enabling the model to precisely distinguish between genuine new waste and existing waste with morphological changes. Introducing an invariant loss term effectively enhances the model's adaptability to lighting changes, mitigating false detections caused by ambient light fluctuations. A two-stream twin network structure improves the consistency of feature extraction from images before and after delivery, while a cross-attention mechanism delves into subtle differences between paired images. This comprehensive training strategy ensures that the final difference detection model maintains stable high performance under various challenging scenarios, providing a reliable new area detection capability for the entire waste sorting quality assessment system, thereby improving classification accuracy.

[0019] Optionally, in this embodiment of the application, after performing instance segmentation on the post-garbage delivery image to obtain a garbage instance mask, the method further includes: emitting infrared bands to the region corresponding to the garbage instance mask using an infrared analysis sensor and receiving reflectance spectral signals; comparing the reflectance spectral signals with a spectral database to obtain the predicted category and confidence level corresponding to the garbage instance mask; and identifying newly delivered garbage based on the garbage mask, the garbage instance mask, and the newly added garbage region mask, including: identifying newly delivered garbage based on the garbage mask, the garbage instance mask, the newly added garbage region mask, and the predicted category and confidence level corresponding to the garbage instance mask.

[0020] In the aforementioned implementation process, the introduction of infrared spectroscopy, a material identification technology, adds a verification dimension independent of visible light image analysis to the waste sorting system, significantly improving the accuracy and reliability of category identification. Infrared spectroscopy can directly detect the molecular structure and chemical composition of substances. This identification method, based on the essential properties of matter, reduces the misjudgment problems caused by external factors such as color changes, surface contamination, and shape deformation in visible light identification. It can even provide reliable discrimination criteria when distinguishing between different materials that are extremely similar in appearance (such as different types of plastics).

[0021] Optionally, in this embodiment of the application, after determining the initial newly added area, the method further includes: cropping the minimum bounding rectangle region corresponding to the mask of the newly added garbage area from the garbage delivery image; inputting the minimum bounding rectangle region into the semantic segmentation model for secondary inference to obtain the inference result; and determining the garbage category of the garbage instance based on the inference result.

[0022] In the aforementioned implementation process, targeted region cropping and secondary inference mechanisms significantly improved the accuracy and reliability of waste category identification. By extracting the minimum bounding rectangle of newly added waste regions and performing specialized processing, interference from complex backgrounds and irrelevant areas in the original image was effectively eliminated, allowing the semantic segmentation model to focus on the detailed features of the target region. This focused processing approach is beneficial for identifying smaller waste instances or objects with inconspicuous appearance features, reducing identification failures caused by targets that are too small or located in remote areas. The secondary inference process runs on higher-quality, more concentrated input data, enabling it to capture subtle feature differences that might be overlooked in the initial global analysis, thereby providing higher confidence and accuracy in category discrimination.

[0023] Secondly, embodiments of this application also provide a waste disposal identification device, comprising: an image acquisition module for acquiring images of the waste bin before and after disposal; a combination module for combining the images of the waste before and after disposal into channels to obtain a channel-combined image; a segmentation module for segmenting the image of the waste after disposal to obtain a waste mask; the waste mask includes an overall waste area mask and a waste category mask; an instance segmentation module for performing instance segmentation on the image of the waste after disposal to obtain a waste instance mask; a difference detection module for inputting the channel-combined image into a difference detection model to obtain a newly added waste area mask; and an identification module for identifying newly disposed waste based on the overall waste area mask, the waste category mask, the waste instance mask, and the newly added waste area mask to obtain an identification result.

[0024] Thirdly, embodiments of this application also provide a computer program product, including computer program instructions, which are executed by a processor to perform the method provided in the first aspect or any implementation thereof.

[0025] Fourthly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing computer program instructions, which are executed by the processor to perform the method provided in the first aspect or any implementation thereof.

[0026] Fifthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the method provided in the first aspect or any implementation thereof.

[0027] The waste sorting method, program product, electronic device, and storage medium provided in this application significantly improve the accuracy and robustness of waste sorting through multi-model collaboration and temporal image analysis. The difference detection model effectively avoids misjudgments caused by waste displacement or changes in lighting. The combined use of semantic segmentation and instance segmentation makes waste category identification more refined, capable of distinguishing individual instances and calculating their proportions. This reduces the need for manual supervision and improves identification efficiency. Furthermore, by using overall waste area masks, waste category masks, waste instance masks, and newly added waste area masks, multiple pieces of information are used to identify newly disposed waste, considering more comprehensive factors and thus resulting in more accurate identification results. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a waste identification method provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0032] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0033] Traditional methods for identifying waste sorting involve using cameras to capture static images of waste after it has been disposed of, and then classifying each pixel in the image to more precisely distinguish waste categories such as kitchen waste, recyclables, hazardous waste, and other waste.

[0034] Research has found that this traditional method of identifying trash, which analyzes a single image after disposal, cannot distinguish which trash in the bin is newly disposed and which is leftover old trash, resulting in historical interference in the identification results.

[0035] Furthermore, the act of disposing of trash cans can cause displacement, overlap, or deformation of the trash. For example, if a water bottle was originally in the trash can, it might be knocked off by new trash during disposal, causing it to shift in position or even be crushed. Traditional solutions, relying solely on images after disposal for recognition, are prone to misclassifying the displaced and deformed original water bottle as newly disposed trash. These issues render traditional solutions inaccurate in real-world scenarios, making them unsuitable for automated supervision.

[0036] This application provides a method for identifying disposed waste. Through multi-model collaboration and temporal image analysis, it significantly improves the accuracy and robustness of waste classification. The difference detection model effectively avoids misjudgments caused by waste displacement, deformation, or changes in lighting. The combined use of semantic segmentation and instance segmentation makes waste category identification more refined, capable of distinguishing individual instances and calculating their proportions. This reduces the need for manual supervision and improves identification efficiency. Furthermore, by using overall waste area masks, waste category masks, waste instance masks, and newly added waste area masks, multiple pieces of information are used to identify newly disposed waste, considering more comprehensive factors and thus resulting in more accurate identification results.

[0037] Please see Figure 1The illustration shows a flowchart of a waste disposal identification method provided in an embodiment of this application. The waste disposal identification method provided in this application can be applied to a monitoring and identification system. This monitoring and identification system may include a camera. The camera can be installed inside or on top of a trash can, aimed at the waste disposal area, and its shooting range can cover the interior space of the trash can. The monitoring and identification system also includes a computing unit, which can be located inside the trash can or in an identification server. For example, the camera transmits the captured images to the identification server, and the identification server performs waste disposal identification based on the received images. This waste disposal identification method may include: Step S110: Collect images of the trash can before and after trash is disposed of.

[0038] Step S120: Combine the images before and after garbage disposal to obtain a combined channel image.

[0039] Step S130: Segment the image after garbage disposal to obtain the garbage mask; the garbage mask includes the overall garbage area mask and the garbage category mask.

[0040] Step S140: Perform instance segmentation on the image after garbage disposal to obtain the garbage instance mask.

[0041] Step S150: Input the channel combined image into the difference detection model to obtain the mask of the newly added garbage region.

[0042] Step S160: Based on the overall waste area mask, waste category mask, waste instance mask, and newly added waste area mask, identify the newly delivered waste and obtain the identification result.

[0043] In step S110, the image before garbage disposal refers to a static image of the interior of the garbage bin before the user disposes of the garbage. At this time, there may already be some garbage remaining in the garbage bin. The image after garbage disposal refers to the image of the interior of the garbage bin after the user completes the disposal action. New garbage has been added, which may cause garbage to pile up, shift, or deform. It can be understood that the image before garbage disposal changes dynamically after the garbage disposal action. For example, before the third garbage disposal, the image after the second garbage disposal will be used as the image before garbage disposal.

[0044] As one implementation method, motion sensors, weight sensors, and other devices can be used to determine whether new trash has been added. For example, when a user approaches the trash can, a motion sensor (such as an infrared sensor) detects a signal and can capture an image of the trash before it is disposed of. Of course, the image of the trash before disposal can also be taken and saved during the previous trash disposal and directly acquired and used after the next trash disposal is detected. After the user's disposal action is completed, the weight sensor detects the change in the weight of the trash can, or a delay of 2-3 seconds is set to wait for the trash to stabilize before automatically capturing an image of the trash after disposal.

[0045] Another method is to install waste disposal recognition devices near trash cans, which detect human movements to determine if waste has been disposed of. After confirming successful waste disposal, an image of the disposed waste is captured.

[0046] In step S120, channel combining is an image preprocessing technique that stitches the channel dimensions of two RGB images into a multi-channel image. Each pixel in an RGB image has 3 channels (red, green, and blue), and the image before and after delivery each has 3 channels, resulting in a 6-channel image after combining.

[0047] Channel combining can be achieved by sequentially merging the three channels of the image before delivery with the three channels of the image after delivery to generate a tensor of shape [H, W, 6] (where H is the image height and W is the width). Channel combining can also be done at the feature level, for example, by first extracting feature maps from two images using a convolutional network and then concatenating them along the feature dimension.

[0048] In an optional embodiment, the two images can be normalized in size and pixel value before channel combining. The advantage of channel combining images is that it preserves the spatiotemporal information before and after delivery, allowing subsequent difference detection models to directly learn the correlation features between the two images, rather than relying on simple pixel differences, thereby reducing false detections caused by changes in lighting, shadows, or deformation or displacement of debris.

[0049] In step S130, segmenting the post-garbage disposal image refers to pixel-level classification of the image based on a deep learning model. Classification can be based on separating garbage and non-garbage areas in the post-garbage disposal image, or it can be based on segmenting each distinct garbage instance in the post-garbage disposal image. For example, an image segmentation model (such as DeepLabv3) can be used to process the post-disposal image to obtain an overall garbage area mask. This overall garbage area mask can be a binary mask, where each pixel value is either 0 or 1. 1 indicates that the pixel belongs to garbage (regardless of category), and 0 indicates that it is not garbage (such as the garbage bin wall or background).

[0050] The waste category mask is a multi-category mask, where each pixel value represents a specific waste category (such as food waste, recyclables, hazardous waste, etc.). The segmentation process involves inputting the image of the waste after disposal into a semantic segmentation model, which then performs image segmentation to obtain the waste category mask. The semantic segmentation model is used to identify the category of waste in the image after disposal.

[0051] In step S140, instance segmentation not only identifies pixel categories but also distinguishes different object instances. The garbage instance mask is an integer value mask, and each instance's corresponding region mask is assigned a unique ID (e.g., 1, 2, 3...). Pixel values ​​for the same instance are identical, while background pixels are 0. For example, if the image after delivery contains two bottles and a cardboard box, instance segmentation will generate three independent mask regions. Each independent mask region has the same pixel value, and these three independent mask regions correspond to each object; for example, the region with garbage instance mask 1 corresponds to the first bottle, the region with garbage instance mask 2 corresponds to the second bottle, and the region with garbage instance mask 3 corresponds to the cardboard box.

[0052] As one implementation method, instance segmentation can use the Mask2Former model, which can be a Transformer-based architecture that generates instance masks through a query mechanism. The garbage-delivered image is input into the Mask2Former model, which extracts features based on a Swin-L backbone network and then generates a set of binary masks and class labels through a Transformer decoder. Each mask region represents an instance, and the model optimizes mask accuracy through dynamic masking.

[0053] In step S150, the difference detection model is used to identify the changed areas in the image before garbage disposal and the image after garbage disposal, and outputs a mask for the newly added garbage area. For example, the mask for the newly added garbage area can be a binary mask, which can use 0 or 1 to distinguish between the original garbage area and the newly added garbage area mask, or different colors can be used to distinguish between the original garbage area and the newly added garbage area mask.

[0054] In step S160, as one implementation, a preliminary newly added region can be determined based on the overall waste region mask and the newly added waste region mask. For example, the newly added waste region mask and the overall waste region mask are multiplied pixel by pixel to obtain the preliminary newly added region. After the multiplication operation, only the newly added part within the waste region needs to be considered. Then, the preliminary newly added region is fused with the waste instance mask to identify at least one newly delivered waste instance within the preliminary newly added region. The purpose of this step is to determine the category label for the pixels in the preliminary newly added region. Pixels with the same value belong to the same waste instance (e.g., the same bottle).

[0055] Then, based on the number of pixels occupied by the instance, the area and category proportion of the instance can be calculated. The category proportion is calculated by dividing the number of pixels of the instance by the total number of newly added garbage area masks. The recognition results include the area proportion of each category in the newly added garbage, the category and location of each instance, and a quality score based on the proportion (e.g., when disposed of in the food waste bin, a low score is given if the proportion of non-food waste exceeds a threshold).

[0056] In the implementation of the above embodiments: through multi-model collaboration and temporal image analysis, the accuracy and robustness of waste sorting are significantly improved. The difference detection model effectively avoids misjudgments caused by waste displacement or changes in lighting. The combined use of semantic segmentation and instance segmentation makes waste category identification more refined, capable of distinguishing individual instances and calculating proportions. This reduces the need for manual supervision and improves identification efficiency. Simultaneously, by using overall waste area masks, waste category masks, waste instance masks, and newly added waste area masks, multiple pieces of information are used to identify newly delivered waste, considering more comprehensive factors and thus resulting in more accurate identification results.

[0057] The purpose of channel combining is to merge spatially aligned color images acquired at two different time points (before and after delivery) into a single data entity containing richer information for subsequent difference detection model analysis.

[0058] Optionally, in this embodiment of the application, the images before and after garbage disposal are combined by channel compositing to obtain a channel-composited image, including: Images before and after garbage disposal are represented as three-channel image patches; these three channels include the R channel, G channel, and B channel. A color image is typically constructed by superimposing these three color channels. The channel value (usually an integer from 0 to 255) represents the intensity of that point in the corresponding color channel. Therefore, a color image with a resolution of H (height) × W (width) can be represented in a computer as a three-dimensional array or tensor: H × W × 3. Here, "image patch" refers to such a complete three-dimensional tensor data.

[0059] Before combining the images, the pre-delivery and post-delivery images can be spatially aligned. This means that the static backgrounds (such as the walls of a trash can) in both images should overlap, which can be achieved, for example, by fixing the camera and fine-tuning it using a feature point matching algorithm. The two images are then uniformly scaled to the same input size required by the model. To facilitate neural network processing, pixel values ​​are typically normalized to a floating-point range of [0, 1].

[0060] Along the channel dimension, the three-channel image blocks corresponding to the pre-garbage disposal image and the post-garbage disposal image are superimposed to generate a channel-combined image. For example, the three channels of the pre-garbage disposal image are stitched together with the three channels of the post-garbage disposal image to generate a channel-combined image. After the above stitching operation, the two original independent 3-channel images are now transformed into a single 6-channel image, i.e., a channel-combined image, with a tensor shape of H×W×6. For example, the channel order of the channel-combined image can be: Channel 0: The R channel of the image before garbage disposal; Channel 1: G channel of the image before garbage disposal; Channel 2: The B channel of the image before garbage disposal; Channel 3: R channel of the image after garbage disposal; Channel 4: G channel of the image after garbage disposal; Channel 5: The B channel of the image after garbage disposal.

[0061] In the implementation of the above embodiments: by superimposing the images before and after delivery along the channel dimension, a six-channel combined image is generated, providing the most original and complete information source for subsequent deep learning models. The model can automatically learn the complex and nonlinear relationship between the scene before and after delivery through the channel combined image, rather than relying on manually preset difference thresholds, thereby enhancing robustness to complex situations such as illumination fluctuations, shadow interference, and garbage deformation, and thus improving the accuracy and reliability of difference detection.

[0062] Optionally, in this embodiment of the application, segmenting the image after garbage disposal to obtain a garbage mask includes: An image segmentation model is used to segment images after garbage disposal to obtain an overall garbage area mask; the image segmentation model is used to identify garbage areas and non-garbage areas.

[0063] As one implementation, the image segmentation model uses ResNet-50 or ResNet-101 as the backbone network, extracting deep features from the image through multi-layer convolutional operations. The feature maps are then fed into the ASPP (Atriovented Spatial Pyramid Pooling) module, which performs convolutions in parallel using different sampling intervals, thereby capturing multi-scale contextual information of garbage regions in the image without reducing resolution. The model decoder upsamples the fused multi-scale feature maps to restore them to the size of the input image.

[0064] The image segmentation model outputs a global garbage region mask, which can be a tensor of shape H×W×2. Here, "2" corresponds to the original scores for two categories: category 0 (non-garbage) and category 1 (garbage). The Softmax function can be used to convert the original scores for both categories into probability values, ensuring that the sum of the probabilities of each pixel in both categories is 1. A preset threshold is obtained; if the probability value of the "garbage" category is greater than or equal to this threshold, the value at that position in the output mask matrix is ​​set to 1 (representing a garbage region); otherwise, it is set to 0 (representing a non-garbage region).

[0065] A semantic segmentation model is used to segment images after garbage disposal to obtain garbage category masks; the semantic segmentation model is used to identify the category of garbage in the images after garbage disposal.

[0066] The semantic segmentation model also uses the DeepLabV3 architecture. However, unlike the image segmentation model, the semantic segmentation model is a multi-class classifier. It is trained to distinguish the categories of junk pixels in an image.

[0067] Similar to image segmentation models, the forward propagation and feature extraction in semantic segmentation models also involve multi-scale feature extraction and fusion via a ResNet backbone and ASPP module. However, this model is trained on a dataset containing multiple garbage labels, and its learned feature representations are optimized for subtle differences (such as color, texture, and shape) among different garbage categories.

[0068] The model's final output is a tensor of shape H×W×N, where N represents the number of garbage categories, for example, 13 categories. This tensor contains the raw scores for each pixel belonging to all 13 categories. The Softmax function can then be used to convert these raw scores into probability values. For each pixel, its probability values ​​across all 13 categories are compared, and the index (an integer from 0 to 12) of the category with the highest probability value is selected as the final value for that pixel in the output mask.

[0069] For example, if a pixel has the highest probability of belonging to the "kitchen waste" category, that pixel will be marked as the category index value (e.g., 1) corresponding to "kitchen waste" in the final waste category mask.

[0070] One implementation method for training a multi-class semantic segmentation model is to use the AdamW optimizer, combined with linear warm-up and stepped decay strategies. The loss function employs a combined loss function, including a Focal Loss for classification and a combination of Dice Loss and BCE Loss for mask prediction, with a lower weight for classification loss to balance the task. Data augmentation strategies include resizing, random flipping, and pruning. Class weights can also be adjusted during training to address the issue of imbalanced sample sizes across different garbage categories. These strategies enable the model to effectively identify rare garbage categories as well.

[0071] In the implementation of the above embodiments: The image segmentation model performs precise binary segmentation on the image after garbage disposal, effectively eliminating complex background interference, such as the inner wall of the garbage can, reflections, or shadows, thereby purely extracting the pixel areas occupied by all the garbage. Through a semantic segmentation model, refined classification of garbage pixels in the image is achieved, accurately distinguishing various garbage categories. Pixel-level classification capability not only tells us "where the garbage is," but also clearly tells us "what kind of garbage it is," thus providing crucial data support for subsequent calculations of the accurate proportion of various types of garbage and identification of incorrectly disposed garbage categories, greatly improving the precision and accuracy of garbage classification quality judgment.

[0072] Optionally, in this embodiment of the application, instance segmentation is performed on the image after garbage disposal to obtain a garbage instance mask, including: The image after garbage disposal is input into a pre-trained instance segmentation model. The instance segmentation model is used to segment the image after garbage disposal to obtain garbage instance masks. The instance segmentation model is used to identify and segment different garbage individuals in the image after garbage disposal.

[0073] The instance segmentation model includes a multi-scale image feature extraction network, an encoder, and a decoder. The decoder includes a mask attention module and a multi-level decoding layer. The mask attention module is used to update local features within the query mask region. The multi-level decoding layer is used to receive image features of the corresponding scale output by the multi-scale image feature extraction network and fuse the multi-scale image features.

[0074] For example, an image of a garbage dumped is input into a pre-trained instance segmentation model. The image is then processed by a multi-scale image feature extraction network, which uses a hierarchical structure to output a set of multi-scale feature maps. These feature maps have the same semantic information but different spatial resolutions, thus simultaneously containing rich global contextual information and fine local detail information. The local detail information is helpful for locating the boundaries of small objects.

[0075] The multi-level decoding layer receives feature maps of corresponding scales from the multi-scale features generated by the backbone network as input, and enhances the model's ability to recognize small targets in images by fusing features from different levels.

[0076] In each decoding layer, a mask attention operation is performed. The mask attention module updates the object query by limiting the query to local features within the previously predicted mask region, thereby improving the convergence speed of model training. When calculating attention, the mask attention module focuses on pixels inside the mask region predicted in the previous stage, while ignoring a large number of irrelevant pixels outside the region. This greatly improves the model's convergence speed and computational efficiency, and makes feature updates more targeted, resulting in more accurate instance boundaries.

[0077] After iterative optimization through multiple decoding layers, the final query vector is input into a prediction head. Each query vector includes a binary mask and corresponding confidence information, which filters out predictions with too low confidence and merges the binary masks of all remaining instances to generate the final garbage instance mask. The merging rule is: each pixel is assigned the ID of the instance with the highest confidence. If a pixel is covered by the masks of multiple instances, the model resolves the conflict based on the prediction scores, ensuring that a pixel belongs to only one instance.

[0078] In the implementation of the above embodiments: The instance segmentation model achieves accurate identification and segmentation of each individual piece of waste in the image. The model's built-in multi-scale feature extraction network and fusion mechanism effectively enhance the ability to identify waste of different sizes, reducing the problem of missing small targets. The mask attention module, by focusing on local features, not only accelerates model convergence but also generates instance masks with more accurate boundaries. This pixel-level instance discrimination capability fundamentally avoids misclassifying multiple adjacent or adhered pieces of waste as a single entity, providing a crucial data foundation for accurately calculating newly added waste areas and analyzing the specific number and proportion of individuals under each category. This comprehensively improves the recognition accuracy and analytical precision of the entire waste classification quality assessment system in complex real-world scenarios.

[0079] Optionally, in this embodiment, a model can be used to obtain identification results based on the overall waste area mask, waste category mask, waste instance mask, and newly added waste area mask. Furthermore, these results can be fused and logically processed to accurately calculate the proportion of each category in the newly delivered waste, providing a quantitative basis for the final quality score.

[0080] Based on the overall waste area mask, waste category mask, waste instance mask, and newly added waste area mask, newly delivered waste is identified, and the identification results are obtained, including: The initial new area is determined based on the overall waste area mask and the newly added waste area mask. For example, the initial new area is determined based on the intersection of the overall waste area mask and the newly added waste area mask; if the intersection is empty, the entire newly added waste area mask is used as the initial new area; the new waste area does not include the area corresponding to the original waste after transformation; the original waste refers to the waste that was originally contained in the waste bin before the waste was disposed of; transformation includes at least one of displacement, deformation, and flipping.

[0081] The intersection operation performs a pixel-by-pixel logical AND operation between the overall garbage region mask and the newly added garbage region mask, which is usually achieved through dot multiplication in image processing. For each pixel location in the image, only when both the overall garbage region mask and the newly added garbage region mask are 1 will that pixel value be identified as a preliminary newly added region. This operation effectively excludes regions that are misjudged as "changes" by the difference detection model but are not actually garbage, such as newly generated light and shadow, flying insects, etc., which may be identified as newly added garbage region masks, but through this operation, they will not be identified as preliminary newly added regions, thus obtaining a higher confidence and cleaner newly added garbage region.

[0082] The initial newly added region is fused with the garbage instance mask to identify at least one newly delivered garbage instance within the initial newly added region. Joint analysis is then performed on the initial newly added region and the garbage instance mask. Pixels in the garbage instance mask are traversed, and their corresponding instance IDs are recorded. For each overlapping sub-region in the initial newly added region and the garbage instance mask, the garbage instances in different sub-regions within the initial newly added region are determined based on the instance IDs of the sub-regions in the garbage instance mask.

[0083] One implementation approach is to create a list that collects all unique instance IDs appearing within the initial newly added area. For example, if pixels with instance IDs 5, 7, and 12 are found within the initial newly added area, then we can determine that instances 5, 7, and 12 are newly delivered garbage instances. Garbage instances whose instance IDs do not appear within the initial newly added area are considered as residual garbage that existed before delivery and are not considered in this scoring process.

[0084] Based on the garbage category mask, the garbage category of the garbage instance is determined. For each newly delivered garbage instance identified in the previous step (e.g., instance 5), first, locate all pixels belonging to that instance (i.e., whose pixel value equals the instance ID) on the garbage instance mask. Then, obtain the category index corresponding to these pixels on the garbage category mask. Since an instance is physically a continuous object, most of its pixels should be assigned the same category index in the garbage category mask. Count the category indices of all pixels in the instance, and use the mode (i.e., the most frequently occurring category index) as the final category of the garbage instance. For example, if more than 95% of the pixels in instance 5 are classified as "plastic" (category index 3), then the instance is determined to be "plastic".

[0085] The proportion of waste by category is determined by calculating the proportion of pixel information of one or more waste instances of the same category in the initial newly added area. For example, the proportion of plastic waste by calculating the proportion of pixel information of one or more waste instances of the aforementioned plastic category in the initial newly added area is used to determine the proportion of plastic waste.

[0086] In the implementation of the above embodiments: by performing multi-level data fusion and cross-validation of the overall waste area, newly added areas, instance segmentation, and semantic segmentation results, the accuracy and robustness of waste classification judgment are greatly improved. Logical AND operations effectively eliminate interference from non-waste change areas, improving the purity of the analysis target; by using associated instance masks, precise tracking of each independent waste item in this delivery is achieved; finally, pixel-level category and area statistics yield the proportion of each category. This reduces problems such as misclassifying residual waste as this delivery or misclassifying multiple sticky waste items.

[0087] Optionally, in this embodiment of the application, determining the initial newly added region based on the overall garbage region mask and the newly added garbage region mask includes: The initial new area is determined based on the intersection of the overall waste area mask and the newly added waste area mask; if the intersection is empty, the entire newly added waste area mask is taken as the initial new area; the newly added waste area does not include the area corresponding to the original waste after transformation; the original waste refers to the waste that was originally contained in the waste bin before the waste was disposed of; transformation includes at least one of displacement, deformation and flipping.

[0088] In the implementation of the above embodiments: The model can identify and eliminate false new signals generated after the transformation of existing waste. Through deep learning, the difference detection model can distinguish between genuine new waste and changes in the morphology of existing waste. For displacement cases, the model analyzes instances to identify that it is a change in the position of the same object rather than the addition of a new object. For deformation cases, the model uses instance analysis and category analysis, among other invariant features, to determine that it is a change in the morphology of an existing object. None of these transformation cases will be incorrectly identified as newly added waste areas.

[0089] Optionally, in this embodiment of the application, the identification result includes the newly delivered waste category and the proportion of each waste category; after obtaining the identification result, the method further includes: According to the delivery mapping table, each newly delivered waste category is matched with the waste bin category. If the match fails, it means that the waste was delivered incorrectly, and the waste category and percentage of the waste delivered incorrectly are obtained. The delivery mapping table is used to represent the waste bin category that each waste category should be delivered to.

[0090] The type of trash can can be obtained through camera monitoring, such as currently monitoring the "food waste" bin, or through stored information. Each newly disposed trash instance is iterated through. For each instance, its identified trash category is used as the lookup key to search the set of allowed trash can categories in the disposal mapping table. It is then determined whether the current trash can's category is within this allowed set. If not, the trash instance is considered an incorrect disposal. The IDs of all incorrectly disposed trash instances and their corresponding error categories are recorded. For example, trash from a bottle instance does not match the trash can's category, while trash from a leftover food instance does.

[0091] The score for this new waste disposal is determined based on the percentage of incorrect waste disposals and the weighting coefficients; the weighting coefficients are dynamically adjusted based on the type of waste and / or the type of waste bin where the waste was disposed of incorrectly.

[0092] For example, we can count the total number of pixels contained in all garbage instances marked as delivery errors, count the total number of pixels of all newly delivered garbage instances (i.e., the total number of pixels in the initially added area), and obtain the percentage of garbage delivery errors.

[0093] The weighting coefficient is not a fixed value, but is dynamically determined based on the specific error scenario and can be adjusted according to the type of waste or the type of waste bin. For example, the weighting coefficient for mistakenly putting "batteries" (hazardous waste) into the food waste bin is much higher than the weighting coefficient for mistakenly putting "corn cobs" into the food waste bin.

[0094] As one implementation method, the score for this new delivery is 100 - (the percentage of incorrect garbage delivery * weighting coefficient * 100).

[0095] In the implementation of the above embodiments: a refined and intelligent waste sorting quality evaluation system is constructed by introducing a delivery mapping table and dynamic weighting coefficients. The delivery mapping table provides a benchmark for judging accuracy and consistency, improving the correctness of sorting. The introduction of dynamic weighting coefficients enables the system to intelligently distinguish the severity of different delivery errors, reducing the limitations of "one-size-fits-all" scoring, and imposing more punitive scores on delivery errors with high harm and pollution, thereby guiding users to pay more attention to key sorting links. This scoring mechanism, by quantitatively reflecting delivery quality and linking with reward and punishment mechanisms, effectively promotes the improvement of users' waste sorting awareness and the standardization of their behavior, improving the accuracy of waste sorting and overall management efficiency from a results-oriented perspective.

[0096] Optionally, in this embodiment of the application, before inputting the channel-combined image into the difference detection model to obtain the mask of the newly added garbage region, the method further includes: Acquire paired images; paired images include temporally consecutive pre-disposal sample images and post-disposal sample images. A paired image refers to a combination of two images with a temporal relationship, including a pre-disposal sample image and a post-disposal sample image. The pre-disposal image captures the state of the trash can before the user's disposal action, while the post-disposal image records the state after the disposal action is completed. Together, they constitute the paired images for model training.

[0097] Each pair of paired images is pixel-annotated to generate an annotated image carrying a ground truth mask of the newly added garbage area; the first value of the pixel annotation represents the newly added garbage area in this delivery, and the second value represents the original garbage area; the original garbage area includes the unchanged area and the area corresponding to the original garbage after transformation.

[0098] Professional image annotation tools can be used to simultaneously display paired images and identify newly added waste areas through comparative analysis. For clearly new waste, it is directly labeled with the first value (e.g., 1); for areas that existed before disposal and remained unchanged after disposal, it is labeled with the second value (e.g., 0); for areas where existing waste has changed, it is determined whether it belongs to a morphological change of the same waste object, and if so, it is labeled with the second value. The annotation results are saved as a standardized binary mask format, establishing a pixel-level correspondence with the original image.

[0099] By using a loss function and labeled images, the initial model for difference detection is trained to obtain a well-trained difference detection model.

[0100] The loss function includes an invariance loss, which is used to constrain the difference detection model to maintain consistent output for the same image pair under different lighting enhancements. The initial difference detection model includes a two-stream twin network and a cross-attention module. The two-stream twin network is used to extract image features of the sample image before delivery and the sample image after delivery, respectively. The cross-attention module is used to calculate the difference between paired images.

[0101] A two-stream Siamese network extracts feature representations from the pre-delivery and post-delivery images, respectively, with the two sub-networks sharing weights to ensure consistency in feature extraction. A cross-attention module receives the two feature streams and captures the difference patterns between the two images by calculating attention weights between the query, key, and value. The model ultimately outputs a prediction mask of the same size as the input image, where each pixel value represents the probability that the location belongs to a newly added garbage region.

[0102] The invariance loss applies different illumination enhancements to the same image pair, requiring the model's output to remain consistent under different enhancement conditions. This is achieved by calculating the KL divergence or mean square error between two prediction results.

[0103] As one implementation method, multiple loss functions can be used to form the total loss function. For example, a binary cross-entropy loss can be used to calculate the difference between the predicted mask and the ground truth mask. The total loss is determined jointly using the binary cross-entropy loss and the invariance loss.

[0104] In the implementation of the above embodiments: Annotated images are used to train the initial difference detection model, improving its performance and robustness in complex real-world environments. Pixel-level labeled ground truth masks provide accurate supervision signals for model learning, enabling the model to precisely distinguish between genuine new waste and existing waste morphological changes. Introducing an invariant loss term effectively enhances the model's adaptability to lighting changes, mitigating false detections caused by ambient light fluctuations. A two-stream twin network structure improves the consistency of feature extraction from images before and after delivery, while a cross-attention mechanism delves into subtle differences between paired images. This comprehensive training strategy ensures that the final difference detection model maintains stable high performance under various challenging scenarios, providing a reliable new area detection capability for the entire waste sorting quality assessment system, thereby improving classification accuracy.

[0105] Optionally, in this embodiment of the application, after performing instance segmentation on the post-garbage delivery image to obtain the garbage instance mask, the method further includes: Infrared analysis sensors are used to emit infrared light into the region corresponding to the mask of a waste instance and receive the reflected spectral signal. These sensors can emit infrared light of specific wavelengths towards the target object and accurately measure the reflected spectral signal. Their operating bands typically cover the near-infrared and mid-infrared regions, which are sensitive to the chemical structural characteristics of organic molecules and inorganic materials.

[0106] First, the coordinate transformation relationship between the visible light camera and the infrared sensor can be established. Through a pre-calibration process, the intrinsic and extrinsic parameters of both sensors are acquired, establishing a mapping model from image pixel coordinates to real-world 3D coordinates. After the instance segmentation model outputs a garbage instance mask, the system calculates the center point and bounding box coordinates of this masked region in real space. The mechanical rotation device of the infrared sensor is controlled to adjust the emission angle, ensuring that the infrared beam completely covers the physical area of ​​the target garbage instance.

[0107] The sensor drives an infrared light source to emit pre-designed infrared wavelengths, with the intensity and duration of each wavelength controlled. Simultaneously, the sensor's receiver measures the intensity signal reflected from the target area at a high sampling rate. For each emitted wavelength, the corresponding reflected light intensity value is recorded, forming a preliminary reflectance spectrum curve.

[0108] The reflected spectral signal is compared with the spectral database to obtain the predicted category and confidence level corresponding to the garbage instance mask.

[0109] Key feature parameters are extracted from the preprocessed spectrum, including absorption peak position, peak height, peak area, and full width at half maximum (FWHM) for specific bands. Shape descriptors of the spectrum, such as spectral slope, curvature features, and asymmetry, are calculated. These features are combined into a multi-dimensional feature vector for subsequent similarity calculation. A multi-stage matching strategy is employed. First, rapid pre-screening is performed using features reduced in dimensionality by principal component analysis (PCA) for coarse matching, narrowing down the candidate categories. Then, fine matching is performed, calculating the similarity between the measured spectral feature vector and the reference vector for each candidate category. The overall similarity score is converted to a confidence value in the range of 0-1 using a sigmoid function, and the candidate category with the highest confidence is selected as the predicted category.

[0110] Based on the garbage mask, garbage instance mask, and newly added garbage area mask, newly delivered garbage is identified, including: identifying newly delivered garbage based on the garbage mask, garbage instance mask, newly added garbage area mask, and the predicted category and confidence level corresponding to the garbage instance mask.

[0111] For the process of identifying newly delivered waste, please refer to the above. In the process of identifying at least one newly delivered waste instance in the initially added area, and in determining the waste category of the waste instance, it is necessary to refer to the predicted category and confidence level corresponding to the waste instance mask.

[0112] In the implementation of the above embodiments: by introducing infrared spectroscopy analysis, a material identification technology, a verification dimension independent of visible light image analysis is added to the waste sorting system, significantly improving the accuracy and reliability of category identification. Infrared spectroscopy can directly detect the molecular structure and chemical composition of substances. This identification method based on the essential properties of substances reduces the misjudgment problems caused by external factors such as color changes, surface contamination, and shape deformation in visible light identification. Even in distinguishing between different materials with extremely similar appearances (such as different types of plastics), it can provide a reliable basis for judgment.

[0113] Optionally, in this embodiment of the application, after determining the initial newly added area, the method further includes: From the post-garbage disposal image, crop the minimum bounding rectangle region corresponding to the newly added garbage area mask. For example, connected component analysis can be performed on the newly added garbage area mask to identify all interconnected white pixel regions. Use a contour tracing algorithm (such as a boundary tracing algorithm) to obtain its precise set of external boundary points, thus obtaining a polygonal contour. Determine the minimum bounding rectangle region based on the polygonal contour.

[0114] The smallest bounding rectangle region is input into the semantic segmentation model for secondary inference to obtain the inference result; the inference result determines the garbage category of the garbage instance.

[0115] Based on the parameters of the calculated minimum bounding rectangle region, pixels within the rectangle region are extracted from the corresponding positions in the original post-garbage delivery image to generate a cropped sub-image. This sub-image is then input into the semantic segmentation model for secondary inference to obtain the inference result.

[0116] In the implementation of the above embodiments: targeted region cropping and secondary inference mechanisms significantly improve the accuracy and reliability of waste category identification. By extracting the minimum bounding rectangle of newly added waste regions and performing specialized processing, interference from complex backgrounds and irrelevant regions in the original image is effectively eliminated, allowing the semantic segmentation model to focus on the detailed features of the target region. This focused processing approach is beneficial for identifying smaller waste instances or objects with inconspicuous appearance features, reducing identification failures caused by targets that are too small or located in remote areas. The secondary inference process runs on higher-quality, more concentrated input data, enabling it to capture subtle feature differences that may be overlooked in the initial global analysis, thereby providing higher confidence and accuracy in category discrimination.

[0117] This application provides a waste disposal identification device, including: The image acquisition module is used to acquire images of the trash can before and after the trash is disposed of. The combination module is used to combine the images before and after garbage disposal into channels to obtain a combined channel image. The segmentation module is used to segment the image after garbage disposal to obtain a garbage mask; the garbage mask includes an overall garbage area mask and a garbage category mask; The instance segmentation module is used to segment images after garbage disposal and obtain garbage instance masks. The difference detection module is used to input the channel-combined image into the difference detection model to obtain the mask of the newly added garbage region; The identification module is used to identify newly delivered waste based on the overall waste area mask, waste category mask, waste instance mask, and newly added waste area mask, and obtain the identification results.

[0118] It should be understood that this device corresponds to the above-described embodiment of the garbage disposal identification method and is capable of performing the various steps involved in the above-described method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0119] Please see Figure 2 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0120] Figure 2 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a server or PC, or a virtual device, such as a virtual machine or virtualization container. Furthermore, electronic device 300 is not limited to a single device; it can be a combination of multiple devices or a cluster of numerous devices.

[0121] This application also provides a storage medium storing a computer program, which is executed by a processor to perform the above-described method.

[0122] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0123] This application also provides a computer program product, including computer program instructions, which are executed by a processor to perform the method described above.

[0124] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, given the several embodiments provided in this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0125] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0126] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A method of identifying a delivery of refuse, characterized by, The method comprises the following steps: collecting a pre-litter-throwing image and a post-litter-throwing image of a garbage can; combining the pre-litter-throwing image and the post-litter-throwing image to obtain a channel combined image; segmenting the post-litter-throwing image to obtain a garbage mask; the garbage mask comprises a whole garbage region mask and a garbage category mask; instance segmentation is performed on the post-litter-throwing image to obtain a garbage instance mask; inputting the channel combined image into a difference detection model to obtain a newly added garbage region mask; based on the whole garbage region mask, the garbage category mask, the garbage instance mask and the newly added garbage region mask, identifying newly thrown garbage to obtain an identification result.

2. The method of claim 1, wherein, The method of combining the pre-litter-throwing image and the post-litter-throwing image to obtain a channel combined image comprises: the pre-litter-throwing image and the post-litter-throwing image are respectively taken as three-channel image blocks; the three channels include R channel, G channel and B channel; the three-channel image blocks corresponding to the pre-litter-throwing image and the post-litter-throwing image are superimposed along the channel dimension to generate the channel combined image.

3. The method of claim 1, wherein, The method of segmenting the post-litter-throwing image to obtain a garbage mask comprises: using an image segmentation model to perform image segmentation on the post-litter-throwing image to obtain the whole garbage region mask; the image segmentation model is used to identify garbage regions and non-garbage regions; using a semantic segmentation model to perform image segmentation on the post-litter-throwing image to obtain the garbage category mask; the semantic segmentation model is used to identify the category of garbage in the post-litter-throwing image.

4. The method of claim 1, wherein, The method of performing instance segmentation on the post-litter-throwing image to obtain a garbage instance mask comprises: inputting the post-litter-throwing image into a pre-trained instance segmentation model, and using the instance segmentation model to perform instance segmentation on the post-litter-throwing image to obtain a garbage instance mask; the instance segmentation model is used to identify and segment different garbage individuals in the post-litter-throwing image; the instance segmentation model comprises a multi-scale image feature extraction network, an encoder and a decoder; the decoder comprises a mask attention module and a multi-level decoding layer; the mask attention module is used to update the local features in the query mask region; the multi-level decoding layer is used to receive the image features of the corresponding scale output by the multi-scale image feature extraction network, and fuse the image features of multiple scales.

5. The method of claim 1, wherein, The method of identifying newly thrown garbage based on the whole garbage region mask, the garbage category mask, the garbage instance mask and the newly added garbage region mask to obtain an identification result comprises: determining a preliminary newly added region according to the intersection of the whole garbage region mask and the newly added garbage region mask; if the intersection is empty, the newly added garbage region mask is taken as the preliminary newly added region as a whole; the newly added garbage region does not contain the region corresponding to the original garbage after transformation; the original garbage refers to the garbage originally contained in the garbage can before the garbage is thrown; the transformation includes at least one of displacement, deformation and flipping. fuse the preliminary new-added region with the garbage instance mask to identify at least one newly delivered garbage instance in the preliminary new-added region; determine a garbage category of the garbage instance according to the garbage category mask; determine the proportion of the garbage of the category based on a proportion of pixel information of the garbage instance of the same category in the preliminary new-added region.

6. The method of claim 1, wherein, The identification result includes a garbage category newly delivered and a proportion of each garbage category; After obtaining the identification result, the method further includes: According to the delivery mapping table, each newly delivered garbage category is matched with the category of the garbage can respectively, and if the matching is unsuccessful, it represents a garbage delivery error, and the garbage category and the proportion of the garbage delivery error are obtained; the delivery mapping table is used to represent the garbage can category required for delivery of each garbage category; According to the proportion of the garbage delivery error and a weight coefficient, a score of the new delivery is determined; the weight coefficient is dynamically adjusted according to the garbage category of the garbage delivery error and / or the category of the garbage can.

7. The method of claim 1, wherein, Before inputting the channel combination image into the difference detection model to obtain the new garbage region mask, the method further includes: Collecting paired images; the paired images include a pre-delivery sample image and a post-delivery sample image in time sequence; Pixel labeling is performed on each pair of paired images to generate a labeled image carrying a new garbage region ground truth mask; wherein a first value of pixel labeling represents a new garbage region delivered this time, and a second value represents an original garbage region; the original garbage region includes an unchanged region and a region corresponding to the original garbage after transformation; A trained difference detection model is obtained by training the difference detection initial model using the labeled image through a loss function; The loss function includes an invariance loss for constraining the difference detection model to keep consistent output of the same image pair under different light enhancement; the difference detection initial model includes a dual-flow twin network and a cross-attention module; the dual-flow twin network is used to extract image features of the pre-delivery sample image and the post-delivery sample image respectively; the cross-attention module is used to calculate the difference between the paired images.

8. A computer program product, characterised in that, Computer program instructions are included, which are executed by a processor to perform the method of any one of claims 1 to 7.

9. An electronic device, comprising: including: a processor and a memory, the memory storing computer program instructions, the computer program instructions being executed by the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, which are executed by a processor to perform the method of any one of claims 1 to 7.

Images