A method for identifying kitchen waste occluded objects and judging occlusion relationships

Abstract

The invention discloses a method for identifying kitchen waste occluded objects and judging occlusion relations, constructs an occlusion data set of kitchen waste images, and marks the visible mask, complete mask, and category of each target object therein; first, uses the occlusion data set to train an instance segmentation model and a mask restoration model; the input of the instance segmentation model is the kitchen waste image, and the output is the category and visible mask of each target object; the input of the mask restoration model is the visible mask and the kitchen waste image output by the instance segmentation model, and the output is the complete mask; secondly, the kitchen waste image to be identified and judged is used to obtain the category, visible mask, and complete mask of each target object using the trained instance segmentation model and mask restoration model; finally, the visible mask and the complete mask are combined to judge whether each target object is occluded and the occlusion relationship. The invention performs instance segmentation and mask restoration on recyclable garbage, and can obtain object category, position, occlusion relationship and complete shape information.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for identifying kitchen waste occluded objects and determining occlusion relationships. Background Art

[0002] In recent years, the recycling and recovery of solid waste has been further improved. Among them, the green and low-carbon model of "garbage classification + comprehensive utilization + incineration power generation" has gradually become the mainstream idea of ​​solid waste utilization and treatment. However, due to the diversity and complexity of kitchen waste, kitchen waste can be divided into plastic, glass, fiber, metal and other categories of waste. How to identify and sort these categories of waste is a key problem.

[0003] In recent years, many advanced algorithms have been proposed as a means to improve the efficiency of garbage classification. However, in the process of garbage classification and identification, the stacking of garbage objects and the complexity of the background lead to serious occlusion, which seriously affects the accuracy and efficiency of identification. Therefore, it is necessary to accurately identify occluded objects to ensure the smooth progress of classification. Most of the currently available garbage datasets are designed based on simple garbage scene recognition, and there is a lack of relevant research on garbage recognition under real scene occlusion. In addition, kitchen waste images often contain multiple targets. The task of accurately identifying the spatial positions of these targets is image instance segmentation, in which an accurate judgment of the relationship between target objects is a key link to improve the efficiency of garbage classification. As one of the tasks in the field of computer vision, image instance segmentation mainly involves pixel-level segmentation of each object instance in the image. In the real world, garbage objects usually have variable shapes, which is more suitable for the task of instance segmentation.

[0004] From the perspective of kitchen waste object recognition, real scenes have high requirements for the real-time performance of the model. Due to the complex image background and occlusion between object categories, the current method has achieved good results in object instance segmentation for this problem, but the method of restoring the mask of the entire object from the existing visible mask and then judging the occlusion relationship between objects is missing. Therefore, it is of great practical value and significance to study the intelligent kitchen waste occlusion recognition and occlusion relationship judgment algorithm in real scenes. Summary of the invention

[0005] The present invention provides a method for identifying kitchen waste occluded objects and determining occlusion relationships, which performs instance segmentation on recyclable waste, and can obtain not only the category and position, but also the posture of the visible part of the object.

[0006] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0007] A method for identifying kitchen waste occluded objects and determining occlusion relationships, comprising:

[0008] Step 1, construct an occlusion dataset of kitchen waste images, where each image is labeled with the visible mask, complete mask and category of each target object in the image;

[0009] Step 2: Use the occlusion dataset in step 1 to train the instance segmentation model and the mask restoration model;

[0010] The input of the instance segmentation model is the kitchen waste image, and the output is the category and visible mask of each target object in the image;

[0011] The input of the mask restoration model is: the visible mask of the target object output by the instance segmentation model, and the kitchen waste image of the target object in the occlusion dataset; the output is: the complete mask of the target object;

[0012] Step 3: Input the kitchen waste image to be identified and judged into the trained instance segmentation model, and output the category and visible mask of each target object; then input the obtained visible mask into the trained mask restoration model, and output the complete mask of each target object;

[0013] Step 4: Combine the visible mask obtained in step 3 with the complete mask to determine whether each target object is occluded and the occlusion relationship.

[0014] Furthermore, the instance segmentation model uses a YOLOV8 instance segmentation network.

[0015] Furthermore, the mask restoration model uses a lightweight network Unet as the backbone network, and adds a residual module that fuses spatial attention and channel attention between each layer of Unet downsampling to form a neural network Unet-RC.

[0016] Furthermore, the loss function of the mask restoration model trained using the occlusion dataset includes segmentation loss, similarity loss and boundary loss.

[0017] Furthermore, the segmentation loss uses the CrossEntropyLoss function to determine the category loss of each pixel, and its calculation formula is:

[0018] (1)

[0019] Where x is each pixel, is the target distribution, Represents the predicted matching distribution Represents the segmentation loss, which describes the distance between the two distributions. The more consistent the predicted category of the pixel is with the target category, the The smaller the value, the more pixels are classified into: complete masks that belong to the target object and complete masks that do not belong to the target object.

[0020] Furthermore, the similarity loss adopts the DiceLoss function, and the calculation formula is:

[0021] (2)

[0022] Among them, X is the complete mask predicted by the mask restoration model, and Y is the annotated true complete mask; Represents the intersection of the X set and the Y set, representing the dot product between the corresponding pixels of the predicted complete mask X and the annotated true complete mask Y, and adding the dot product results; It represents the sum of the corresponding pixels of the predicted complete mask X and the annotated true complete mask Y.

[0023] Furthermore, the boundary loss is achieved by extracting the boundary of the mask from the model prediction mask map and the target mask map, and the prediction of the mask boundary is achieved, wherein the calculation formula is:

[0024] (3)

[0025] (4)

[0026] (5)

[0027] Among them, Sobel(X) and Sobel(Y) represent that the mask uses the Sobel filter to calculate the gradient of the prediction set X and the target set Y. Represents the cosine similarity calculation, To predict the cosine similarity between the gradient Sobel(X) and the target gradient Sobel(Y), MagPenalty represents the magnitude difference between Sobel(X) and the target gradient Sobel(Y). represents the parameter that measures the difference in boundary direction and magnitude penalty, Represents the weight of the control amplitude penalty.

[0028] Furthermore, the specific determination method of step 4 is:

[0029] First, determine whether each target object is occluded by whether the visible mask of the target object exceeds 5% to the complete mask. If so, the object is initially determined to be occluded.

[0030] Then, the target objects in the same image are judged to be adjacent or not. The calculation basis is as follows:

[0031] (6)

[0032] (7)

[0033] (8)

[0034] Where: X is the output mask map of the instance segmentation model; Z is the set of visible mask maps that do not contain X in the same image; represents the dilation operation; DK is the core element involved in the dilation operation, which is used to expand the non-zero numbers in the mask image X; represents the mask image after the expansion operation of X; C represents the mask image Whether there is an intersection with the mask map Z; judge whether X and Z are adjacent according to the true or false value of C, if there is an intersection, they are adjacent; Np.any means judging the mask map Whether there are pixels at the same position in the mask image Z that are both 1;

[0035] Finally, for every two adjacent target objects, the occlusion relationship is determined based on the overlap ratio of their complete masks.

[0036] The present invention provides a method for identifying kitchen waste occluded objects and judging occlusion relationships. The instance segmentation algorithm is used to segment and crop the kitchen waste target object in the color image and output it. At the same time, the instance segmentation model uses the first-stage YOLO framework as a benchmark to capture the category information and the mask of the visible part of the target object. The second part is to input the visible mask output by the instance segmentation network and the color image of the object into a lightweight mask restoration model. While lightweighting, an attention mechanism is added to reduce the performance loss caused by the reduction of model parameters, and to improve the adaptive capture of the semantic perception area, further enhance the restoration ability, and improve the occlusion recognition and occlusion relationship judgment effect. Compared with the existing kitchen waste segmentation and occlusion processing technology, it has the following advantages:

[0037] Different from target detection and image classification, the instance segmentation of kitchen waste performed by the present invention can obtain not only the category and position, but also the posture of the visible part of the object. Compared with other instance segmentation algorithms, the instance segmentation model of the present invention identifies unconnected areas of the same object as the same one, solving the problem of the middle of the target object being occluded in the occlusion problem.

[0038] The present invention adds an attention mechanism to the lightweight mask restoration model to reduce the performance loss caused by the reduction of model parameters, while further enhancing the recognition ability and improving the restoration effect of the object mask.

[0039] The present invention has strong practicality and generalization ability. It not only achieves good results in the current kitchen waste occlusion recognition, but also obtains excellent restoration accuracy and occlusion relationship discrimination results on the COCOA dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of occluded object recognition and occlusion relationship judgment according to an embodiment of the present invention.

[0041] Figure 2 Schematic diagram of a mask restoration model module according to an embodiment of the present invention.

[0042] Figure 3 The figure shows the accuracy relationship of different YOLO network parameters on the OC dataset according to the embodiment of the present invention.

[0043] Figure 4 The accuracy of different Unet-RC network parameters on the COCOA dataset according to the embodiments of the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is to realize occlusion recognition and occlusion relationship discrimination of kitchen waste images based on an instance segmentation model and a mask restoration network. The Python programming language can be used for experiments, and the C / C++ programming language can also be used for engineering applications.

[0045] This embodiment provides a method for identifying kitchen waste occluded objects and determining occlusion relationships. Figure 1 , including the following steps:

[0046] Step 1: Construct an occlusion dataset OC of kitchen waste images, where each image is marked with the visible mask, complete mask, category, and occlusion relationship of each target object in the image.

[0047] The occlusion data described in the embodiment is collected on the kitchen waste sorting conveyor belt, and the 2,000 kitchen waste images obtained complete the production of the OC data set, which is closer to the real scene and has strong engineering application significance. In order to reflect the characteristics of the complexity of kitchen waste in different time periods, this embodiment screens images from different quantities and complexities, and divides them into four levels of occlusion image sets, namely 200 occluded images in pairs, 700 slightly occluded images, 600 moderately occluded images, and 500 severely occluded images. By cleaning and screening the image data, a kitchen waste occlusion data set containing 13 categories and a total of 2,000 fully annotated images is finally obtained, and the data set is divided into a training set and a test set in a ratio of 8:2. At the same time, it is verified on the COCO Amodal dataset, which is widely used in the field of occlusion recognition. The total of 5,000 images include a training set (2,500 images), a validation set (1,250 images), and a test set (1,250 images), and cover more than 1,000 common categories in life, with good generalization test performance.

[0048] The definition of occlusion degree is:

[0049] Mild occlusion: The background is relatively clean, the degree of occlusion of the object does not exceed 25%, and the occlusion depth does not exceed 3 layers. For example: (AB, BC) A blocks B, B blocks C, and the occlusion depth of C is 3;

[0050] Moderate occlusion: The background is somewhat cluttered, the foreground is unobstructed, the degree of object occlusion does not exceed 50%, and the number of objects in the image does not exceed 7.

[0051] Severe occlusion: The background is dirty, non-targets in the foreground are occluded (such as plastic bags and transparent films), the degree of occlusion of objects exceeds 50%, or the number of objects exceeds 7.

[0052] Step 2: Use the occlusion dataset to train the instance segmentation model and the mask restoration model.

[0053] (1) Instance segmentation model;

[0054] The input of the instance segmentation model is the kitchen waste image, and the output is the category and visible mask of each target object in the image.

[0055] This embodiment uses the latest YOLOV8 instance segmentation network to process the input kitchen waste image to achieve category judgment and shape segmentation of recyclable waste objects.

[0056] In order to adapt the present invention to the instance segmentation model of kitchen waste, the original resolution of the image, 1080×1080, is adjusted to 640×640. Before training the model, the pre-trained YOLOV8-seg on the COCO dataset is first used as the pre-training weight. During the training process, data preprocessing is first performed to read the labels in. Unlike other datasets, the occlusion dataset OC of the present invention includes two types of occlusions, one in which a part of the object is occluded and the rest of the object is completely connected, and the other is that the middle part of the object is occluded, resulting in the visible part of the object being separated. Therefore, when processing this part of the data, the separated parts of the same object must be merged into the same Mask image, and then a series of data enhancements are performed to complete the early data processing. The YOLOV8-seg model adopts a multi-scale feature extraction network, which captures the details and contextual information of targets of different sizes through feature fusion at different levels, thereby improving the segmentation accuracy of targets of various sizes.

[0057] Some hyperparameters in the training process are as follows: The optimizer uses SGD, with a momentum of 0.937 and a weight decay coefficient of 0.0005. The batch size is 8, the initial learning rate (LRO) is 0.01, and a total of 250 epochs are trained. The multi-stage learning rate is updated with a decay coefficient of 0.1. Ubuntu 20.04LTS or above is required, and the system environment requires Pytorch1.12 and Python3.9 or above. The hardware platform includes graphics cards NVIDIA GeForce RTX 3090×2, CUDA11.8, CuDNN8700, and the CPU memory is not less than 64G, and the solid-state drive is not less than 512G.

[0058] (2) Mask restoration model;

[0059] The input of the mask restoration model is: the visible mask of the target object output by the instance segmentation model, and the kitchen waste image of the target object in the occlusion dataset; the output is: the complete mask of the target object.

[0060] The implementation example adjusts the output of the instance model (640×640) to the input size (256×256) of the mask restoration model according to the bounding box cropping. The entire lightweight mask restoration model is a new neural network based on Unet: specifically, a residual module that fuses spatial attention and channel attention is added between each layer of Unet downsampling and upsampling. In order to make the network more lightweight, the input category information is taken out and only the binary mask map is retained, so that the output of the mask restoration model only requires 2 category predictions: a complete mask that belongs to or does not belong to the target object. The binary visible mask is subjected to a series of data enhancements (including erasing, flipping, mosaic, etc.) to complete the data preparation.

[0061] In this embodiment, the loss function of the training instance segmentation model adopts the commonly used cross entropy loss, while the loss function of the training mask restoration model includes three parts: segmentation loss, similarity loss and boundary loss.

[0062] The first part is the normal segmentation loss function, which uses the CrossEntropyLoss function to determine the loss of each pixel category, and its calculation formula is:

[0063] (1)

[0064] Among them, p(x) is the target distribution, q(x) represents the predicted matching distribution, and H(p, q) describes the distance between the two distributions. The more consistent the predicted category of the pixel is with the target category, the smaller the H(p, q) value is. The second part is the similarity loss function dice-loss, which can effectively solve the problem of category imbalance. Its calculation formula is:

[0065] (2)

[0066] Among them, X is the complete mask predicted by the mask restoration model, and Y is the annotated true complete mask; represents the intersection of the X set and the Y set, and in the present invention represents the dot product between the predicted complete mask and the real mask pixels, and the dot product results are added, Represents the sum of the pixels in the corresponding images. This loss function can alleviate the negative impact of foreground-background imbalance in samples.

[0067] The third part is the boundary loss BoundaryLoss, which extracts the boundary of the mask by comparing the model prediction mask map and the target mask map to achieve accurate prediction of the mask boundary. The calculation formula is:

[0068] (3)

[0069] (4)

[0070] (5)

[0071] Among them, Sobel(X) and Sobel(Y) represent the mask using the Sobel filter to calculate the gradient. represents the cosine similarity between the predicted gradient and the target gradient, MagPenalty represents the magnitude difference between the predicted gradient and the target gradient, represents the parameter that measures the difference in boundary direction and magnitude penalty, Represents the weight of the control amplitude penalty.

[0072] Some hyperparameters in the training process are as follows: the optimizer uses SGD, the momentum is 0.9, the weight decay is 0.001, the learning rate uses a phased learning rate, the batch size is 16, and a total of 15,000 iterations are trained. When the iterations reach 8,000 and 13,000, the learning rate is reduced by half respectively. The software and hardware environment is consistent with that in the instance segmentation model.

[0073] Step 3: Input the kitchen waste image to be identified and judged into the trained instance segmentation model, and output the category and visible mask of each target object; then input the obtained visible mask into the trained mask restoration model, and output the complete mask of each target object.

[0074] Step 4: Combine the visible mask obtained in step 3 with the complete mask to determine whether each target object is occluded and the occlusion relationship. Specifically:

[0075] The first step is to determine whether each target object is occluded by whether the visible mask of the object exceeds 5% to the complete mask. If it exceeds, the object is initially determined to be occluded. The second step is to determine whether each object in the same image is adjacent. The calculation basis is as follows:

[0076] (6)

[0077] (7)

[0078] (8)

[0079] Where: X is the mask image, DK is the kernel element involved in the dilation operation, which is used to expand the non-zero numbers in the X image. Represents the mask image after X is expanded, and Np.any represents the judgment mask image Whether there are pixels at the same position in the mask image Y that are both 1, C represents Whether there is an intersection between the mask image X and the mask image Y, in the present invention, whether the mask images X and Y are adjacent is determined by judging the true or false value of C.

[0080] In the third step, the occlusion relationship between the adjacent pairs of masks obtained in the second step and the complete mask obtained in step 3 is determined to obtain the occlusion relationship between different objects in the entire image.

[0081] Table 1 and Figure 3 This is the experimental result of the YOLO instance segmentation model of the present invention on the self-built data set. It includes different degrees of parameter amounts (Parameter) of the YOLO model, and mainly compares the performance of different parameter amounts on the OC data set, including the accuracy (Precision) of the bounding box (B) and the mask (M), and the average precision (mAP) includes IoU=0.5 and IoU=[0.5:0.95]. The performance values ​​in the figure are relatively high, which shows that the YOLO segmentation model has good adaptability to the self-built data.

[0082]

[0083] Table 2 and Figure 4 This is the experimental result of the present invention using the mask restoration model on the COCOA dataset. The quality of the model is judged according to the evaluation indicators of general semantic segmentation tasks, including pixel accuracy (PA), mean pixel accuracy (MPA), average IOU of each class (MIOU), weighted sum of IOU of each class by the frequency of occurrence of each class (FWIOU) and occlusion relationship accuracy (OA). It is worth noting that the present invention is designed for the kitchen waste dataset, and its performance on COCOA is relatively high, which also proves the generalization and scalability of the method.

[0084]

[0085] Table 3 is a comparison of the occlusion relationship accuracy (OA) of the mask restoration model (Unet-RC) used in the present invention on the kitchen waste occlusion dataset (OC). When the model parameters are different, the OA value changes very little, which shows the adaptability and superiority of the mask restoration model in processing occlusion relationships.

[0086]

[0087] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. A method for identifying kitchen waste occluded objects and determining occlusion relationships, characterized in that: include: Step 1, construct an occlusion dataset of kitchen waste images, where each image is labeled with the visible mask, complete mask and category of each target object in the image; Step 2: Use the occlusion dataset in step 1 to train the instance segmentation model and the mask restoration model; The input of the instance segmentation model is the kitchen waste image, and the output is the category and visible mask of each target object in the image; The input of the mask restoration model is: the visible mask of the target object output by the instance segmentation model, and the kitchen waste image of the target object in the occlusion dataset; the output is: the complete mask of the target object; The loss functions for training the mask restoration model using the occlusion dataset include segmentation loss, similarity loss, and boundary loss; Step 3: Input the kitchen waste image to be identified and judged into the trained instance segmentation model, and output the category and visible mask of each target object; then input the obtained visible mask into the trained mask restoration model, and output the complete mask of each target object; Step 4: Combine the visible mask obtained in step 3 with the complete mask to determine whether each target object is occluded and the occlusion relationship.

2. The method for identifying kitchen waste obstructing objects and determining obstructing relationships according to claim 1, characterized in that: The instance segmentation model uses the YOLOV8 instance segmentation network.

3. The method for identifying kitchen waste obstructing objects and determining obstructing relationships according to claim 1, characterized in that: The mask restoration model uses a lightweight network Unet as the backbone network, and adds a residual module that fuses spatial attention and channel attention between each layer of Unet downsampling and upsampling to form a neural network Unet-RC.

4. The method for identifying kitchen waste obstructing objects and determining obstructing relationships according to claim 1, characterized in that: The segmentation loss uses the CrossEntropyLoss function to determine the category loss of each pixel, and its calculation formula is: （1）， Where x is each pixel, is the target distribution, represents the predicted matching distribution; Represents the segmentation loss, which describes the distance between the two distributions. The more consistent the predicted category of the pixel is with the target category, the The smaller the value, the more pixels are classified into: complete masks that belong to the target object and complete masks that do not belong to the target object.

5. The method for identifying kitchen waste obstructing objects and determining obstructing relationships according to claim 1, characterized in that: The similarity loss adopts the DiceLoss function, and the calculation formula is: （2）； Among them, X is the complete mask predicted by the mask restoration model, and Y is the annotated true complete mask; Represents the intersection of the X set and the Y set, representing the dot product between the corresponding pixels of the predicted complete mask X and the annotated true complete mask Y, and adding the dot product results; It represents the sum of the corresponding pixels of the predicted complete mask X and the annotated true complete mask Y.

6. The method for identifying kitchen waste obstructing objects and determining obstructing relationships according to claim 1, characterized in that: The boundary loss is achieved by extracting the boundary of the mask from the model prediction mask map and the target mask map, and predicting the boundary of the mask. The calculation formula is: （3）； （4）； （5）； Among them, Sobel(X) and Sobel(Y) represent that the mask uses the Sobel filter to calculate the gradient of the prediction set X and the target set Y. Represents the cosine similarity calculation, To predict the cosine similarity between the gradient Sobel(X) and the target gradient Sobel(Y), MagPenalty represents the magnitude difference between Sobel(X) and the target gradient Sobel(Y). represents the parameter that measures the difference in boundary direction and magnitude penalty, Represents the weight of the control amplitude penalty.

7. The method for identifying kitchen waste obstructing objects and determining obstructing relationships according to claim 1, characterized in that: The specific judgment method in step 4 is: First, determine whether each target object is occluded by whether the visible mask of the target object exceeds 5% to the complete mask. If so, the object is initially determined to be occluded. Then, the target objects in the same image are judged to be adjacent or not. The calculation basis is as follows: （6）； （7）； （8）； Where: X is the output mask map of the instance segmentation model; Z is the set of visible mask maps that do not contain X in the same image; represents the dilation operation; DK is the core element involved in the dilation operation, which is used to expand the non-zero numbers in the mask image X; represents the mask image after the expansion operation of X; C represents the mask image Whether there is an intersection with the mask map Z; judge whether X and Z are adjacent according to the true or false value of C. If there is an intersection, they are adjacent; Np.any indicates that the mask map is judged Whether there are pixels at the same position in the mask image Z that are both 1; Finally, for every two adjacent target objects, the occlusion relationship is determined based on the overlap ratio of their complete masks.

Images