A garbage data annotation system and method based on deep learning

Abstract

This invention provides a deep learning-based garbage data labeling system and method. Through a raw data set acquisition module, a raw data processing module, a semi-automatic model training module, and a semi-automatic model inference module, the system starts with zero labels, collects an unlabeled data set, generates a labeled data set through background removal and fusion methods, and then trains a semi-automatic labeling model for prediction and iteration, ultimately obtaining a two-dimensional detection bounding box for the target object. This system can generate a data set that better reflects real-world scenarios, reducing data acquisition costs. It can also automatically acquire labels, reducing data labeling costs. Ultimately, this system achieves self-driven data labeling.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a junk data labeling system and method based on deep learning. Background Art

[0002] Deep learning in artificial intelligence can abstractly represent the inherent patterns and semantic features of target data, enabling perception functions such as object detection, image segmentation, and instance segmentation. Given the unique advantages of deep learning convolutional layers in image recognition, they are also widely used in the field of garbage image recognition, reducing manual labor, improving cleaning rates, and achieving a clean and environmentally friendly living environment.

[0003] Currently, most deep learning-based visual models are supervised, requiring large labeled datasets. However, due to the diverse nature of garbage classification and complex backgrounds, data labeling is particularly challenging. Currently, there are three common approaches to labeling garbage data: manual labeling, semi-automatic labeling, or simulator generation, all of which have limitations. Manual labeling requires significant manual labor and results in high data acquisition costs. Furthermore, time constraints and delays make it difficult to meet the demands of agile AI model iteration. Semi-automatic models can improve labeling speed to some extent, but they lack the ability to generate data. Furthermore, initial training requires a certain amount of labeled data, which then undergoes repeated iterations to achieve optimal results, rendering them ineffective for detecting new categories. Furthermore, simulator synthesis can generate a large number of simulated images from scratch, but due to limitations in its model library and rendering technology, the resulting simulated images differ significantly from actual scene images, compromising the effectiveness of model transfer to real-world scenarios.

[0004] In short, the above three annotation methods all have certain limitations. Summary of the Invention

[0005] To address the challenges of the prior art, this paper provides a deep learning-based junk data labeling system and method. This system can generate datasets that better reflect real-world scenarios, reducing data acquisition costs. It can also automatically acquire labels, further reducing data labeling costs. Ultimately, this system achieves self-driven data labeling.

[0006] The present invention provides a garbage data labeling system based on deep learning, comprising an original data set acquisition module, an original data processing module, a semi-automatic model training module and a semi-automatic model reasoning module connected in sequence;

[0007] The original dataset acquisition module collects unlabeled datasets through web crawling, open source datasets, or scene collection as the subsequent processing object;

[0008] The raw data processing module performs saliency detection and cutout operations on the raw data, extracts the foreground image, and obtains the target detection frame using the minimum bounding rectangle;

[0009] The semi-automatic model training module inputs the labeled fusion dataset into the semi-automatic annotation model, calculates the loss, and uses the backpropagation method to calculate and update the model parameters until the model loss no longer decreases. The training is stopped and the trained semi-automatic annotation model is finally obtained.

[0010] The semi-automatic model inference module uses the semi-automatic annotation model in the previous step to infer the new original data set and obtain the two-dimensional detection box of the target object.

[0011] In the raw data processing module, if fusion with other images is not required, it is included in the labeled dataset. If image fusion is required, the foreground image and the background image are fused. During this process, the minimum bounding rectangle of the foreground image is used to obtain the labeled object detection box, and finally a labeled fused dataset is obtained. To make the fused image more harmonious, the fused image needs to be harmonized.

[0012] The present invention also provides a garbage data labeling method based on deep learning. Starting from zero labels, unlabeled data sets are collected, and a small number of labeled data sets are generated through background removal and fusion methods. Then a semi-automatic labeling model is trained to infer the new unlabeled data sets, and finally a large number of labeled garbage data sets are obtained.

[0013] The specific process of the present invention is as follows:

[0014] a) First, determine whether there is a mature semi-automatic model. If not, perform data processing on the original data set. If there is a semi-automatic model, use the semi-automatic model to infer the original data.

[0015] b) After processing the data, a labeled dataset is obtained, and the dataset is used to train and update the semi-automatic model.

[0016] c) The trained semi-automatic model is used to predict a new batch of raw data sets. If the prediction is accurate, the data is classified into the labeled data set. If the prediction is wrong, the data is returned to step b) for reprocessing.

[0017] The steps of generating a labeled dataset through background removal and fusion methods are as follows:

[0018] 1) First, a saliency detection operation is performed on the original data to obtain a binary mask with a value of 0 or 1. Then, based on the mask and the original color image as input, a deep learning cutout model is trained to complete the cutout operation;

[0019] 1.1) Use the saliency detection model to predict the original data and obtain the mask map;

[0020] 1.2) The mask image is preprocessed by edge dilation and erosion to obtain a ternary image consisting of foreground, background, and undetermined parts. Specifically, the mask image is dilated and eroded to obtain the edge transition region, whose value is between 0 and 1. The mask image is then classified and divided. When the pixel value is greater than an adjustable threshold, it is classified as the foreground image. When the mask value is less than a second threshold, it is classified as the background image. The transition region between the two is classified as the undetermined part.

[0021] 1.3) The obtained ternary image and the original image are input into a deep learning-based cutout model to predict an alpha channel image with fine edges. This channel describes the transparency of an image, with a value between 0 and 1, where 0 represents transparency and 1 represents opacity.

[0022] 1.4) Based on the above alpha channel map, the following principle is used to obtain a refined cutout, thus completing the cutout operation:

[0023] I=alpha*F+1-alpha)*B

[0024] Where F is the foreground area, B is the background area, alpha is the weight coefficient ranging from 0 to 1, and I is the composite image; when alpha is equal to 1, the foreground image is obtained, and when alpha is equal to 0, the background image is obtained.

[0025] 2) Manually detect the cutout effect. If the cutout effect is complete, the image is classified into the foreground image library. If the cutout effect is incomplete, readjust the model parameters and perform saliency detection and cutout again.

[0026] 3) After obtaining the foreground image, determine whether it needs to be fused with other images. If fusion is required, it is combined with the background image according to the set rules; if fusion is not required, the target detection frame is obtained based on the minimum enclosing rectangle of the foreground image and included in the labeled dataset.

[0027] 4) For the image after fusion in the previous step, since the features of the foreground image and the background image are different, it is necessary to use the style transfer method to transfer the style of the background image to the foreground image to make the styles of the two harmonious. The loss function of the style transfer model is obtained by the mean square error loss of the predicted value and the true value, as shown below:

[0028]

[0029] in is the value predicted by the model, I h,w is the true value, M h,w is the number of pixels occupied by the foreground image, and L(I) is the loss value.

[0030] 5) Through the above style transfer model, the foreground and background styles are made harmonious and unified. Then, the target detection frame is obtained based on the minimum bounding rectangle of the foreground image and included in the labeled dataset.

[0031] The beneficial effects of the present invention are:

[0032] 1. It can generate data sets that are more consistent with real-world scenarios, reducing data collection costs. It can also automatically obtain labels, reducing data annotation costs. Ultimately, it can achieve data-driven data annotation capabilities.

[0033] 2. This method starts from zero labeled data, reduces human intervention, and quickly labels new categories, thereby meeting the agile development and iteration of AI models and having important practical engineering value. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is the software module architecture diagram of the entire algorithm;

[0036] Figure 2 It is the software flow chart of the entire algorithm;

[0037] Figure 3 It is the software flow chart of the raw data processing module. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] The present invention provides a method of generating a labeled data set starting from zero labels through background removal and fusion methods, and then training a semi-automatic annotation model for prediction and iteration. Figure 1 As shown, this method includes the following modules:

[0040] 1. The module for acquiring the original data set. Figure 1As shown in the S100 module, unlabeled datasets can be collected through web crawling, open source datasets or scene collection as subsequent processing objects.

[0041] 2. Raw data processing module. Figure 1 As shown in module S200, saliency detection and image cropping are performed on the original data. The foreground image is removed and the minimum bounding rectangle is used to obtain the object detection bounding box. If fusion with other images is not required, the image is included in the labeled dataset. If image fusion is required, the foreground image and background image are fused together. During this process, the minimum bounding rectangle of the foreground image is used to obtain the labeled object detection bounding box. To make the fused image more harmonious, the fused image is harmonized. Finally, a labeled fused dataset is obtained.

[0042] 3. Semi-automatic model training module. Figure 1 As shown in module S300, the labeled fused dataset is fed into a semi-automatic annotation model (this method uses the YOLO series object detection algorithm), the loss is calculated, and the model parameters are updated using backpropagation until the model loss stops decreasing. Finally, a trained semi-automatic annotation model is obtained.

[0043] 4. Semi-automatic model reasoning module. Figure 1 As shown in the S400 module, the semi-automatic annotation model in the previous step is used to infer the new original data set to obtain the two-dimensional detection frame of the target object.

[0044] The above four modules perform actual processing according to the following flowchart based on business needs.

[0045] Figure 2 The software flow chart of the entire algorithm is shown. The specific process is as follows:

[0046] 1. First, determine whether there is a mature semi-automatic model. If not, perform data processing on the original data set. If there is a semi-automatic model, use the semi-automatic model to infer the original data.

[0047] 2. After data processing, obtain a labeled dataset, and use the dataset to train and update the semi-automatic model.

[0048] 3. The trained semi-automatic model is used to predict a new batch of raw data sets. If the prediction is accurate, the data is classified into the labeled data set. If the prediction is wrong, the data is placed in the data processing module for processing.

[0049] Figure 3 The software flow chart of the data processing module (S200) is shown. The specific process is as follows:

[0050] 1. First, perform saliency detection on the original data to obtain a mask image (a binary image with values of 0 or 1). Then, based on the mask image and the original color image as input, train a deep learning matting model (matting) to complete the matting operation. The specific steps are as follows:

[0051] a) Use the saliency detection model to predict the original data and obtain the mask map.

[0052] b) The mask image is then preprocessed by edge dilation and erosion to obtain a ternary image (trimap, which consists of three parts: foreground, background, and undetermined portion). Specifically, the mask image is dilated and eroded to obtain the edge transition region, whose value is between 0 and 1. The mask image is then classified and divided. When the pixel value is greater than a certain threshold (which can be adjusted), it is classified as the foreground image. When the mask value is less than a second threshold, it is classified as the background image. The transition region between the two is classified as the undetermined portion.

[0053] c) The obtained ternary image and the original image are then input into a deep learning-based cutout model to predict an alpha channel image with fine edges (this image describes the transparency of an image, with values ranging from 0 to 1, where 0 represents transparency and 1 represents opacity).

[0054] d) Based on the above alpha channel map, a refined cutout is obtained according to the following principle, thus completing the cutout operation.

[0055] I=alpha*F+(1-alpha)*B

[0056] Where F is the foreground area, B is the background area, alpha is the weight coefficient, ranging from 0 to 1, and I is the composite image. When alpha is equal to 1, the foreground image is obtained, and when alpha is equal to 0, the background image is obtained.

[0057] 2. Manually check the cutout effect. If the cutout effect is complete, place the image in the foreground image library. If the cutout effect is incomplete, readjust the model parameters and perform saliency detection and cutout again.

[0058] 3. After obtaining the foreground image, determine whether it needs to be fused with other images. If fusion is required, it is fused with the background image according to certain rules (such as larger near objects and smaller far objects). If fusion is not required, the object detection box is obtained based on the minimum bounding rectangle of the foreground image and included in the labeled dataset.

[0059] 4. After the fusion step, the foreground and background images have different characteristics such as color, lighting, and brightness. Therefore, it is necessary to use style transfer to transfer the style of the background image to the foreground image to make the two styles harmonious. The loss function of the style transfer model is obtained by the mean square error loss between the predicted value and the true value, as shown below.

[0060]

[0061] in is the value predicted by the model, I h,w is the true value, M h,w is the number of pixels occupied by the foreground image, and L(I) is the loss value.

[0062] 5. Through the above style transfer model, the foreground and background styles are made harmonious and unified. Then, the target detection frame is obtained based on the minimum bounding rectangle of the foreground image and included in the labeled dataset.

[0063] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, the above is only a preferred embodiment of the present invention. Since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this. Any technical personnel familiar with this technical field is within the technical scope disclosed by the present invention. For ordinary technical personnel in this technical field, changes or replacements that can be easily thought of should be covered within the protection scope of the present invention without departing from the principle of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A deep learning-based garbage data labeling system, characterized by: It includes an original data set acquisition module, an original data processing module, a semi-automatic model training module and a semi-automatic model reasoning module which are connected in sequence; The original dataset acquisition module collects unlabeled datasets through web crawling, open source datasets, or scene collection as the subsequent processing object; The raw data processing module performs saliency detection and cutout operations on the raw data, extracts the foreground image, and uses the minimum enclosing rectangle of the foreground image as the target detection frame to obtain a labeled dataset; The semi-automatic model training module inputs the labeled dataset into the semi-automatic annotation model, calculates the loss, and uses the backpropagation method to calculate and update the model parameters until the model loss stops decreasing. The training is then stopped, and the trained semi-automatic annotation model is finally obtained. The semi-automatic model inference module uses the semi-automatic annotation model in the previous step to infer the new original dataset and obtain the target detection box of the target object to expand the labeled dataset; In the raw data processing module, if it does not need to be fused with other pictures, it is included in the labeled data set; If image fusion is required, the foreground image and the background image need to be fused. In this process, the minimum enclosing rectangle of the foreground image is used to obtain the labeled target detection frame, and finally a labeled fusion dataset is obtained.

2. The deep learning-based garbage data labeling system according to claim 1, characterized in that: In the raw data processing module, harmony processing is performed when the images are fused.

3. The labeling method of the deep learning-based garbage data labeling system according to claim 1, characterized in that: Starting from zero labels, we collect unlabeled datasets, generate a small number of labeled datasets through background removal and fusion methods, and then train a semi-automatic annotation model to infer the new unlabeled datasets, and finally obtain a large number of labeled garbage datasets.

4. The deep learning-based garbage data labeling method according to claim 3, characterized in that: The specific process of the marking method is as follows: a) First, determine whether there is a mature semi-automatic model. If not, process the original data set. If a semi-automatic model is available, use the semi-automatic model to reason about the original data. b) Processing the data to obtain a labeled dataset, and using the dataset to train and update the semi-automatic model; c) The trained semi-automatic model is used to predict a new batch of original data sets. If the prediction is accurate, the data is classified into the labeled data set. If the prediction is wrong, the data is returned to step b) for reprocessing.

5. The deep learning-based garbage data labeling method according to claim 3 or 4, characterized in that: The steps of generating a labeled dataset through background removal and fusion methods are as follows: 1) First, a saliency detection operation is performed on the original data to obtain a binary mask with a value of 0 or 1. Then, based on the mask and the original color image as input, a deep learning cutout model is trained to complete the cutout operation; 2) Manually check the cutout effect. If the cutout effect is complete, the image is classified as a foreground image. If the cutout effect is incomplete, readjust the model parameters and perform saliency detection and cutout again. 3) After obtaining the foreground image, determine whether it needs to be fused with other images. If fusion is required, it is fused with the background image according to the set rules; If fusion is not required, the target detection frame is obtained based on the minimum bounding rectangle of the foreground image and included in the labeled dataset; 4) For the image after fusion in the previous step, since the features of the foreground image and the background image are different, it is necessary to use the style transfer method to transfer the style of the background image to the foreground image to make the styles of the two harmonious. The loss function of the style transfer model is obtained by the mean square error loss of the predicted value and the true value, as shown below. in is the value predicted by the model, I h,w is the true value, M h,w is the number of pixels occupied by the foreground image, and L(I) is the loss value; 5) Through the above style transfer model, the foreground and background styles are made harmonious and unified. Then, the target detection frame is obtained based on the minimum bounding rectangle of the foreground image and included in the labeled dataset.

6. The deep learning-based garbage data labeling method according to claim 5, characterized in that: The specific steps of step 1) are as follows: 1.1) Use the saliency detection model to predict the original data and obtain the mask map; 1.2) Perform preprocessing such as edge dilation and corrosion on the mask image to obtain a ternary image containing the foreground, background, and the part to be determined; 1.3) The obtained ternary image and the original image are input into a deep learning-based cutout model to predict an alpha channel image with fine edges. This channel describes the transparency of an image, with a value between 0 and 1, where 0 represents transparency and 1 represents opacity. 1.4) Based on the above alpha channel map, the following principle is used to obtain a refined cutout, thus completing the cutout operation: I=alpha*F+(1-alpha)*B Where F is the foreground area, B is the background area, alpha is the weight coefficient ranging from 0 to 1, and I is the composite image; when alpha is equal to 1, the foreground image is obtained, and when alpha is equal to 0, the background image is obtained.

7. The deep learning-based garbage data labeling method according to claim 6, characterized in that: The specific operation process of step 1.2) is as follows: the mask image is expanded and eroded to obtain the edge transition area, whose value is between 0 and 1; the mask image is then classified and divided. When the pixel value is greater than a certain adjustable threshold, it is classified as the foreground image; when the value of the mask image is less than a second threshold, it is classified as the background image, and the transition area between the two is classified as the part to be determined.

Images