Weakly supervised object detection method based on semi-supervised method and noise processing

Abstract

The application provides a weakly supervised object detection method based on a semi-supervised method and noise processing. The weakly supervised object detection method based on the semi-supervised method and the noise processing comprises the following steps: S1: obtaining a weakly supervised object detection data set, wherein the labels of all images are image-level labels; S2: pre-extracting candidate boxes for all images in the weakly supervised object detection data set, and using the candidate boxes for training of a weakly supervised object detection model; S3: constructing a weakly supervised object detection model M w training and generating initial instance-level noise pseudo labels; S4: constructing a semi-supervised object detection model M s training the semi-supervised object detection model M s using the noise pseudo labels generated in the step S3; S5: processing the noise pseudo labels in the training, and dividing the noise pseudo labels into credible labels and uncredible labels, corresponding to labeled data and unlabeled data in a semi-supervised task; and inputting the labels and data into the M s for semi-supervised task training; S6: using the trained M s to predict new label information and replace the label information to be inputted into the step S4 for iterative training, and repeating K times to obtain a final trained object detection model.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to a target detection method under weakly supervised learning. Background Technology

[0002] Object detection is a popular area in computer vision and digital image processing, enabling the automatic detection of object categories and locations, reducing the consumption of manpower and capital, and has significant practical implications. Compared to more basic image classification tasks, object detection adds a regression task, requiring not only the algorithm to determine the presence of objects in an image, but also to mark their locations within the image and perform regression prediction on the marked bounding boxes. In recent years, due to the widespread application of deep learning, object detection algorithms have developed rapidly. However, typical fully supervised object detection models are trained on large datasets with precise manual annotations. These methods require fine-grained, high-quality annotations for each training image. Often, an image contains multiple object instances, each potentially belonging to different categories, all of which require manual annotation. Some instances are even difficult to discern with the naked eye due to their small size, environmental influences, or image distortion, further increasing the difficulty of dataset annotation and the financial and human resource costs.

[0003] Therefore, to reduce the resource consumption caused by labeling large-scale object detection datasets, researchers have begun to focus on how to achieve good performance in object detection tasks using coarse-grained labels that are easier to obtain and label. A typical coarse-grained task setting is where the labels in the dataset only have image-level category information, lacking instance-level localization information. This setting allows for direct research using existing image classification datasets and also enables rapid labeling on new datasets, significantly reducing the labeling cycle and accelerating deployment. Currently, coarse-grained datasets with only image-level (ImageLayer) annotations have become a major focus of weakly supervised object detection research.

[0004] However, due to the lack of instance information, the final performance of most weakly supervised object detection methods still lags significantly behind that of fully supervised object detection methods. Therefore, further research on weakly supervised object detection methods is of great significance. Summary of the Invention

[0005] Currently, existing weakly supervised target detection methods, both domestically and internationally, still lag significantly behind fully supervised methods and are limited to fixed weakly supervised detection frameworks, with little innovation in methodology and procedures.

[0006] Based on this, this invention designs a weakly supervised object detection method based on semi-supervised methods and noise processing, establishes a self-training iterative method, and fully utilizes the instance-level supervision information that is lacking but extremely crucial in weakly supervised tasks using semi-supervised object detection methods, while additionally processing pseudo-label noise to ensure model performance. A similar iterative self-training paradigm was proposed in patent CN115019133A, but the advantages of this invention are: ① It utilizes instance-level pseudolabels in self-training instead of just image-level pseudolabels, which can better improve object detection prediction performance. ② It does not require pre-extraction of candidate boxes from images during prediction; the trained model can be used immediately for prediction. ③ It creatively combines the excellent achievements of current semi-supervised object detection, establishing a semi-weakly supervised framework training paradigm, which not only achieves better model performance but also has significant implications for exploring new research ideas.

[0007] The specific content of this invention is as follows: A weakly supervised object detection method based on semi-supervised and noise processing includes the following steps: S1: Obtain a weakly supervised object detection dataset, wherein all images are labeled at the image level; S2: Pre-extract candidate boxes from all images in the weakly supervised object detection dataset for training the weakly supervised object detection model; S3: Construct a weakly supervised object detection model M. w S4: Train and generate initial instance-level noisy pseudo-labels; S5: Construct a semi-supervised object detection model M s M is trained using the noise pseudo-labels generated in step S3. s Noise and pseudo-labels are processed during training, categorized into trustworthy and untrustworthy labels, corresponding to labeled and unlabeled data in the semi-supervised task. The labels and data are then fed into M. s Perform semi-supervised task training; S5: Use the trained M s Predict and generate new label information and replace The label information is re-input into step S4 for iterative training, repeated K times, to obtain the final trained object detection model.

[0008] Compared with the prior art, the present invention has the following innovative features:

[0009] 1. It innovatively incorporates semi-supervised target detection methods and establishes a training paradigm for a semi-weakly supervised framework. This can be organically combined with the development of other fields to achieve collaborative innovation and progress, which is of great significance for exploring new research ideas in the future.

[0010] 2. Pseudo-labels are treated as noise labels, and an effective learning strategy is designed for noise labels, avoiding the use of loss size sorting method to achieve better model prediction results.

[0011] 3. Experiments showed that, while keeping other factors constant, the mean average precision (mAP) of the previous model on the NWPU VHR-10 dataset (https: / / opendatalab.com / NWPU_VHR-10) was improved by 5%, demonstrating the effectiveness of the method (this model was only used for testing the effectiveness of the method, and its numerical results do not represent the actual performance of the model).

[0012] The method execution flow provided in the embodiments of this application can be run on devices such as personal computers, servers, embedded computing devices, and cloud computing platforms. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1 The flowchart is a weakly supervised target detection method based on semi-supervised methods and noise processing according to the present invention.

[0015] Figure 2A and 2B The results of object detection on image data according to embodiments of this application are shown. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0017] According to embodiments of this application, the data used comes from the NWPU VHR-10 remote sensing image dataset (https: / / opendatalab.com / NWPU_VHR-10). Compared to ordinary images in daily life, remote sensing images are characterized by small targets, densely packed objects, and arbitrary orientations, which brings more challenges.

[0018] 1. Obtain the NWPU VHR-10 remote sensing image dataset. Select 650 images as training data, of which 65 images retain their annotation information as the labeled dataset, and 585 images are used as the unlabeled dataset.

[0019] 2. Use the selection search method to pre-extract candidate boxes for all images in the training set, which are then used to train the weakly supervised object detection model.

[0020] 3. Construct a basic weakly supervised object detection model M w Its backbone network is ResNet. After extracting image features, it extracts pre-extracted candidate bounding box features from the feature map and feeds them into a region pooling layer (https: / / arxiv.org / abs / 1506.01497) to normalize the extracted features. Each candidate region generates a fixed-length feature vector after pooling. Then, a multi-instance learning detection head feeds the fixed-length feature vector into two prediction branches: a classification branch and a detection branch. The predicted class probability scores are then multiplied to obtain the image-level class probability. and the image-level label c of the i-th image i Constructing the cross-entropy loss function

[0021] Where N c Let be the total number of categories in the dataset, and let i represent the i-th image. This represents the predicted probability that the i-th image belongs to a certain category, for example, y represents the probability that the i-th image belongs to category Nc, where c represents the image category. i ={c i} is the image x i ∈R C×H×W The label information, where C is the number of feature map channels, H is the image height, and W is the image width. N is the image-level label vector. c c is the total number of categories in the dataset. i Each dimension value indicates whether the corresponding category exists in the current image; for example, a value of 1 indicates existence, and a value of 0 indicates non-existence. Label c i It uses image-level tags and does not have instance-level tags, y icThe α parameter represents whether the i-th image in the labeled image-level data belongs to class c. A fixed-length feature vector is then input into a feature refinement module (e.g., https: / / arxiv.org / abs / 1704.00138), which generates predictions using m branches and self-trains to optimize them. The k-1 branches provide training and supervision information for the k-th branch. During prediction, the results of all m branches are combined to generate the final result. This final result is then used as the initial instance-level bounding box pseudo-labels. This includes all pseudo-label bounding boxes on the image; since these pseudo-labels are not perfectly accurate predictions, they contain potential noise information; the noise pseudo-label for the i-th image. At this point, the iteration count is 0. The noise pseudo-labels are automatically generated by the algorithm rather than being verified by humans as true labels. Furthermore, due to the lack of instance-level supervision information, the labels generated in step 3 are usually inaccurate and contain noise.

[0022] 4. Construct a basic semi-supervised object detection model M s Use the noise pseudo-labels generated in step 3 Training M s During training, noisy pseudo-labels are classified using confidence levels. Pseudo-labels with an instance class prediction probability greater than a threshold δ are considered reliable labels; otherwise, they are considered unreliable labels (see also...). Figure 1 (Label noise filtering). Trusted and untrusted labels correspond to labeled and unlabeled data in a semi-supervised task, respectively. The labels and data are then fed into M... s Semi-supervised training is performed. The dataset is divided into labeled dataset D based on the credibility of the labels. l and unlabeled dataset D u Two parts, in the labeled dataset D l Use the usual classification loss. and regression loss As the loss function, the total loss function for labeled data is: Where i represents the sample number, N l The labeled dataset D used in training l The total number of samples, Refers to the i-th image in the labeled data; in the unlabeled dataset D u The above uses self-training loss. Optimize the model, denoted as Where i represents the sample number, N u The unlabeled dataset D used in training u The total number of samples, The i-th image in the unlabeled data refers to the final semi-supervised object detection model M. s Using the total loss function Conduct training.

[0023] 5. Use the trained semi-supervised object detection model M s Predict and generate new label information and replace The label information is then used to re-enter step 4 and repeat K times, where i represents the sample number. Let the generated label information in the k-th iteration be... Used to replace the label information in the (k-1)th iteration After K iterations, the final trained weakly supervised target detection model based on semi-supervised methods and noise reduction is obtained.

[0024] Figure 2A and 2B The results of object detection on image data according to embodiments of this application are shown.

[0025] The image to be detected is provided to the semi-supervised object detection model M trained according to the embodiments of this application. s The model outputs the target recognition result, in Figure 2A and Figure 2B The image below shows the annotation results for different images. In addition to annotating the detected categories such as airplane, storage, and ship, the location of each target and the predicted probability of category recognition are also marked with boxes.

[0026] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application. Clearly, those skilled in the art can make various alterations and variations to this application without departing from its spirit and scope. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A weakly supervised target detection method based on semi-supervised methods and noise processing, characterized in that, Includes the following steps: S1: Obtain a weakly supervised object detection dataset, where all images are labeled at the image level; S2: Pre-extract candidate boxes from all images in the weakly supervised object detection dataset for training the weakly supervised object detection model; S3: Construct a weakly supervised object detection model Train and generate initial instance-level noise pseudo-labels based on pre-extracted candidate boxes. The instance-level noise pseudo-labels include the predicted instance's location, category prediction result, and corresponding prediction probability; wherein, a weakly supervised target detection model is constructed. The backbone network is ResNet, which is used to extract image features and extract the features of pre-extracted candidate boxes on the feature map. Region pooling layers are used to normalize the features of the pre-extracted candidate boxes, resulting in fixed-length feature vectors. A multi-instance learning detection head feeds these fixed-length feature vectors into two prediction branches: a classification branch and a detection branch. The predicted class probability scores from each branch are then multiplied to obtain the image-level class probability. and with image-level tags Constructing the cross-entropy loss function ;in Let be the total number of categories in the dataset, and let i represent the i-th image. This represents the predicted probability that the i-th image belongs to a certain category. For image Label information; input fixed-length feature vectors into the feature refinement module, and use... The first branch generates predictions and self-trains to optimize them. -1 is the first branch Branch generation provides supervision information for training, integrating all The results from each branch generate the final result; train the weakly supervised object detection model. , will integrate all The results generated from each branch serve as the initial instance-level label. ; S4: Construct a semi-supervised object detection model Using the initial instance-level noise pseudo-labels generated in step S3 Training a semi-supervised object detection model Noise and pseudo-labels are processed during training, and are divided into reliable and unreliable labels, corresponding to labeled and unlabeled data in the semi-supervised task. The labels and data are then fed into the semi-supervised object detection model. Semi-supervised task training was conducted; among them, In step S4, the semi-supervised object detection model The model is trained using a consistent loss method combined with a standard object detection model; the input dataset is divided into labeled datasets based on the credibility of instance-level noise pseudo-labels. and unlabeled datasets Two parts, in the labeled dataset The above semi-supervised target detection model Supervised training was performed using common classification loss methods. and regression loss As the loss function, the total loss function for labeled data is: ,in, This represents the labeled dataset used in training. The total number of samples, Refers to the i-th image in labeled data; in unlabeled datasets... The above uses self-training loss. Optimize the model, denoted as Where i represents the sample number, This represents the unlabeled dataset used in training. The total number of samples, For the i-th image in the unlabeled data, the final semi-supervised object detection model Ms uses the total loss function. Conduct training; S5: Using the trained semi-supervised object detection model Predict and generate new label information and replace The label information is fed back into step S4 for iterative training, repeated K times, where K is 1 to 3 times, to obtain the final trained object detection model. In step S3, a weakly supervised target detection model is used. Generate initial instance-level label boxes representing the 0th iteration. The instance-level label generated during the k-th iteration is: { }, For the first The total number of predicted instances in the image. ,in The x and y coordinates of the center of the instance frame. To calibrate the width and height of the frame, For the class prediction results of the instance, For the first The first image One prediction instance, ; with dataset Image-level label vectors Combined with the updated first Noise pseudo-labels on the images , This represents the current iteration number. For dataset The image-level label vector of the i-th image.

2. The method according to claim 1, characterized in that, In step S1, the dataset Where N is the total number of images in the dataset, Represents the i-th image in the dataset. For image Tag information, Image-level label vectors, The total number of categories in the dataset. Each dimension value indicates whether the corresponding category exists in the current image; a value of 1 indicates existence, and a value of 0 indicates non-existence; label These are image-level labels, excluding instance-level labels, where C is the number of feature map channels, H is the image height, and W is the image width.

3. The method according to claim 1, characterized in that, In step S2, a selection search algorithm is used to pre-extract candidate boxes for the image.

4. The method according to claim 1, characterized in that, In step S4, the confidence level of noisy pseudo-labels is further subdivided based on the instance-level class prediction probability. When using confidence level division, prediction results with instance class prediction probabilities greater than a threshold δ are considered reliable labels; otherwise, they are considered unreliable labels. Reliable labels constitute the dataset. Untrusted labels constitute the dataset .

5. The method according to claim 1, characterized in that, In step S5, the existing noise pseudo-label information is updated after each iteration to continuously optimize the pseudo-label quality and eliminate noise; the model of the final iteration is used as the prediction model for various scenarios.

6. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images