A semi-supervised object detection method for RGB images based on double pseudo-label optimization learning

Abstract

The present invention discloses a semi-supervised target detection method for RGB images based on dual pseudo-label optimization learning. For the labeled images in the training batch, data augmentation operation is first performed on the images, and then the real labels are used to supervise the learning of the student model on the images. For the unlabeled images, their weakly enhanced images and strongly enhanced images are obtained respectively. The constructed pseudo-category optimization network and pseudo-bounding box optimization network are used to optimize the prediction results of the teacher model on the weakly enhanced images, thereby obtaining the pseudo-category labels and pseudo-bounding box labels of the images. The pseudo-category labels and pseudo-bounding box labels of the images are then used to supervise the learning of the student model on the weakly enhanced images. Finally, samples and for training the pseudo-category optimization network and the pseudo-bounding box optimization network are constructed on the labeled images, and then the corresponding labels are used to supervise the learning of the optimization network. The present invention can significantly increase the number of high-quality pseudo-labels, thereby improving the effect of RGB image target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a semi-supervised target detection method for RGB images. Background Art

[0002] Object detection in RGB images aims to identify the category and location of each object in the image. It plays a fundamental role in computer vision fields such as image interpretation, smart cities, security and safety inspections, and intelligent transportation. Deep neural networks, due to their powerful expressive power, offer a promising approach for object detection in RGB images, achieving good accuracy when trained with a large number of labeled samples. However, in practical applications, due to the high cost of labeling, only a few labeled samples are often available, which often leads to overfitting of deep neural networks. On the other hand, unlabeled data is readily available. Therefore, addressing the labeling cost issue in RGB image object detection through semi-supervised learning, which leverages a small amount of labeled data and a large amount of unlabeled data, is gaining increasing attention.

[0003] Existing deep semi-supervised object detection methods can be roughly divided into two categories based on the training process. One is consistency-based methods, which focus on forcing the model's outputs from different views of the same image to be close and consistent. For example, the paper "Jeong, Jisoo, et al. "Consistency-based semi-supervised learning for object detection." Advances in neural information processing systems 32 (2019)" makes the model's classification and regression outputs for the input image and its flipped views tend to be consistent. Another research direction is based on pseudo-labeling methods, which focus on leveraging the prediction results of the teacher model on unlabeled data. For example, the paper "Zhou, Qiang, et al. "Instant-teaching: An end-to-end semi-supervised object detection framework." Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2021" proposed using two data augmentation methods, MixUp and Mosaic, to combine labeled and unlabeled data to generate a large number of reliable objects. The document "Xu, Mengde, et al." End-to-end semi-supervised object detection with softteacher." Proceedings of the IEEE / CVF International Conference on Computer Vision.2021." proposed a new regression quality measurement method to select reliable prediction results as pseudo labels, and proposed to use the prediction confidence of the teacher model to weight the background samples of the student model to alleviate the negative impact of some inaccurate pseudo labels and reduce the negative gradient on unlabeled data. In recent years, both aspects of research have made great progress and tend to be combined with each other. Moreover, the most commonly used network architecture is the end-to-end pseudo-label semi-supervised framework, which uses the exponential sliding average of the student model, that is, the integrated model of the student model in time series as the teacher model, and uses strong and weak enhancement mechanisms to obtain more reliable pseudo labels. However, existing methods directly use the output of the teacher model as.Due to the lack of sufficient labeled samples for fitting the training model, the detection quality of the teacher model on unlabeled data is poor, which correspondingly limits the upper limit of the quality of the pseudo-labels obtained by the screening method, and thus limits the performance of semi-supervised object detection. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a semi-supervised target detection method for RGB images based on dual pseudo-label optimization learning. For the labeled images in the training batch, data augmentation operations are first performed on the images, and then the real labels are used to supervise the learning of the student model on the images. For unlabeled images, their weakly enhanced images and strongly enhanced images are obtained respectively. The constructed pseudo-category optimization network and pseudo-bounding box optimization network are used to optimize the prediction results of the teacher model on the weakly enhanced images, thereby obtaining the pseudo-category labels and pseudo-bounding box labels of the image. The pseudo-category labels and pseudo-bounding box labels of the image are then used to supervise the learning of the student model on the weakly enhanced images. Finally, samples and for training the pseudo-category optimization network and the pseudo-bounding box optimization network are constructed on the labeled images, and then the corresponding labels are used to supervise the learning of the optimization network. The present invention can significantly increase the number of high-quality pseudo-labels, thereby improving the effect of RGB image target detection.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Construct a student model and a teacher model. Both the student model and the teacher model are target detection networks. The student model is trained using a gradient update method, and the exponential sliding average algorithm is used to update the parameters of the student model to the teacher model. The teacher model does not participate in the gradient update.

[0007] For labeled RGB images Among them, h and w represent the height and width of the image respectively, which are marked as y l ={(c i , b i ), i=1…N}, represents the input image x l Contains N targets, for the i-th target, its category is c i , the bounding box coordinates are b i ; Transform the image x l Input into the student model to train and optimize the student model;

[0008] For unlabeled RGB images It will undergo two different image enhancement operations, namely strong image enhancement operation and weak image enhancement operation, to obtain the corresponding strong enhanced image and weak enhanced image:

[0009]

[0010]

[0011] in, For weak image enhancement operations, represents a strong image enhancement operation, and are the corresponding weakly enhanced images and strongly enhanced images respectively; the weakly enhanced images are used to generate pseudo labels for unlabeled images, while the strongly enhanced images and corresponding pseudo labels are used to allow the student model to perform semi-supervised learning;

[0012] Step 2: Input into the teacher model and generate the corresponding prediction results:

[0013]

[0014] in represents the teacher model, and preds is the corresponding test result;

[0015] Decouple the pseudo labels of the classification subtask and the regression subtask, and optimize the prediction results to obtain pseudo category labels and pseudo bounding box labels respectively;

[0016] The pseudo-classification optimization network uses secondary classification to correct the category of the target. For an object, the features corresponding to its position are first re-extracted. Then, the category is re-predicted through a network consisting of three fully connected layers. The new prediction result is combined with the previous one to obtain a final more accurate pseudo-class label:

[0017]

[0018] in, represents the pseudo-category optimization network;

[0019] The pseudo bounding box optimization network aggregates context and boundary information by translation and scaling, thereby improving the regression quality; for a bounding box {o u , o d ,d},o u The coordinates of its upper left point, o d The coordinates of its lower right point and d are the diagonal lengths of the box. First, the bounding box is moved along the four diagonal directions by a distance of γ×d to obtain four new boxes. Then, the bounding box is expanded twice, with the scaling factors being 1+2×t×γ, t∈{1,2}. The features of the original box and the features of the six newly obtained boxes are aggregated through convolution and full connection layers to obtain the optimized pseudo bounding box results as pseudo bounding box labels.

[0020]

[0021] in, represents the pseudo bounding box optimization network;

[0022] Step 3: Use the pseudo category labels and pseudo bounding box labels obtained in step 2 to supervise the prediction results of the student model on the strongly enhanced image:

[0023]

[0024]

[0025] in, represents the classification result predicted by the student model, represents the regression result predicted by the student model, l cls and l reg Represent the classification loss function and regression loss function in target detection, N u is the number of unlabeled images in the training batch; the training loss function of the student model on labeled data is:

[0026]

[0027]

[0028] in, represents the student model, represents the classification results predicted by the student model on the labeled image, Represents the regression results predicted by the student model on the labeled data, N l is the number of labeled images in the training batch; l reg (.) and l cls (.) represent the classification loss function and regression loss function in target detection respectively; during the training process, each training batch contains a certain proportion of labeled images and unlabeled images;

[0029] Step 4: Use random sampling method to optimize network training;

[0030] For a labeled RGB image x l , and the coordinate of the upper left corner of the i-th target o ui and the lower right corner coordinate o di , by randomly sampling pseudo boxes to simulate the pseudo labels output by the teacher model:

[0031]

[0032] in, and They represent the upper left corner coordinate and lower right corner coordinate of the jth pseudo frame obtained by sampling, s irepresents the size vector of the target box, θ is a predefined scale factor to control the sampling range, and are vectors randomly sampled from Gaussian distribution, Θ represents element-wise multiplication;

[0033] For the pseudo bounding box optimization network, set θ = θ reg , sample N for each target box reg pseudo boxes as training samples; for the pseudo category optimization network, the training samples consist of the following three parts: sampling for each target box indivual The pseudo-frame and indivual Pseudo boxes and the output of the region proposal network of the teacher network; θ reg It represents the variance of the Gaussian distribution of the training sample positions sampled by the pseudo bounding box optimization network, represents the variance of the Gaussian distribution of the positive training sample locations sampled by the pseudo-category optimization network, represents the variance of the Gaussian distribution of negative training sample locations sampled by the pseudo bounding box optimization network;

[0034] After constructing the training samples, the two optimization networks are trained with the corresponding real labels. The loss function of the pseudo bounding box optimization network is the GIoU loss function, and the loss function of the pseudo category optimization network is the standard cross entropy loss function. Finally, the total loss function is:

[0035]

[0036] Among them, λ u is the weight coefficient that balances the loss function on the labeled image and the loss function on the unlabeled image, To optimize the loss function of the network; is the loss function on the labeled image, is the loss function on unlabeled images.

[0037] Preferably, the pseudo-category optimization network consists of three fully connected layers: the first fully connected layer is activated by ReLU; the second fully connected layer does not use an activation function; and the third layer is a fully connected layer.

[0038] Preferably, the two target detection network stages are Faster-R-CNN target detection architectures.

[0039] The beneficial effects of the present invention are as follows:

[0040] Due to the limited amount of labeled data, the generalization ability of the teacher model is insufficient. Therefore, directly selecting reliable results from the teacher model's predictions as pseudo-labels can lead to problems such as a small number of pseudo-labels and missed detections. Optimizing the dual pseudo-label optimization network can significantly increase the number of high-quality pseudo-labels, thereby improving the performance of object detection in RGB images. DETAILED DESCRIPTION

[0041] The present invention is further described below with reference to the embodiments.

[0042] In response to the problems in the prior art, the present invention proposes a dual pseudo-label optimization learning method for semi-supervised object detection in RGB images. It includes two new pseudo-label optimization learning modules, namely the optimization network on the classification subtask and the optimization network on the regression subtask. During the training process of the basic detector, the two networks learn and model the mapping relationship between pseudo labels and true labels, and use this mapping relationship to narrow the gap between pseudo labels and true labels, thereby generating higher quality pseudo labels. In order to successfully learn this mapping relationship, a Gaussian random sampling method is proposed to obtain suitable pseudo-label-true label sample pairs, and these sample pairs are used to train the optimization network. The entire process is carried out in an end-to-end framework. Through this method, the optimization network can adaptively model the mapping between the output of the teacher model and the true label, thereby improving the quality of the pseudo labels, and then improving its detection performance and generalization ability of semi-supervised object detection.

[0043] A semi-supervised object detection method for RGB images based on double pseudo-label optimization learning includes the following steps:

[0044] Step 1: Construct a student model and a teacher model. Both the student model and the teacher model are target detection networks. The student model is trained using a gradient update method, and the exponential sliding average algorithm is used to update the parameters of the student model to the teacher model. The teacher model does not participate in the gradient update.

[0045] For labeled RGB images Among them, h and w represent the height and width of the image respectively, which are marked as y l ={(c i , b i ), i=1…N}, represents the input image x l Contains N targets, for the i-th target, its category is c i , the bounding box coordinates are b i ; Transform the image x l Input into the student model and train and optimize the student model using the normal target detection loss function;

[0046] For unlabeled RGB images It will undergo two different image enhancement operations, namely strong image enhancement operation and weak image enhancement operation, to obtain the corresponding strong enhanced image and weak enhanced image:

[0047]

[0048]

[0049] in, For weak image enhancement operations, represents a strong image enhancement operation, and are the corresponding weakly enhanced images and strongly enhanced images respectively; the weakly enhanced images are used to generate pseudo labels for unlabeled images, while the strongly enhanced images and corresponding pseudo labels are used to allow the student model to perform semi-supervised learning;

[0050] Step 2: Input into the teacher model and generate the corresponding prediction results:

[0051]

[0052] in represents the teacher model, and preds is the corresponding test result;

[0053] Decouple the pseudo labels of the classification subtask and the regression subtask, and optimize the prediction results to obtain pseudo category labels and pseudo bounding box labels respectively;

[0054] The pseudo-classification optimization network uses secondary classification to correct the category of the target. For an object, the features corresponding to its position are first re-extracted. Then, the category is re-predicted through a network consisting of three fully connected layers. The new prediction result is combined with the previous one to obtain a final more accurate pseudo-class label:

[0055]

[0056] The pseudo bounding box optimization network aggregates context and boundary information by translation and scaling, thereby improving the regression quality; for a bounding box {o u , o d ,d},o u The coordinates of its upper left point, o d The coordinates of its lower right point and d are the diagonal lengths of the box. First, the bounding box is moved along the four diagonal directions by a distance of γ×d to obtain four new boxes. Then, the bounding box is expanded twice, with the scaling factors being 1+2×t×γ, t∈{1,2}. The features of the original box and the features of the six newly obtained boxes are aggregated through convolution and full connection layers to obtain the optimized pseudo bounding box results as pseudo bounding box labels.

[0057]

[0058] in, represents the pseudo bounding box optimization network;

[0059] Step 3: Use the pseudo category labels and pseudo bounding box labels obtained in step 2 to supervise the prediction results of the student model on the strongly enhanced image:

[0060]

[0061]

[0062] in, represents the classification result predicted by the student model, represents the regression result predicted by the student model, l cls and l reg Represent the classification loss function and regression loss function in target detection, N u is the number of unlabeled images in the training batch; the training loss function of the student model on labeled data is:

[0063]

[0064]

[0065] in, represents the student model, represents the classification results predicted by the student model on the labeled image, represents the regression result predicted by the student model on the labeled data, and Nl is the number of labeled images in the training batch; during the training process, each training batch contains a certain proportion of labeled images and unlabeled images;

[0066] Step 4: Use random sampling method to optimize network training;

[0067] For a labeled RGB image x l , and the coordinate of the upper left corner of the i-th target o ui and the lower right corner coordinate o di , by randomly sampling pseudo boxes to simulate the pseudo labels output by the teacher model:

[0068]

[0069] in, and They represent the upper left corner coordinate and lower right corner coordinate of the jth pseudo frame obtained by sampling, s irepresents the size vector of the target box (i.e., length and width), θ is a predefined scale factor to control the sampling range, and are vectors randomly sampled from Gaussian distribution, Θ represents element-wise multiplication;

[0070] For the pseudo bounding box optimization network, set θ = θ reg , sample N for each target box reg pseudo boxes as training samples; for the pseudo category optimization network, the training samples consist of the following three parts: sampling for each target box indivual The pseudo-frame and indivual Pseudo boxes and the output of the region proposal network of the teacher network;

[0071] After constructing the training samples, the two optimization networks are trained with the corresponding real labels. The loss function of the pseudo bounding box optimization network is the GIoU loss function, and the loss function of the pseudo category optimization network is the standard cross entropy loss function. Finally, the total loss function is:

[0072]

[0073] Among them, λ u is the weight coefficient that balances the loss function on the labeled image and the loss function on the unlabeled image, To optimize the loss function of the network;

[0074] To simultaneously train the student detector model and the optimization network model in an end-to-end training framework while avoiding optimization difficulties caused by gradient coupling between the two models, an alternating optimization mechanism is used to decouple the gradient flow. Specifically, while updating one model, the parameters of the other are fixed. This has the advantage of decoupling gradients and better optimizing the models. Specific embodiment:

[0076] 1. Data preprocessing

[0077] For a given training set It can be divided into labeled data The dataset is represented as and unlabeled data The dataset is represented as in For the labeled data in the dataset in is the input image, y={(c i , b i ), i=1…N} represents the input image x lContains N targets, for the i-th target, its category is c i , the bounding box coordinates are b i For unlabeled data in is the input image.

[0078] In addition, for the labeled data image x l The data enhancement methods of random scale jitter, random overexposure, random brightness jitter, random contrast jitter and random erasure are used. u ,Weak image enhancement operations include random scale jitter, and strong image enhancement operations include random scale jitter, random overexposure, random brightness jitter, random contrast jitter, random translation, random rotation, random shear, and random erasure.

[0079] 2. Model structure and initialization

[0080] This paper adopts the standard two-stage Faster-R-CNN object detection architecture. The backbone network parameters of the detector are initialized using a residual network pre-trained on ImageNet. The remaining parameters are initialized randomly.

[0081] The present invention designs the structures of a pseudo-category optimization network and a pseudo-bounding box optimization network to achieve better optimization results. The pseudo-category optimization network consists of three fully connected layers: the first fully connected layer has an input channel dimension of 12544 and an output channel dimension of 1024, activated by Reluctant Unit (ReLU); the second fully connected layer has an input channel dimension of 1024 and an output channel dimension of 1024, without using an activation function; and the third fully connected layer has an input channel dimension of 1024 and an output channel dimension corresponding to the number of data categories plus one (using the COCO dataset as an example, the output channel dimension is 81).

[0082] 3. Training process

[0083] Each training batch contains a certain proportion of labeled images and unlabeled images. The model is optimized using the SGD optimizer. For the labeled images in the training batch, data augmentation is first performed on the images, and then the true labels are used to supervise the learning of the student model on the images. For the unlabeled images in the training batch, their weakly enhanced images and strongly enhanced images are obtained respectively. The constructed pseudo-category optimization network and pseudo-bounding box optimization network are used to optimize the prediction results of the teacher model on the weakly enhanced images, thereby obtaining the pseudo-category labels and pseudo-bounding box labels of the images. The pseudo-category labels and pseudo-bounding box labels of the images are then used to supervise the learning of the student model on the weakly enhanced images. Finally, samples and for training the pseudo-category optimization network and pseudo-bounding box optimization network are constructed on the labeled images, and then the corresponding labels are used to supervise the learning of the optimization network.

[0084] 4. Update the teacher model

[0085] After the gradient of each training batch of samples is returned, the exponential moving average method is used to update the parameters of the student model to the teacher model:

[0086]

[0087] Among them, θ i represents the parameters of the teacher model at the i-th moment, represents the parameters of the student model at the i-th moment, and α is a hyperparameter used to control the update speed.

Claims

1. A semi-supervised object detection method for RGB images based on double pseudo-label optimization learning, characterized in that: The steps include: Step 1: Construct a student model and a teacher model. Both the student model and the teacher model are target detection networks. The student model is trained using a gradient update method, and the exponential sliding average algorithm is used to update the parameters of the student model to the teacher model. The teacher model does not participate in the gradient update. For labeled RGB images ,in, and Represents the height and width of the image, which are marked as , represents the input image Include goal, for the targets, whose categories are , the bounding box coordinates are ; The image Input into the student model to train and optimize the student model; For unlabeled RGB images , it will undergo two different image enhancement operations, namely strong image enhancement operation and weak image enhancement operation, to obtain the corresponding strong enhanced image and weak enhanced image: in, For weak image enhancement operations, represents a strong image enhancement operation, and are the corresponding weakly enhanced images and strongly enhanced images respectively; the weakly enhanced images are used to generate pseudo labels for unlabeled images, while the strongly enhanced images and corresponding pseudo labels are used to allow the student model to perform semi-supervised learning; Step 2: Input into the teacher model and generate the corresponding prediction results: in (.) represents the teacher model, To obtain the corresponding test results; Decouple the pseudo labels of the classification subtask and the regression subtask, and optimize the prediction results to obtain pseudo category labels and pseudo bounding box labels respectively; The pseudo-classification optimization network uses secondary classification to correct the category of the target. For an object, the features corresponding to its position are first re-extracted. Then, the category is re-predicted through a network consisting of three fully connected layers. The new prediction result is combined with the previous one to obtain a final more accurate pseudo-class label: in, represents the pseudo-category optimization network; The pseudo bounding box optimization network aggregates context and boundary information by translation and scaling, thereby improving the regression quality; for a bounding box , The coordinates of its upper left point, The coordinates of its lower right point, For the diagonal length of the box, first move the bounding box along the four diagonal directions Length to get four new frames, then expand the bounding box twice, the scaling factor is ; Then the features of the original frame corresponding position and the features of the six newly obtained frames corresponding positions are aggregated through convolution and full connection layers to obtain the optimized pseudo bounding box result as the pseudo bounding box label : in, represents the pseudo bounding box optimization network; Step 3: Use the pseudo category labels and pseudo bounding box labels obtained in step 2 to supervise the prediction results of the student model on the strongly enhanced image: in, represents the classification result predicted by the student model, represents the regression results predicted by the student model, and Represent the classification loss function and regression loss function in target detection, respectively. is the number of unlabeled images in the training batch; the training loss function of the student model on labeled data is: in, represents the student model, represents the classification results predicted by the student model on the labeled image, represents the regression results predicted by the student model on the labeled data, is the number of labeled images in the training batch; and They represent the classification loss function and regression loss function in target detection respectively. During the training process, each training batch contains a certain proportion of labeled images and unlabeled images. Step 4: Use random sampling method to optimize network training; For labeled RGB images , and its The upper left corner coordinates of the target and the lower right corner coordinates , by randomly sampling pseudo boxes to simulate the pseudo labels output by the teacher model: in, and Respectively represent the sampling The coordinates of the upper left corner and lower right corner of the pseudo box, represents the size vector of the target box, is a predefined scale factor to control the sampling range, and are vectors randomly sampled from Gaussian distribution, Represents element-wise multiplication; For the pseudo bounding box optimization network, set , sample for each target box pseudo boxes as training samples; for the pseudo category optimization network, the training samples consist of the following three parts: sampling for each target box indivual The pseudo-frame and indivual Pseudo boxes and the output of the region proposal network of the teacher network; It represents the variance of the Gaussian distribution of the training sample positions sampled by the pseudo bounding box optimization network, represents the variance of the Gaussian distribution of the positive training sample locations sampled by the pseudo-category optimization network, represents the variance of the Gaussian distribution of negative training sample locations sampled by the pseudo bounding box optimization network; After constructing the training samples, the two optimization networks are trained with the corresponding real labels. The loss function of the pseudo bounding box optimization network is the GIoU loss function, and the loss function of the pseudo category optimization network is the standard cross entropy loss function. Finally, the total loss function is: in, is the weight coefficient that balances the loss function on the labeled image and the loss function on the unlabeled image, To optimize the loss function of the network; is the loss function on the labeled image, is the loss function on unlabeled images.

2. The method for semi-supervised object detection in RGB images based on double pseudo-label optimization learning according to claim 1, characterized in that: The pseudo-category optimization network consists of three fully connected layers: the first fully connected layer is activated by ReLU; the second fully connected layer does not use an activation function; and the third layer is a fully connected layer.

3. The method for semi-supervised object detection in RGB images based on double pseudo-label optimization learning according to claim 1, characterized in that: The object detection method described adopts a two-stage Faster-R-CNN object detection architecture.

Images