Semi-supervised target detection method based on pseudo labels

Abstract

The invention discloses a semi-supervised target detection method based on pseudo labels, which is applied to the technical field of computer vision and comprises the following steps: acquiring a semi-supervised target detection data set; carrying out training on the labeled data set by referring to a full-supervision target detection method to obtain a student model and copying a teach model; weak data enhancement is carried out on the unlabeled data set, then the unlabeled data set is sent to the teach model, and a pseudo label is obtained after a prediction result is output and subjected to non-maximum suppression; performing strong data enhancement on the pseudo label and the unlabeled data set, then training a student model, and updating a teach model weight parameter through index moving average. Based on a pseudo label direction, an efficient attention mechanism is used on a feature extraction network, the importance of each channel in a feature map is dynamically calibrated, unimportant features are inhibited, global context information is learned, and the feature learning ability of the network is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision, and specifically relates to a semi-supervised target detection method based on pseudo labels. Background Art

[0002] With the rapid development of computer vision technology, object detection, as one of the important tasks, is widely used in fields such as autonomous driving, video surveillance, and medical image analysis. However, traditional object detection methods usually require a large amount of labeled data to train the model, and the collection and labeling of this data is time-consuming and expensive. In practical applications, there is often only a limited amount of labeled data, while a large amount of data is unlabeled. Therefore, how to improve the accuracy of object detection by using a large amount of unlabeled data under limited labeled data has become an important research issue.

[0003] In order to study this problem, semi-supervised target detection came into being. It aims to train a target detection model with superior performance by fusing a small amount of labeled data and a large amount of unlabeled data. Under this framework, researchers have proposed a variety of technical methods, such as pseudo-label generation and consistency regularization, to make full use of the information in unlabeled data. Pseudo-label technology predicts unlabeled data through a trained model and generates pseudo-labels as supervisory signals; while consistency regularization enhances the robustness and generalization ability of the model by requiring the model to maintain consistency in predictions for unlabeled data under different transformations. Semi-supervised target detection not only reduces the dependence on labeled data, but also significantly improves the training efficiency and performance of the model, providing a more economical and efficient solution for practical applications. With the continuous advancement of technology, semi-supervised target detection is expected to play an important role in more fields and promote the further development of computer vision technology. Summary of the invention

[0004] In order to solve the problem of low target detection accuracy caused by the low quality of pseudo-labels, the present invention proposes a semi-supervised target detection method based on pseudo-labels. In view of the uncertainty of pseudo-label generation and the difficulty of screening, this method optimizes the unlabeled data set used to train neural networks by data enhancement without using manual annotation, and processes the pseudo-labels using a deep network. There are two main technical problems to be solved by the present invention. First, a large amount of unlabeled data lacks clear supervisory information, which greatly increases the difficulty of extracting information; second, the reliability of pseudo-labels generated by predicting unlabeled data is limited by the performance of the model, and may be erroneous or misleading, resulting in poor model performance.

[0005] For problem one, the present invention uses strong and weak data enhancement methods for unlabeled data, expands the data set and improves the generalization ability of the model, increases the training samples of the model and reduces the risk of overfitting; for problem two, the present invention designs a new attention mechanism, performs spatial transformation on the input unlabeled data so that the model can better capture key features, and then adaptively recalibrates the features to enhance the expression of important features, introduces an adaptive feature pooling module into feature fusion, obtains information from all feature layers to improve the model's training on unlabeled data, fuses low-level features and shortens the information path to improve positioning accuracy. The technical solutions adopted by the present invention are as follows:

[0006] A semi-supervised target detection method based on unlabeled data includes the following steps:

[0007] Step 1: Obtain a semi-supervised target detection dataset, where the semi-supervised target detection dataset includes a labeled dataset and an unlabeled dataset.

[0008] Step 2: According to the fully supervised target detection method on the labeled data set, for each input sample, the model predicts the label of the sample to make it as consistent as possible with the true value, obtains the student model through loss function training, and copies a teacher model.

[0009] Step 3: Perform weak data enhancement on the unlabeled data, including rotation, flipping, scaling, cropping, color change, Gaussian noise, and Gaussian blur, and send it to the teacher model. After feature extraction and feature fusion, the prediction results are output, and the pseudo labels are obtained through improved non-maximum suppression.

[0010] Step 4: Perform strong data augmentation on the pseudo-labeled and unlabeled datasets, including geometric transformation and pixel-level transformation, and train the student model.

[0011] Step 5: Based on the teacher model, the consistency loss generated is recorded as the unsupervised loss, and based on the supervised training of the student model, the loss generated is recorded as the supervised loss, and the sum is the overall loss.

[0012] Step 6: The student model updates its own parameters based on the overall loss back propagation, and the teacher model updates its own weight parameters based on the student model through exponential moving average.

[0013] Specific:

[0014] Step 1 includes collecting image information and establishing a semi-supervised target detection data set; wherein part of the image data is annotated with a completely accurate calibration frame, and the proportion of the part of the image data in all the collected image data does not exceed 20%;

[0015] The target detection network described in step 2 includes: using a feature extraction network to extract image features, wherein the feature extraction network selects Resnet-50; using a feature fusion network to further enhance the features, wherein the feature fusion network selects FPN; selecting RPN for candidate region generation and preliminary classification and bounding box regression; selecting Rol Align for feature alignment and fixed-size output; optimizing the loss function to keep the output prediction results as consistent as possible with the annotation information;

[0016] The weak data enhancement processing of unlabeled data described in step 3 uses the following weak data enhancement methods: rotation, flipping, scaling, cropping, color transformation, Gaussian noise, and Gaussian blur; wherein rotation refers to rotating the image around its center point by a certain angle; flipping includes horizontally flipping the image along the vertical axis and vertically flipping the image along the horizontal axis; scaling is to adjust the width and height of the image to change the image size; cropping is to randomly select a part of the original image as a new image sample; color transformation is to adjust the brightness, contrast, and saturation of the image to simulate the image changes under different lighting conditions; Gaussian noise refers to the fluctuation of the image pixel value around a certain mean and gradually decreases with the increase of the mean distance; Gaussian blur blurs the image by taking a weighted average of the pixel values ​​of each pixel point and its domain. ;

[0017] In step 3, the unlabeled data after weak data enhancement is sent to the teacher model, and the STN module is added to the residual module in the Resnet-50 feature extraction network. The extracted features are further processed through a unique spatial transformation to enhance the representation ability of the features, alleviate the noise and interference in the data, and suppress unimportant or redundant information; the SENet module is integrated into the STN module to further enhance the network's sensitivity to key features, so that important features receive more attention, thereby improving the quality of feature representation;

[0018] In step 3, the unlabeled data after weak data enhancement is sent to the teacher model. In the FPN feature pyramid network, an adaptive feature pooling module is added to combine the shallow feature map with rich detail information and the deep feature map with stronger semantic information to more accurately detect objects of different scales.

[0019] In step 3, the unlabeled data after weak data enhancement is sent to the teacher model. The output prediction results are passed through NMS to obtain pseudo labels. IoU-Guided NMS is used instead of ordinary NMS. The boxes with higher positioning accuracy are given priority to eliminate errors caused by misleading classification confidence. The IoU prediction branch is introduced to flexibly handle detection tasks in different scenarios.

[0020] Step 4 performs strong data enhancement processing on the pseudo labels and unlabeled data obtained in step 3, and the strong data enhancement methods used include geometric transformation, color transformation, image mixing and synthesis, noise and occlusion, and area cropping and filling; wherein, geometric transformation refers to rotating the image at a random or fixed angle, horizontally flipping and vertically flipping, horizontally or vertically translating the image, changing the scale of the image, and applying more complex transformations to the image; color transformation includes brightness adjustment, contrast adjustment, saturation adjustment, and color jittering; image mixing and synthesis includes image mixing of multiple images mixed in a certain proportion, image splicing of multiple images together, and copying and pasting for small target detection; noise and occlusion include adding random Gaussian noise and salt and pepper noise to the image, and randomly occluding some areas in the image; area cropping and filling refers to randomly cropping a part of the image and filling the cropped area with random values, background or other image content;

[0021] In step 5, the cross entropy loss of the student model trained with labeled data is recorded as the supervised loss, and the consistency loss of the pseudo-label obtained by training the teacher model with unlabeled data is recorded as the unsupervised loss. During the training process, the supervised loss and the unsupervised loss are calculated simultaneously, and the weighted sum of the two is obtained according to the hyperparameters to obtain the final loss function;

[0022] The parameters of the student model described in step 6 are updated through the back-propagation algorithm to minimize the weighted sum of the supervised loss and the unsupervised loss; the parameters of the teacher model are the weighted average of the student model parameters, and the weights decay exponentially over time.

[0023] Beneficial effects of the present invention:

[0024] First, the present invention uses a data enhancement method to preprocess the data, learn more feature representations from limited labeled data, and better utilize the information in unlabeled data, which can improve the generalization ability of the model and reduce the risk of overfitting. Secondly, the present invention designs a new attention mechanism to add to the feature extraction module, adaptively adjusts the spatial layout of the input features and enhances the focus on key features, thereby improving the model's detection ability at different positions and scales. Thirdly, the present invention incorporates an adaptive feature pooling module into the feature fusion module, which can fuse information in all feature levels to generate richer and more expressive feature representations, and compensates for the shortcomings of the high level in positioning by using low-level feature information and shortens the information path from the bottom layer to the top layer. Then, the present invention adopts loU-GuidedNMS instead of ordinary NMS in terms of consistency loss, attenuates the confidence of the detection box according to the size of loU, thereby improving the recall rate, and eliminating the error mechanism caused by misleading classification confidence to improve the accuracy of target detection. Finally, the present invention does not require manual labeling of unlabeled data, and the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of semi-supervised target detection based on pseudo labels;

[0026] Figure 2 This is the structural diagram of the designed attention mechanism;

[0027] Figure 3 It is the structural diagram of feature fusion module; Specific implementation plan

[0028] In order to more clearly demonstrate the technical solutions and features of the present invention, the present invention is explained below with reference to the accompanying drawings, but the present invention is not limited to the examples.

[0029] Embodiment 1:

[0030] A pseudo-label-based semi-supervised target detection method, the detection method comprising:

[0031] Step 1: Obtain a semi-supervised target detection dataset D: The semi-supervised target detection dataset D includes a labeled dataset D S And the unlabeled dataset D U . There are labeled datasets M is the number of samples in the labeled data set, and the sample set is recorded as are the 1st, 2nd, ..., Mth samples respectively; the corresponding labeled set is recorded as They are Corresponding annotation information, for D S Each element in Recorded as sample and annotation The formed pair i is a positive integer; unlabeled dataset N is the number of samples in the unlabeled dataset. U Each element in Recorded as sample and annotation The formed pair i is a positive integer, From unlabeled sample set From unlabeled sample set in The elements In the pseudo-label generation step, the element Assignment;

[0032] Step 2: In the labeled dataset D S According to the fully supervised target detection method, for each input sample Predict sample labels through the model Make it as consistent as possible with the true value, obtain the student model through loss function training, and copy a teacher model;

[0033] Step 3: Train the teacher model for the unlabeled dataset X U Weak data enhancement is performed, including rotation, flipping, scaling, cropping, color change, Gaussian noise, and Gaussian blur. The resulting set is recorded as The network of the teacher model uses the feature extraction network to extract image features, and the feature extraction network selects Resnet-50; the feature fusion network is used to further enhance the features, and the feature fusion network selects FPN; RPN is selected for candidate region generation and preliminary classification and bounding box regression; Rol Align is selected for feature alignment and fixed-size output; the loss function is optimized to keep the output prediction results as consistent as possible with the annotation information; Figure 2 As shown in Figure 2, a new attention mechanism is designed to be added to the feature extraction network Resnet-50. First, the unlabeled dataset with weak data enhancement is After Resnet-50 feature extraction, the feature map XU∈R is obtained H×W×C , the localization network takes a feature map U with a height of H, a width of W, and a channel of C, regresses the output θ through multiple hidden layers, and the parameters of the spatial transformation θ applied to the feature map U are: θ = f loc (U), the size of θ varies according to the type of parameterized change. When the transformation is a two-dimensional radial transformation, θ is a 6-dimensional matrix: Then, the grid generator transforms the input feature map using the transformation parameters θ output by the localization layer, defining the output pixels as being located on a regular grid z = {Z i},in Form an output feature map Assumptions is a two-dimensional affine transformation G θ , then the point-by-point transformation is: in are the target coordinates of the regular grid in the output feature map, G is the source coordinates of the sampling points defined in the input feature map, θ is the affine transformation matrix, using the height and width to normalize the coordinates so that When within the spatial boundaries of the output, Secondly, the feature map F output by the grid generator is used as the input of SENet, and the feature map of each channel is compressed by global average pooling to obtain a channel descriptor that can represent global information. It is compressed by the spatial dimension of F, so that the calculation formula of the Cth element of Q is: Then use a lightweight network with two fully connected layers, one followed by a ReLU activation function and the other followed by a Sigmoid activation function. The formula is: s = σ(g(Q,W)) = σ(W2ξ(W1Q)), where ξ is the ReLU activation function. and To learn the nonlinear relationship between channel descriptors and generate the weight of each channel, the learned weight is then multiplied by the original feature map to achieve the recalibration of channel features. Finally, in order to perform spatial transformation on the input feature map, the sampler takes a set of sampling points Together with the input feature map F, a sampled output feature map P is generated, each In The coordinates define the spatial location in the input at which the sampling kernel is applied to obtain the value at a specific pixel in the output P, ​​and the formula is written as: in and Φ x and Φ y are the parameters that define the general sampling kernel k() for image interpolation; Figure 3As shown in the figure, the feature maps generated by the attention mechanism are then sent to the FPN feature fusion module. First, feature maps are extracted from different layers of Resnet-50. These feature maps have different resolutions and semantic information, including low-level feature maps with high resolution but less semantic information and high-level feature maps with low resolution but rich semantic information. The high-level feature maps are upsampled to a higher spatial resolution to make them consistent with the spatial resolution of the low-level feature maps. The upsampled feature maps are element-by-element added to the corresponding low-level feature maps. This process can fuse high-resolution detail information and low-resolution semantic information to generate new feature maps. These feature maps have richer semantic information and higher spatial resolution. In FPN, proposals are assigned to different feature layers according to their size, with small proposals assigned to lower feature layers and large proposals assigned to higher feature layers. However, this is not optimal. In addition, the importance of a feature may not be strongly correlated with the layer to which it belongs. High-level features are generated by large receptive fields and extract richer contextual information. Allowing small proposals to obtain these features can better utilize useful contextual information for prediction. Similarly, low-level features contain rich details and better positioning accuracy, and it is obviously beneficial to let large proposals obtain this information. Therefore, for each proposal, features of all levels are extracted and fused, and then subsequent classification and regression are performed based on this fused feature.Specifically, for each proposal, first map it to all feature levels, perform ROI Align processing on each feature level, and then fuse the featuregrids of different levels, that is, after entering a parameter layer respectively, fuse between the first and second convolutional layers fc. The fc layer is position-sensitive, and the prediction of different spatial positions is realized through a set of variable parameters. Therefore, it has the ability to adapt to different spatial positions. At the same time, the prediction of each spatial position is based on the global information of the entire candidate domain; the prediction result output by the teacher is loU-Guided NMS is used to screen reliable pseudo-labels, that is, an additional branch is added to the detection framework to predict the positioning confidence of each box. First, the candidate bounding boxes predicted by the teacher output include coordinates and confidence scores. The candidate boxes are preliminarily sorted according to the classification confidence, and then the candidate bounding boxes are sorted in descending order according to their predicted loU values. The bounding box with the highest loU value is selected from the sorted list as the optimal bounding box of the current category, and it is added to the final output list. The remaining bounding boxes are traversed and the loU value with the optimal bounding box is calculated. If it is greater than the preset threshold, the bounding box is removed because the overlap with the optimal bounding box is high and may be redundant. Then repeat the above two steps until all candidate bounding boxes are processed. In each iteration, the one with the highest loU value in the current remaining bounding box is selected, and the bounding box overlapping with it is suppressed. Finally, the bounding box optimized by loU-guided NMS, that is, the pseudo-label, is obtained.

[0034] Step 4: Train the student model for unlabeled data D U The pseudo-label set generated by the teacher model is subjected to strong data enhancement methods, including geometric transformation, color transformation, image mixing and synthesis, noise and occlusion, region cropping and filling, to obtain a new unlabeled dataset. and Then train the student model; iterate the training until the model loss function converges to a stable result.

[0035] Step 5: Calculate the total loss. The cross entropy loss of the student model trained with labeled data is recorded as the supervised loss. The calculation formula is: Where M is the number of samples with labeled data, y ip is 0 or 1. If the true category of the i-th sample is p, then is the probability that the student model predicts that the i-th sample belongs to the p-th class. The consistency loss of the pseudo-label obtained by training the teacher model with unlabeled data is recorded as the unsupervised loss, and the calculation formula is: KLi,p is the pseudo label of the Pth class in the i-th image, is the predicted probability of the teacher model for the pth class of the ith image, where P is the number of categories, is the teacher model’s prediction of the bounding box of the i-th image, B i is the bounding box in the pseudo-label, SmoothL1 is the smoothed L1 loss function, and λ cls and λ reg It is a hyperparameter that calculates supervised loss and unsupervised loss simultaneously during training, and performs a weighted summation of the two according to the hyperparameter to obtain the final loss function: L = L S +L U ;

[0036] Step 6: Weight update, parameter θ of the student model s Update by minimizing the overall loss function L, calculated as: Where η is the learning rate, is the total loss function with respect to the student model parameters θ s The gradient of the teacher model t The parameters of the student model are updated in the form of EMA, and the calculation formula is: Where j is the current iteration number and α is the parameter update weight.

[0037] Experimental results:

[0038] The semi-supervised dataset is sent to both the conventional target detection model and the model of this example to verify the detection effect. The experimental results are shown in Table 1.

[0039] Table 1 Results show

[0040]

[0041] The model of the present invention exceeds the supervised model in terms of detection speed and accuracy. In addition, the present invention processes targets of different sizes and proportions by using the importance of each channel in the dynamic calibration feature map, thereby enhancing the feature learning ability of the network, retaining key features, and improving computational efficiency.

[0042] The above is a detailed description of the specific implementation of the present invention. It should be understood that individual details are not limited to the above specific implementations. Those skilled in the art can make various variations or modifications within the scope of the claims without affecting the essential content of the present invention.

Claims

1. A pseudo-label-based semi-supervised target detection method, characterized in that: The following steps are involved: Step 1: Obtain a semi-supervised target detection dataset, wherein the semi-supervised target detection dataset includes a labeled dataset and an unlabeled dataset; Step 2: According to the fully supervised target detection method on the labeled data set, for each input sample, the model is used to predict the label of the sample so that it is as consistent as possible with the true value, the student model is obtained through loss function training, and a teacher model is copied; Step 3: Perform weak data enhancement on the unlabeled data, including rotation, flipping, scaling, cropping, color change, Gaussian noise, and Gaussian blur, and send it to the teacher model. After feature extraction and feature fusion, the prediction results are output, and the pseudo labels are obtained through improved non-maximum suppression. Step 4: Perform strong data augmentation on the pseudo-label and unlabeled datasets, including geometric transformation and pixel-level transformation, and train the student model; Step 5: Based on the teacher model, the consistency loss generated is recorded as the unsupervised loss, and based on the supervised training of the student model, the loss generated is recorded as the supervised loss, and the sum is the overall loss; Step 6: The student model updates its own parameters based on the overall loss back propagation, and the teacher model updates its own weight parameters based on the student model through exponential moving average.

2. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: Step 1 includes collecting image information and establishing a semi-supervised target detection data set; wherein, part of the image data is annotated with a completely accurate calibration frame, and the proportion of this part of the image data in all the collected image data does not exceed 20%.

3. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: The target detection network described in step 2 includes: using a feature extraction network to extract image features, wherein the feature extraction network selects Resnet-50; using a feature fusion network to further enhance the features, wherein the feature fusion network selects FPN; selecting RPN for candidate region generation and preliminary classification and bounding box regression; selecting RolAlign for feature alignment and fixed-size output; optimizing the loss function to keep the output prediction results as consistent as possible with the annotation information.

4. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: The weak data enhancement processing for unlabeled data described in step three uses the following weak data enhancement methods: rotation, flipping, scaling, cropping, color transformation, Gaussian noise, and Gaussian blur; wherein rotation refers to rotating the image around its center point by a certain angle; flipping includes horizontally flipping the image along the vertical axis and vertically flipping the image along the horizontal axis; scaling is to adjust the width and height of the image to change the size of the image; cropping is to randomly select a part of the original image as a new image sample; color transformation is to adjust the brightness, contrast, and saturation of the image to simulate image changes under different lighting conditions; Gaussian noise refers to the fluctuation of image pixel values ​​​​around a certain mean and gradually decreases with the increase of the mean distance; Gaussian blur is to blur the image by taking a weighted average of the pixel values ​​​​of each pixel point in the image and its area.

5. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: As described in step 3, the unlabeled data after weak data enhancement is sent to the teacher model, and the STN module is added to the residual module in the Resnet-50 feature extraction network. The extracted features are further processed through unique spatial transformation to enhance the feature representation ability, alleviate the noise and interference in the data, and suppress unimportant or redundant information; the SENet module is integrated into the STN module to further enhance the network's sensitivity to key features, so that important features receive more attention, thereby improving the quality of feature representation.

6. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: In step 3, the unlabeled data after weak data enhancement is sent to the teacher model. In the FPN feature pyramid network, an adaptive feature pooling module is added to combine the shallow feature map with rich detail information and the deep feature map with stronger semantic information to more accurately detect objects of different scales.

7. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: As described in step 3, the unlabeled data after weak data enhancement is sent to the teacher model, and the output prediction results are subjected to NMS to obtain pseudo labels. IoU-Guided NMS is used instead of ordinary NMS, and boxes with higher positioning accuracy are given priority to eliminate errors caused by misleading classification confidence. The IoU prediction branch is introduced to flexibly handle detection tasks in different scenarios.

8. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: Step 4 performs strong data enhancement processing on the pseudo labels and unlabeled data obtained in step 3, and the strong data enhancement methods used include geometric transformation, color transformation, image mixing and synthesis, noise and occlusion, and area cropping and filling; wherein, geometric transformation refers to rotating the image at a random or fixed angle, flipping the image horizontally and vertically, translating the image horizontally or vertically, changing the scale of the image, and applying more complex transformations to the image; color transformation includes brightness adjustment, contrast adjustment, saturation adjustment, and color jittering; image mixing and synthesis includes image mixing of multiple images mixed in a certain proportion, image stitching of multiple images together, and copying and pasting for small target detection; noise and occlusion include adding random Gaussian noise and salt and pepper noise to the image, and randomly occluding some areas in the image; area cropping and filling refers to randomly cropping a part of the image and filling the cropped area with random values, background or other image content.

9. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: In step 5, the cross entropy loss of the student model trained with labeled data is recorded as the supervised loss, and the consistency loss of the pseudo-label obtained by training the teacher model with unlabeled data is recorded as the unsupervised loss. During the training process, the supervised loss and the unsupervised loss are calculated simultaneously, and the weighted sum of the two is performed according to the hyperparameters to obtain the final loss function.

10. The pseudo-label based semi-supervised target detection method according to claim 1, characterized in that: The parameters of the student model described in step 6 are updated through the back-propagation algorithm to minimize the weighted sum of the supervised loss and the unsupervised loss; the parameters of the teacher model are the weighted average of the student model parameters, and the weights decay exponentially over time.

Images