Semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting

Abstract

The invention relates to a semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting. The method comprises the following steps: S1, constructing a remote sensing target detection data set comprising annotated and unannotated samples; s2, designing a confidence coefficient weighting mechanism, and adjusting the weight of the false labeling confidence coefficient of the teacher model on the unlabeled sample in training according to the false labeling confidence coefficient of the teacher model on the unlabeled sample; s3, constructing a semi-supervised target detection network by using an average teacher architecture, and integrating supervision loss and confidence coefficient weighted semi-supervised loss; and S4, completing model evaluation and selection, outputting a final target detection model, and completing a remote sensing target detection task by using the target detection model. According to the method, the unlabeled image can be effectively utilized, the accuracy and robustness of target detection are improved, and the method is suitable for a remote sensing target detection task in a complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a semi-supervised remote sensing target detection method based on pseudo-label confidence weighting. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, target detection methods based on this technology have been widely used in automatic driving, face detection and other fields. Although we are currently in the era of big data, in specific remote sensing application scenarios, due to factors such as data scarcity and high labeling cost, there are fewer labeled samples. Traditional fully supervised learning methods face serious challenges, and this situation makes the research of semi-supervised target detection particularly important. Among them, insufficient labeled samples and the accumulation of error information in pseudo-labeling are important factors affecting the accuracy improvement of semi-supervised target detection models.

[0003] In view of the problem that false pseudo-labeling has a negative impact on model accuracy, researchers usually alleviate this problem by improving the pseudo-labeling screening strategy. This improvement generally uses confidence to preliminarily screen the prediction results of the teacher model, and obtains pseudo-labeling data with high confidence, so as to reduce the error information in pseudo-labeling. Although methods such as double-threshold filtering and dynamic threshold filtering have been used to screen more effective pseudo-labeling, the prediction results after screening are not further refined, and the utilization efficiency of high-confidence pseudo-labeling data needs to be further improved.

[0004] Therefore, it is a technical problem to be solved to design a semi-supervised remote sensing target detection method based on pseudo-labeling confidence weighting. SUMMARY

[0005] To solve the technical problems existing in the prior art, the purpose of the present application is to provide a semi-supervised remote sensing target detection method based on pseudo-labeling confidence weighting, which can effectively utilize unlabeled images, improve the accuracy and robustness of target detection, and is suitable for remote sensing target detection tasks in complex scenes.

[0006] To achieve the above application purpose, the present application provides a semi-supervised remote sensing target detection method based on pseudo-labeling confidence weighting, comprising the following steps:

[0007] Step S1, constructing a remote sensing target detection dataset including labeled and unlabeled samples;

[0008] Step S2, designing a confidence weighting mechanism to adjust the weight of the unlabeled samples in training according to the confidence of the teacher model;

[0009] Step S3, constructing a semi-supervised target detection network using an average teacher architecture, integrating a supervised loss and a confidence-weighted semi-supervised loss;

[0010] Step S4, complete model evaluation and selection, output the final target detection model, and complete the remote sensing target detection task using the target detection model.

[0011] According to one of the technical solutions of the present application, in the step S1, specifically comprising:

[0012] Step S11, obtaining remote sensing image data of the scene where the target detection task is located, and pre-processing;

[0013] Step S12, labeling according to the target category;

[0014] Step S13, dividing into labeled data set and unlabeled data set according to whether having category label and position label information;

[0015] Step S14, taking part of the labeled data set as verification data, and the remaining labeled images and all unlabeled images constitute the training data set.

[0016] According to one of the technical solutions of the present application, in the step S2, the reciprocal of the confidence of the teacher model output result is taken as the weight of the unlabeled image classification and regression loss, and the remaining loss does not apply additional weight, the overall loss is recalculated, and the weight parameters of the model are updated by back propagation;

[0017] The semi-supervised overall loss of the unlabeled image is represented as:

[0018]

[0019]

[0020] Where L u , respectively represent the semi-supervised overall loss function of the unlabeled image, the classification loss function in the semi-supervised loss, the regression loss function and the confidence loss function, ω(.) represents the indicator function, if the condition is met, the function value is 1, if the condition is not met, the function value is 0, p h,w represents the confidence of the teacher model prediction result at the feature map h, w position, CE(.,.) is the cross entropy loss function, L GIOU is the IOU loss function based on GIOU metric, τ1 and τ2 are preset threshold values;

[0021] The semi-supervised classification loss of the unlabeled image is respectively represent the classification results of the prediction of the student model and the teacher model at the feature map h, w position, is the indicator function, and the classification prediction result of the current teacher model is selected according to the confidence of the teacher prediction result whether to join the calculation of the semi-supervised loss function; if the confidence p h,w is higher than the preset threshold τ2, the indicator function is 1, otherwise 0.

[0022] The semi-supervised regression loss of the unlabeled image is respectively represent the regression results of the prediction of the student model and the teacher model at the feature map h, w position, is an indicator function, which filters the regression prediction result of the current teacher model according to the confidence of the teacher prediction result whether to join the calculation of the semi-supervised loss function; if the confidence p h,w is higher than the preset threshold τ2, or the indicator function is 1, otherwise 0, wherein represents the confidence that the object is an object.

[0023] The semi-supervised confidence loss of the unlabeled image is respectively represent the confidence results of the student model and the teacher model that the feature map h, w position is a foreground object.

[0024] According to one of the technical solutions of the present application, in the step S3, specifically comprising:

[0025] Step S31, input the labeled image into the student model, then compare the prediction result with the labeled information, calculate the classification loss regression loss confidence loss The sum is represented as:

[0026]

[0027] wherein, respectively represent the classification results of the student model and the real label at the feature map h, w position, respectively represent the regression results of the student model and the real label at the feature map h, w position, respectively represent the confidence results of the student model and the real label that the feature map h, w position is a foreground object and the corresponding confidence label;

[0028] Step S32, the unlabeled image is input into the student model and the teacher model after strong and weak data augmentation, the prediction result of the teacher model is filtered and screened by double thresholds to obtain pseudo-labels, guiding the training process of the student model, represented as:

[0029]

[0030] Wherein, ω(·) represents an indicator function, if the condition is met, the function value is 1, if the condition is not met, the value is 0, τ1 represents the low threshold value in the double threshold value, τ2 represents the high threshold value in the double threshold value;

[0031] Step S33, add the supervised loss and the semi-supervised loss as the final overall loss L, denoted as:

[0032] L=L s +λL u ;

[0033] Wherein λ is used to adjust the proportion of semi-supervised loss in the total loss.

[0034] Step S34, input all labeled images and unlabeled images in the training data set into the algorithm according to the requirements, calculate the overall loss and update the student model parameters by back propagation Denoted as:

[0035]

[0036] Wherein, SGD represents the stochastic gradient descent algorithm, and θ represents the model weight parameter;

[0037] Step S35, the student model parameters are transmitted to the teacher model by sliding average, denoted as:

[0038]

[0039] Wherein, The weight parameter of the teacher model at the t time step is represented as, The weight parameter of the student model at the t time step is represented as, and α is a smoothing parameter for controlling the weight update of the teacher model;

[0040] Step S36, repeat the steps S31 to S35 until the model training round reaches the preset number of times.

[0041] According to one of the technical solutions of the application, in the step S4, specifically comprising:

[0042] Step S41, evaluate the effect of the model on the target detection task, and select the model with the highest precision on the validation set during the training process as the final output model;

[0043] Step S42, complete the remote sensing target detection task through the target detection model.

[0044] According to one of the technical solutions of the application, in the step S41, the effect of the model on the target detection task is evaluated using mAP@50 and mAP@50:95, and the evaluation score of the model on the validation set is calculated by 0.1*mAP@50+0.9*mAP@50:95.

[0045] Select the model with the highest evaluation score as the final semi-supervised target detection model.

[0046] According to one of the technical solutions of the application, in step S42, an image to be detected is selected from the test data set, the image to be detected is preprocessed, the processed image is input into the trained semi-supervised target detection model, and the corresponding class information and position information of the target in the image to be detected are output, thereby completing the target detection task.

[0047] Compared with the prior art, the application has the following beneficial effects:

[0048] By adding the confidence weighted semi-supervised loss function into the remote sensing target detection task, on the one hand, the unlabeled data is effectively utilized through the semi-supervised learning method, and on the other hand, based on the double threshold filtering, the confidence reciprocal weighting strategy is introduced to refine the processing of the prediction result. This method not only fully utilizes the effective information in the teacher model result, improves the importance of high confidence prediction in the loss function, relatively reduces the influence of low confidence prediction, guides the model to learn more from reliable prediction, effectively reduces the accumulation of pseudo-labeling errors in training, greatly improves the utilization efficiency of high confidence pseudo-labeling data, effectively utilizes unlabeled images, and improves the accuracy and robustness of target detection. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0050] Figure 1 A flowchart schematically showing a pseudo-labeling confidence weighted semi-supervised remote sensing target detection method according to an embodiment of the present application;

[0051] Figure 2 A structural schematic diagram schematically showing a pseudo-labeling confidence weighted semi-supervised remote sensing target detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0053] As shown in Figure 1 and Figure 2 , a semi-supervised remote sensing target detection method based on pseudo-labeled confidence weighting of the present application comprises the following steps:

[0054] Step S1, constructing a remote sensing target detection dataset comprising labeled and unlabeled samples, specifically comprising:

[0055] Step S11, obtaining remote sensing image data of the scene where the target detection task is located and performing preprocessing;

[0056] Step S12, labeling according to target categories;

[0057] Step S13, dividing into a labeled dataset and an unlabeled dataset according to whether there is category label and position label information;

[0058] Step S14, taking part of the labeled dataset as verification data, and the remaining labeled images and all unlabeled images constitute a training dataset.

[0059] Through diversified preprocessing of the dataset, its adaptability under different environments and conditions is ensured, and the detection capability of the model for various targets is enhanced. At the same time, the dataset is divided into a labeled dataset and an unlabeled dataset, so that the model can further mine information in unlabeled data on the basis of using existing labeled data, thereby improving the overall detection performance, and then under the condition of few labeled data and massive unlabeled data in most task scenarios, high-precision remote sensing target detection is realized.

[0060] Step S2, designing a confidence weighting mechanism to adjust the weight of unlabeled samples in training according to the confidence of the teacher model;

[0061] The confidence weighted semi-supervised loss used refers to adjusting the weight of the unlabeled sample in the training according to the confidence of the model on the unlabeled sample. Specifically, in order to increase the weight proportion of the high confidence prediction result in the overall loss, the reciprocal of the confidence is used as an amplification factor on the semi-supervised loss. After the unlabeled image is input into the teacher model, the confidence of the prediction result is also saved, and after the pseudo label is screened out by the pseudo label assigner, the confidence of the pseudo label is added to the calculation of the semi-supervised loss function. The specific implementation method is as follows: the reciprocal of the confidence of the teacher model output result is used as the weight of the unlabeled image classification and regression loss, and the remaining loss does not apply additional weight, then the overall loss is recalculated, and the model weight parameters are updated by back propagation.

[0062] The semi-supervised overall loss of the unlabeled image is represented as:

[0063]

[0064]

[0065] wherein L u , respectively represent the semi-supervised overall loss function of the unlabeled image, the classification loss function, the regression loss function and the confidence loss function in the semi-supervised loss, ω(.) represents an indicator function, the function value is 1 if the condition is met, and the function value is 0 if the condition is not met, p h,w represents the confidence of the teacher model prediction result at the feature map h, w position, CE(.,.) is a cross entropy loss function, L GIOU is an IOU loss function based on GIOU metric, τ1 and τ2 are preset threshold values;

[0066] The semi-supervised classification loss of the unlabeled image is respectively represent the predicted classification results of the student model and the teacher model at the feature map h, w position, is an indicator function, which filters the classification prediction result of the current teacher model according to the confidence of the teacher prediction result whether to join the calculation of the semi-supervised loss function; if the confidence p h,w is higher than the preset threshold τ2, the indicator function is 1, otherwise it is 0;

[0067] The semi-supervised regression loss of the unlabeled image is respectively represent the predicted regression results of the student model and the teacher model at the feature map h, w position, is an indicator function, which filters the regression prediction result of the current teacher model according to the confidence of the teacher prediction result whether to join the calculation of the semi-supervised loss function, if the confidence p h,w is higher than the preset threshold τ2, or the indicator function is 1, otherwise 0, where represents the confidence that the object object is.

[0068] The semi-supervised confidence loss of the unlabeled image is respectively represent the confidence results of the student model and the teacher model that the feature map h, w position is a foreground object.

[0069] Step S3, using the average teacher architecture to construct a semi-supervised target detection network, integrating the supervised loss and the confidence weighted semi-supervised loss;

[0070] In the step S3, the semi-supervised target detection model is constructed, specifically including:

[0071] Step S31, inputting the labeled image into the student model, then comparing the prediction result with the labeled information, calculating the classification loss regression loss confidence loss The sum is represented as:

[0072]

[0073] wherein, respectively represent the classification results of the student model and the real label at the feature map h, w position, respectively represent the regression results of the student model and the real label at the feature map h, w position, respectively represent the confidence results of the student model and the real label that the feature map h, w position is a foreground object and the corresponding confidence label;

[0074] Step S32, after the unlabeled image is subjected to strong and weak data enhancement, it is respectively input into the student model and the teacher model, the prediction result of the teacher model is filtered and selected by double threshold value to obtain pseudo label, guiding the training process of the student model, represented as:

[0075]

[0076] wherein, ω(·) represents the indicator function, if the condition is met, the value is 1, if the condition is not met, the value is 0, τ1 represents the low threshold value in the double threshold value, and τ2 represents the high threshold value in the double threshold value;

[0077] Step S33, adding the supervised loss and the semi-supervised loss as the final overall loss L, represented as:

[0078] L = L s + λL u ;

[0079] wherein λ is used to adjust the proportion of semi-supervised loss in the total loss.

[0080] Step S34, input all labeled images and unlabeled images in the training data set into the algorithm according to the requirements, calculate the overall loss, and then update the student model parameters through back propagation, which is represented as:

[0081]

[0082] wherein SGD represents the stochastic gradient descent algorithm, and θ represents the model weight parameter.

[0083] Step S35, pass the student model parameters to the teacher model through the sliding average method, which is represented as:

[0084]

[0085] wherein, represents the weight parameter of the teacher model at the t-th time step, is the weight parameter of the student model at the t-th time step, and α is a smoothing parameter for controlling the weight update of the teacher model.

[0086] Step S36, repeat the steps S31 to S35 until the model training round reaches the preset number of times.

[0087] Step S4, complete the model evaluation and selection, output the final target detection model, and complete the remote sensing target detection task using the target detection model. In the step S4, specifically comprising:

[0088] Step S41, evaluate the effect of the model on the target detection task, and select the model with the highest precision on the validation set during the training process as the final output model.

[0089] Use mAP@50 and mAP@50:95 to evaluate the effect of the model on the target detection task, calculate the evaluation score of the model on the validation set by 0.1*mAP@50+0.9*mAP@50:95, and select the model with the highest evaluation score as the final semi-supervised target detection model.

[0090] Step S42, complete the remote sensing target detection task through the target detection model.

[0091] Select the image to be detected from the test data set, pre-process the image to be detected, input the processed image into the trained semi-supervised target detection model, output the class information and position information of the target in the image to be detected, and complete the target detection task.

[0092] Taking the LEVIR dataset as an example, the LEVIR dataset contains three types of targets, which are airplanes, ships and oil tanks. 3792 remote sensing images in the LEVIR dataset are divided into a training set (2398 images) and a verification set (1394 images) in proportion; then the training set is randomly divided into a labeled data set (1%, 23 images) and an unlabeled data set (99%, 2375 images); since all the data in the set are labeled data, the application uses the form of removing the corresponding labels of the training data set to construct the required unlabeled data set.

[0093] The average detection accuracy of the semi-supervised remote sensing target detection method based on pseudo-labeled confidence weighting of the application can improve the mAP@50 index by 5.8% and the mAP@50:95 index by 2% compared with the basic method without using confidence weighted semi-supervised loss.

[0094] It should be noted that although the above embodiments of the application are illustrative, this is not a limitation of the application, therefore the application is not limited to the above specific embodiments. Any other embodiments obtained by those skilled in the art under the inspiration of the application without departing from the principles of the application are considered to be within the protection of the application.

Claims

1. A semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting, characterized in that: The following steps are involved: Step S1: construct a remote sensing target detection dataset including labeled and unlabeled samples; Step S2: Design a confidence weighting mechanism to adjust the weight of the teacher model in training according to the confidence of the pseudo-labeling of unlabeled samples; Step S3: construct a semi-supervised object detection network using the average teacher architecture, integrating the supervision loss and the confidence-weighted semi-supervised loss; Step S4: Complete model evaluation and selection, output the final target detection model, and use the target detection model to complete the remote sensing target detection task.

2. The semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting according to claim 1 is characterized in that: In the step S1, it specifically includes: Step S11: Acquire remote sensing image data of the scene where the target detection task is located and perform preprocessing; Step S12: labeling according to target category; Step S13: Divide the dataset into annotated dataset and unannotated dataset according to whether the dataset has category annotation and position annotation information; Step S14: A portion of the labeled data set is used as verification data, and the remaining labeled images and all unlabeled images constitute the training data set.

3. The semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting according to claim 1 is characterized in that: In step S2, the inverse of the confidence level of the teacher model output result is used as the weight of the unlabeled image classification and regression loss, and no additional weight is applied to the other losses. The overall loss is recalculated, and backpropagation is performed to update the weight parameters of the model; The overall semi-supervised loss for unlabeled images is expressed as: Among them L u , They represent the overall semi-supervised loss function of the unlabeled image, the classification loss function, the regression loss function, and the confidence loss function in the semi-supervised loss, respectively. ω(.) represents the indicator function. If the condition is met, the function value is 1. If the condition is not met, the function value is 0. h,w represents the confidence of the teacher model in predicting the position of feature map h, w, CE(.,.) is the cross entropy loss function, L GIOU is the IOU loss function based on GIOU metric, τ1 and τ2 are preset thresholds; The semi-supervised classification loss for unlabeled images is Represents the classification results of the student model and the teacher model at the feature map h, w position, respectively, is an indicator function that filters the classification prediction results of the current teacher model according to the confidence of the teacher's prediction results Whether to add the calculation of the semi-supervised loss function; if the confidence p h,w If the value is higher than the preset threshold τ2, the indicator function is 1, otherwise it is 0; The semi-supervised regression loss for unlabeled images is Represents the regression results of the predictions of the student model and the teacher model at the feature map h, w positions, respectively. As the indicator function, the regression prediction results of the current teacher model are filtered according to the confidence of the teacher's prediction results Whether to add the calculation of the semi-supervised loss function, if the confidence p h,w is higher than the preset threshold τ2, or The indicator function is 1, otherwise it is 0, where Represents the confidence of the object; The semi-supervised confidence loss for unlabeled images is They represent the confidence results of the student model and the teacher model that the feature map h, w position is the foreground object.

4. The semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting according to claim 1, characterized in that: In the step S3, it specifically includes: Step S31: Input the labeled image into the student model, then compare the prediction result with the labeled information and calculate the classification loss. Regression loss Confidence loss The sum is expressed as: in, Represents the classification results of the student model and the real annotation at the feature map h, w position, Represent the regression results of the student model and the true annotation at the feature map h, w position, respectively, Respectively represent the confidence results of the student model and the true annotation at the feature map h, w position are foreground objects and the corresponding confidence labels; Step S32: After strong and weak data enhancement, the unlabeled images are input into the student model and the teacher model respectively. The prediction results of the teacher model are filtered through double thresholds to obtain pseudo-labels to guide the training process of the student model, which is expressed as: Where ω(·) represents the indicator function. If the condition is met, the function value is 1. If the condition is not met, the function value is 0. τ1 represents the low threshold in the double threshold, and τ2 represents the high threshold in the double threshold. Step S33: Add the supervised loss and the semi-supervised loss as the final overall loss L, which is expressed as: L=L s +λL u ; Where λ is used to adjust the proportion of semi-supervised loss in the total loss; Step S34: All labeled images and unlabeled images in the training data set are input into the algorithm as required, and after calculating the overall loss, back propagation is performed to update the student model parameters. Expressed as: Among them, SGD represents the stochastic gradient descent algorithm, and θ represents the model weight parameter; Step S35: transfer the student model parameters to the teacher model by sliding average, which is expressed as: in, represents the weight parameter of the teacher model at the tth time step, is the weight parameter of the student model at time step t, and α is the smoothing parameter that controls the weight update of the teacher model; Step S36: Repeat steps S31 to S35 until the number of model training rounds reaches a preset number.

5. The semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting according to claim 1, characterized in that: The step S4 specifically includes: Step S41: Evaluate the effect of the model on the target detection task, and select the model with the highest accuracy on the validation set during the training process as the final output model; Step S42: Complete the remote sensing target detection task through the target detection model.

6. The semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting according to claim 5, characterized in that: In step S41, mAP@50 and mAP@50:95 are used to evaluate the effect of the model on the object detection task, and the evaluation score of the model on the validation set is calculated by 0.1*mAP@50+0.9*mAP@50:95; The model with the highest evaluation score is selected as the final semi-supervised object detection model.

7. The semi-supervised remote sensing target detection method based on pseudo-annotation confidence weighting according to claim 5, characterized in that: In step S42, an image to be detected is selected from the test data set, the image to be detected is preprocessed, the processed image is input into the trained semi-supervised target detection model, and the category information and position information of the target in the corresponding image to be detected are output to complete the target detection task.

Images