Semi-supervised target detection method for complex remote sensing scene

Abstract

The invention discloses a semi-supervised target detection method for a complex remote sensing scene, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a student model and a teacher model; respectively training the student model and the teacher model by adopting a semi-supervised learning method based on pseudo labels and a semi-supervised learning method based on dense pseudo labels by adopting the training set to obtain a first loss and a second loss; performing dynamic weighted summation on the first loss and the second loss to obtain third loss; optimizing the student model and the teacher model by adopting third loss; and performing target detection on the remote sensing image based on the trained student model and teacher model. The method can combine the advantages of a semi-supervised learning method based on a pseudo label and a semi-supervised learning method based on a pseudo label, not only effectively deals with a large length-width ratio target in a remote sensing scene, but also reduces false detection in a complex remote sensing background.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a semi-supervised target detection method for complex remote sensing scenes. Background Art

[0002] When performing target detection on remote sensing images, target detection methods based on deep learning have gradually replaced traditional target detection methods based on manual features. Target detection methods based on deep learning rely on a large amount of finely labeled data, so that their models can learn the visual features of targets from a large amount of finely labeled data. Finely labeled data requires a lot of manpower and material resources. Therefore, a semi-supervised training method can be used so that the model can be trained using both labeled data and unlabeled data at the same time, reducing the cost of manual labeling.

[0003] In the related technology, semi-supervised learning methods include semi-supervised learning methods based on pseudo labels (Unbiased teacher) and semi-supervised learning methods based on dense labels (Dense teacher). There are many large aspect ratio targets in remote sensing scenes. Semi-supervised learning methods based on pseudo labels will cause inaccurate learning areas due to unstable pseudo labels. Although the semi-supervised learning method based on dense pseudo labels can better avoid this problem, the flexible supervision strategy it adopts is difficult to identify complex false detections in complex remote sensing scenes. Summary of the invention

[0004] The present disclosure provides a semi-supervised target detection method for complex remote sensing scenes, which can combine the advantages of a semi-supervised learning method based on pseudo labels and a semi-supervised learning method based on pseudo labels, effectively dealing with large aspect ratio targets in remote sensing scenes and reducing false detections in complex remote sensing backgrounds. The technical solution includes at least the following solutions: In a first aspect, a semi-supervised target detection method for complex remote sensing scenes is provided, comprising: obtaining a student model and a teacher model; using a training set to train the student model and the teacher model using a pseudo-label-based semi-supervised learning method and a dense pseudo-label-based semi-supervised learning method, respectively, to obtain a first loss and a second loss, wherein the first loss is the loss obtained after training using the pseudo-label-based semi-supervised learning method, and the second loss is the loss obtained after training using the dense pseudo-label-based semi-supervised learning method, and the training set includes a plurality of remote sensing images for training; dynamically weighted summing the first loss and the second loss to obtain a third loss; using the third loss to optimize the student model and the teacher model; and performing target detection on remote sensing images based on the trained student model and the teacher model.

[0005] Optionally, dynamically weighted summing the first loss and the second loss to obtain a third loss includes: using a dense prediction consistency function to evaluate the credibility of the prediction results output by the teacher model; and based on the credibility, weighted summing the first loss and the second loss to obtain a third loss.

[0006] Optionally, the adopting of a dense prediction consistency function to evaluate the credibility of the prediction result output by the teacher model includes: The degree of credibility is determined using the following formula:

[0007] in, For the stated degree of credibility, is the number of sample points in the first positive sample regression region, For intersection-union ratio calculation, is the prediction result output by the teacher model, Output a prediction result for the student model at the i-th sample point in the first positive sample regression region, where the first positive sample regression region is a positive sample regression region determined by the pseudo-label-based semi-supervised learning method.

[0008] Optionally, based on the credibility, performing weighted summation on the first loss and the second loss to obtain a third loss includes: The third loss is calculated using the following formula:

[0009] in, For the third loss, For the stated degree of credibility, For the first loss, This is the second loss.

[0010] Optionally, the training set is used to train the student model and the teacher model using a semi-supervised learning method based on pseudo labels and a semi-supervised learning method based on dense pseudo labels, respectively, to obtain a first loss and a second loss, including: inputting the first remote sensing image into the student model after a strong data expansion process to obtain a prediction result and a prediction feature map output by the student model, wherein the first remote sensing image is any unlabeled remote sensing image in the training set; inputting the first remote sensing image into the teacher model after a weak data expansion process to obtain a prediction result and a prediction feature map output by the teacher model; based on the prediction result and the prediction feature map output by the student model, and the prediction result and the prediction feature map output by the teacher model, calculating the loss using a first method to obtain a first loss, and the first method is the method for calculating the loss in the semi-supervised learning method based on pseudo labels; based on the prediction result and the prediction feature map output by the student model, and the prediction result and the prediction feature map output by the teacher model, calculating the loss using a second method to obtain a second loss, and the second method is the method for calculating the loss in the semi-supervised learning method based on dense pseudo labels.

[0011] Optionally, the calculating the loss in a first manner based on the prediction result and the prediction feature graph output by the student model, and the prediction result and the prediction feature graph output by the teacher model to obtain a first loss comprises: calculating the loss in the first manner to obtain the first loss according to the following formula:

[0012] in, For the first loss, is the number of sample points in the first positive sample regression region, where the first positive sample regression region is the positive sample regression region determined by the pseudo-label-based semi-supervised learning method, is the total number of sample points in the first remote sensing image, represents the classification loss in target detection, represents the positioning loss in target detection, Outputting a predicted feature map for the student model at the j-th sample point in the first remote sensing image, Outputting a prediction feature map for the student model at the i-th sample point in the first positive sample regression region, The prediction result output by the teacher model; The method of calculating the loss in a second manner based on the prediction result and the prediction feature graph output by the student model and the prediction result and the prediction feature graph output by the teacher model to obtain a second loss comprises: calculating the loss in the second manner to obtain a second loss according to the following formula:

[0013] in, For the second loss, is the number of sample points in the second positive sample regression region, where the second positive sample regression region is the positive sample regression region determined by the semi-supervised learning method based on dense pseudo-labels, Outputting a prediction feature map for the teacher model at the j-th sample point in the first remote sensing image, Outputting a prediction feature map for the student model at the kth sample point in the second positive sample regression region, Output a prediction feature map for the teacher model at the kth sample point in the second positive sample regression region.

[0014] In the second aspect, a semi-supervised target detection device for complex remote sensing scenes is also provided, including: an acquisition module for acquiring a student model and a teacher model; a training module for using a training set to train the student model and the teacher model using a pseudo-label-based semi-supervised learning method and a dense pseudo-label-based semi-supervised learning method, respectively, to obtain a first loss and a second loss, wherein the first loss is the loss obtained after training using the pseudo-label-based semi-supervised learning method, and the second loss is the loss obtained after training using the dense pseudo-label-based semi-supervised learning method, and the training set includes multiple remote sensing images for training; a dynamic weighting module for dynamically weighted summing the first loss and the second loss to obtain a third loss; an optimization module for optimizing the student model and the teacher model using the third loss; and a target detection module for performing target detection on remote sensing images based on the trained student model and the teacher model.

[0015] Optionally, the dynamic weighting module is also used to use a dense prediction consistency function to evaluate the credibility of the prediction results output by the teacher model; based on the credibility, the first loss and the second loss are weighted and summed to obtain a third loss.

[0016] Optionally, the dynamic weighting module is further used to determine the credibility using the following formula:

[0017] in, For the stated degree of credibility, is the number of sample points in the first positive sample regression region, For intersection-union ratio calculation, is the prediction result output by the teacher model, Output a prediction result for the student model at the i-th sample point in the first positive sample regression region, where the first positive sample regression region is a positive sample regression region determined by the pseudo-label-based semi-supervised learning method.

[0018] Optionally, the dynamic weighting module is further used to calculate the third loss using the following formula:

[0019] in, For the third loss, For the first loss, This is the second loss.

[0020] Optionally, the training module is also used to input the first remote sensing image into the student model after a strong data expansion process to obtain the prediction result and prediction feature map output by the student model, wherein the first remote sensing image is any unlabeled remote sensing image in the training set; input the first remote sensing image into the teacher model after a weak data expansion process to obtain the prediction result and prediction feature map output by the teacher model; based on the prediction result and prediction feature map output by the student model, as well as the prediction result and prediction feature map output by the teacher model, a first method is used to calculate the loss to obtain a first loss, and the first method is the method for calculating the loss in the semi-supervised learning method based on pseudo-labels; based on the prediction result and prediction feature map output by the student model, as well as the prediction result and prediction feature map output by the teacher model, a second method is used to calculate the loss to obtain a second loss, and the second method is the method for calculating the loss in the semi-supervised learning method based on dense pseudo-labels.

[0021] Optionally, the training module is further used to calculate the loss in the first manner according to the following formula to obtain a first loss:

[0022] in, For the first loss, is the number of sample points in the first positive sample regression region, where the first positive sample regression region is the positive sample regression region determined by the pseudo-label-based semi-supervised learning method, is the total number of sample points in the first remote sensing image, represents the classification loss in target detection, represents the positioning loss in target detection, Outputting a predicted feature map for the student model at the j-th sample point in the first remote sensing image, Outputting a prediction feature map for the student model at the i-th sample point in the first positive sample regression region, The prediction result output by the teacher model; And the second loss is obtained by calculating the loss in the second manner according to the following formula:

[0023] in, For the second loss, is the number of sample points in the second positive sample regression region, where the second positive sample regression region is the positive sample regression region determined by the semi-supervised learning method based on dense pseudo-labels, Outputting a prediction feature map for the teacher model at the j-th sample point in the first remote sensing image, Outputting a prediction feature map for the student model at the kth sample point in the second positive sample regression region, Output a prediction feature map for the teacher model at the kth sample point in the second positive sample regression region.

[0024] In a third aspect, a computer device is also provided, comprising: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor, thereby executing the semi-supervised target detection method for complex remote sensing scenes described in the above embodiment.

[0025] In a fourth aspect, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to execute the semi-supervised target detection method for complex remote sensing scenes described in the above embodiment.

[0026] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0027] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least: In the disclosed embodiment, the first loss is the loss obtained after training using a semi-supervised learning method based on pseudo-labels, and the second loss is the loss obtained after training using a semi-supervised learning method based on dense pseudo-labels. When the first loss and the second loss are dynamically weighted and summed to obtain the third loss, it is equivalent to that the third loss is also dynamic. When the third loss is used to optimize the model, the model will continuously adjust the training bias according to the proportion of the first loss and the second loss in the third loss. When the proportion of the first loss in the third loss is large, it is equivalent to that the model tends to be trained using a semi-supervised learning method based on pseudo-labels; when the proportion of the second loss in the third loss is large, it is equivalent to that the model tends to be trained using a semi-supervised learning method based on dense pseudo-labels. In this way, the model can be dynamically switched between the two semi-supervised learning methods, and the advantages of the two semi-supervised learning methods can be dynamically combined. The final trained model can adapt to remote sensing targets with different aspect ratios and complex remote sensing backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A flowchart of a semi-supervised target detection method for complex remote sensing scenes provided by an exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart of a semi-supervised target detection method for complex remote sensing scenes provided by another exemplary embodiment of the present disclosure is shown; Figure 3 is a schematic diagram of an algorithm flow of a semi-supervised target detection method for complex remote sensing scenes in an embodiment of the present disclosure; Figure 4 A schematic diagram of the structure of a semi-supervised target detection device for complex remote sensing scenes provided by an exemplary embodiment of the present disclosure is shown; Figure 5 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Unless otherwise defined, the technical or scientific terms used herein shall have the usual meanings understood by persons of ordinary skill in the field to which the present disclosure belongs. The words "first", "second", "third" and similar words used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" and similar words mean that the elements or objects appearing before "include" or "comprises" cover the elements or objects listed after "include" or "comprises" and their equivalents, and do not exclude other elements or objects.

[0031] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0032] To facilitate understanding of the embodiments of the present disclosure, a semi-supervised learning method based on pseudo labels and a semi-supervised learning method based on dense pseudo labels are first described below.

[0033] Semi-supervised learning methods based on pseudo-labels mainly use labeled data to train a reliable teacher model, and then use the teacher model to predict unlabeled data. After a series of processing, high-quality prediction results, that is, pseudo-labels, are obtained, and pseudo-labels are used to guide the training of the model on unlabeled data. However, pseudo-labels are generated by the teacher model rather than manually labeled, so there may be errors in the pseudo-labels, that is, the quality of the pseudo-labels is not stable enough. This type of semi-supervised learning method trusts pseudo-labels more. When the quality of pseudo-labels is not stable enough, this type of semi-supervised learning method is prone to cause the model to be disturbed by poor-quality pseudo-labels.

[0034] The semi-supervised learning method based on dense pseudo-labels is improved on the semi-supervised learning method based on pseudo-labels. It adopts a form of flexible supervision, which can avoid the interference caused by the unstable pseudo-labels in the semi-supervised learning based on pseudo-labels. However, flexible supervision is prone to false detection in complex remote sensing scenes.

[0035] There are many implementation methods of the semi-supervised learning method based on pseudo labels and the semi-supervised learning method based on dense pseudo labels in the related technologies, and the detailed description is omitted here.

[0036] Figure 1 A flowchart of a semi-supervised target detection method for complex remote sensing scenes provided by an exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 1 , the method comprising: In step 101, a student model and a teacher model are obtained.

[0037] Optionally, the following three steps are used to obtain the student model and the teacher model.

[0038] The first step is to build an initial model and copy the initial model into two identical models, one of which is trained according to the student model and the other is trained according to the teacher model.

[0039] Exemplarily, the initial model is a deep learning model, which is used for target detection.

[0040] The model trained according to the student model is denoted as model S, and the model trained according to the teacher model is denoted as model T.

[0041] In the second step, in the first m generations, the model S is trained using the labeled remote sensing images in the training set, and the model T does not participate in the training.

[0042] The value of m is an empirical value, which is not limited in the embodiments of the present disclosure.

[0043] The third step is to completely copy the parameters in model S to model T after the first m generations of training are completed.

[0044] In this way, model S is the student model and model T is the teacher model, so that the student model and the teacher model can be obtained.

[0045] In step 102, the student model and the teacher model are trained using the training set using a pseudo-label-based semi-supervised learning method and a dense pseudo-label-based semi-supervised learning method, respectively, to obtain a first loss and a second loss.

[0046] The first loss is the loss obtained after training using a semi-supervised learning method based on pseudo labels, and the second loss is the loss obtained after training using a semi-supervised learning method based on dense pseudo labels. The training set includes multiple remote sensing images for training, such as labeled remote sensing images and unlabeled remote sensing images.

[0047] In step 103, a dynamic weighted sum is performed on the first loss and the second loss to obtain a third loss.

[0048] In step 104, the student model and the teacher model are optimized using a third loss.

[0049] In step 105, target detection is performed on the remote sensing image based on the trained student model and teacher model.

[0050] In the disclosed embodiment, the first loss is the loss obtained after training using a semi-supervised learning method based on pseudo-labels, and the second loss is the loss obtained after training using a semi-supervised learning method based on dense pseudo-labels. When the first loss and the second loss are dynamically weighted and summed to obtain the third loss, it is equivalent to that the third loss is also dynamic. When the third loss is used to optimize the model, the model will continuously adjust the training bias according to the proportion of the first loss and the second loss in the third loss. When the proportion of the first loss in the third loss is large, it is equivalent to that the model tends to be trained using a semi-supervised learning method based on pseudo-labels; when the proportion of the second loss in the third loss is large, it is equivalent to that the model tends to be trained using a semi-supervised learning method based on dense pseudo-labels. In this way, the model can be dynamically switched between the two semi-supervised learning methods, and the advantages of the two semi-supervised learning methods can be dynamically combined. The final trained model can adapt to remote sensing targets with different aspect ratios and complex remote sensing backgrounds.

[0051] Figure 2 A flowchart of a semi-supervised target detection method for complex remote sensing scenes provided by another exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 2 , the method comprising: In step 201, a student model and a teacher model are obtained.

[0052] The relevant contents of step 201 refer to the aforementioned step 101, and the detailed description is omitted here.

[0053] In step 202, the student model and the teacher model are trained using a training set using a pseudo-label-based semi-supervised learning method and a dense pseudo-label-based semi-supervised learning method, respectively, to obtain a first loss and a second loss.

[0054] The first loss is the loss obtained after training using a semi-supervised learning method based on pseudo labels, and the second loss is the loss obtained after training using a semi-supervised learning method based on dense pseudo labels. The training set includes multiple remote sensing images for training.

[0055] During training, inputting all remote sensing images in the training set into the model completes one generation of training. When the second remote sensing image in the training set is input into the model, the training method for the student model and the teacher model is exactly the same as the second step in the aforementioned step 101, and the first loss and the second loss cannot be obtained at this time. Here, the second remote sensing image is any labeled remote sensing image in the training set.

[0056] Figure 3 is a schematic diagram of the algorithm flow of the semi-supervised target detection method for complex remote sensing scenes in the embodiment of the present disclosure. Figure 3 Steps 202 to 205 are described.

[0057] When the first remote sensing image in the training set is input into the model, the following steps ad are used to implement step 202. Here, the second remote sensing image is any unlabeled remote sensing image in the training set.

[0058] Step a, inputting the first remote sensing image into the student model after a strong data expansion process, and obtaining the prediction result and prediction feature map output by the student model.

[0059] Strong data augmentation, also known as strong data enhancement, is a process that transforms the original data to a large extent, usually changing some basic features of the original data. Large-scale transformations can include rotation, flipping, shearing, adding noise, mixing different data, etc. The purpose of strong data enhancement is to increase the diversity of training data and improve the robustness of the model without changing the data category.

[0060] The prediction results output by the student model can be used To represent, the predicted feature map output by the student model can be used To express.

[0061] Step b: input the first remote sensing image into the teacher model after the weak data expansion process to obtain the prediction result and prediction feature map output by the teacher model.

[0062] Weak data augmentation, also known as weak data enhancement, is a process that transforms the original data to a small degree, usually without changing the basic characteristics of the original data. Smaller transformations can include scaling, translation, changing color space, etc. The purpose of weak data enhancement is to improve model performance by making smaller perturbations to the data so that the model can learn more local features.

[0063] The prediction results output by the teacher model can be used To represent, the predicted feature map output by the teacher model can be used To express.

[0064] Step c, based on the prediction results and prediction feature graph output by the student model, as well as the prediction results and prediction feature graph output by the teacher model, the loss is calculated using the first method to obtain a first loss.

[0065] The first method is to calculate the loss in the semi-supervised learning method based on pseudo labels.

[0066] Optionally, the loss is calculated in a first manner according to formula (1) to obtain a first loss.

[0067] (1) In formula (1), For the first loss, is the number of sample points in the first positive sample regression region, and the first positive sample regression region is a positive sample regression region determined by a semi-supervised learning method based on pseudo labels. is the total number of sample points in the first remote sensing image, represents the classification loss in target detection, represents the positioning loss in target detection, Output the predicted feature map for the student model at the jth sample point in the first remote sensing image. Output the predicted feature map for the student model at the i-th sample point in the first positive sample regression region. The prediction result output by the teacher model.

[0068] Among them, i and j are integers, and the value range of i is 1 to , the value range of j is 1 to .

[0069] Exemplarily, in a pseudo-label-based semi-supervised learning method, positive and negative samples are defined for the predicted feature map output by the student model according to the prediction results output by the teacher model, thereby dividing the positive sample learning area and the negative sample learning area. The positive sample learning area is the first positive sample regression area.

[0070] The classification loss in target detection is, for example, cross entropy loss, and the positioning loss in target detection is, for example, mean absolute error or mean square error, etc. There are many implementation methods of classification loss and positioning loss in target detection in related technologies, and detailed description is omitted here.

[0071] Step d, based on the prediction results and prediction feature graph output by the student model, as well as the prediction results and prediction feature graph output by the teacher model, the loss is calculated using the second method to obtain a second loss.

[0072] The second method is to calculate the loss in a semi-supervised learning method based on dense pseudo-labels.

[0073] Optionally, the loss is calculated in a second manner according to formula (2) to obtain a second loss.

[0074] (2) In formula (2), For the second loss, is the number of sample points in the second positive sample regression region, which is the positive sample regression region determined by the semi-supervised learning method based on dense pseudo-labels. Output the predicted feature map for the teacher model at the jth sample point in the first remote sensing image, Output the predicted feature map for the student model at the kth sample point in the second positive sample regression region. Output the predicted feature map for the teacher model at the kth sample point in the second positive sample regression region. Where k is an integer, and the value range of k is 1 to The meanings of other parameters in formula (2) are the same as those in formula (1), and their detailed description is omitted here.

[0075] Exemplarily, in a semi-supervised learning method based on dense pseudo-labels, point-to-point regression supervision is performed on the predicted feature map output by the student model according to the predicted feature map output by the teacher model, and the total sample points in the predicted feature map output by the teacher model are sorted and sampled, and the sample points with prediction scores in the top k% are taken as the positive sample learning area (in the remote sensing field, k is usually 1, that is, 1%), and the positive sample learning area is the second positive sample regression area.

[0076] From the analysis of the first loss and the second loss, it can be seen that in the process of calculating the first loss, there is a clear definition of the positive sample regression area and the negative sample regression area, but this definition is too dependent on the pseudo-label (that is, the prediction result output by the teacher model). Since the pseudo-label may be inaccurate, the definition of the first positive sample learning area may be incorrect, affecting the semi-supervised learning process.

[0077] In the process of calculating the second loss, the predicted feature map (dense pseudo-label) output by the teacher model is used to perform point-to-point regression on the predicted feature map output by the student model. This is a flexible supervision method that is relatively fuzzy and does not rely on pseudo-labels. It can avoid problems caused by inaccurate pseudo-labels in the process of calculating the first loss. However, this fuzzy point-to-point regression method based on dense pseudo-labels also blurs the definition of positive and negative samples, resulting in insufficient negative sample learning of the model and false detection.

[0078] In step 203, a dense prediction consistency function is used to evaluate the credibility of the prediction results output by the teacher model.

[0079] The first positive sample regression region can be determined based on the prediction results output by the teacher model. For each sample point in the first positive sample regression region, the corresponding sample point can be found in the prediction results output by the corresponding student model. The dense prediction consistency function can be used to judge the consistency between the prediction results output by the student model and the prediction results output by the teacher model. The higher the consistency, the more credible the prediction results output by the teacher model are.

[0080] Optionally, the credibility is determined using formula (3): (3) In formula (3), For credibility, For intersection-union ratio calculation, The prediction result output by the student model is the i-th sample point in the first positive sample regression region. The meanings of other parameters in formula (3) are the same as those in formula (1), and their detailed description is omitted here.

[0081] There are many methods for implementing the intersection-over-union ratio calculation in the related art, and the detailed description is omitted here.

[0082] If the credibility is high, that is, the prediction results output by the student model are highly consistent with the prediction results output by the teacher model, then the credibility of the prediction results output by the teacher model (that is, the pseudo-label) is very high, and the semi-supervised learning method based on pseudo-labels can be used for training. If the credibility is low, that is, the consistency between the prediction results output by the student model and the prediction results output by the teacher model is low, then the credibility of the prediction results output by the teacher model (that is, the pseudo-label) is low, and the semi-supervised learning method based on dense pseudo-labels can be used for training to avoid the impact of pseudo-label errors on the semi-supervised learning process.

[0083] Moreover, in the process of semi-supervised learning and training of the model, as the model continues to learn, the credibility changes dynamically (gradually increases), which is equivalent to the subsequent process of weighted summation of the first loss and the second loss based on the credibility, which is a dynamic weighted process.

[0084] In step 204, based on the credibility, the first loss and the second loss are weightedly summed to obtain a third loss.

[0085] Optionally, formula (4) is used to calculate the third loss.

[0086] (4) In formula (4), is the third loss. The meanings of other parameters in formula (4) are the same as those in formula (1), formula (2) and formula (3), and their detailed description is omitted here.

[0087] It can be seen from formula (4) that when the credibility changes, the third loss will also change. When the credibility is 0, the third loss is equal to the second loss, which is equivalent to training the model only by using the semi-supervised learning method based on dense pseudo-labels; when the credibility is 1, the third loss is equal to the first loss, which is equivalent to training the model only by using the semi-supervised learning method based on pseudo-labels; when the credibility is between 1 and 0, it is in a chaotic state, which is equivalent to combining the semi-supervised learning method based on pseudo-labels and the semi-supervised learning method based on dense pseudo-labels to train the model.

[0088] Usually, in the early stage of model training, the teacher model has not been trained more, and the credibility of pseudo labels is low. At this time, semi-supervised learning methods based on dense pseudo labels are mainly used for learning, and flexible supervision is used to transition the early stage of training with incomplete supervision information. In the late stage of model training, the teacher model has been fully trained, and the credibility of pseudo labels is high. At this time, semi-supervised learning methods based on pseudo labels are mainly used for training, so as to strengthen the training of negative samples and reduce the interference of complex remote sensing background.

[0089] In step 205, the student model and the teacher model are optimized using the third loss.

[0090] Optionally, step 205 includes: using a back propagation algorithm to optimize the parameters of the student model based on the third loss; after the parameters of the student model are optimized, using an EMA (Exponential Moving Average) algorithm to optimize the parameters of the teacher model.

[0091] There are many implementation methods of the back propagation algorithm and the EMA algorithm in the related technologies, and the detailed description is omitted here.

[0092] By repeatedly executing steps 202-205, the student model and the teacher model can be iteratively trained for multiple times until the training is completed. The trained student model and the teacher model can be used to detect targets in remote sensing images.

[0093] In step 206, target detection is performed on the remote sensing image based on the trained student model and teacher model.

[0094] In the disclosed embodiment, the current training status of the model can be estimated through the dense prediction consistency evaluation function. A dynamically adjustable weight is established through formula (4), so that the model can continuously switch between the two training methods: the semi-supervised learning method based on pseudo labels and the semi-supervised learning method based on dense pseudo labels. In this way, the final model can not only effectively deal with large aspect ratio targets in remote sensing scenes, but also reduce false detections in complex remote sensing backgrounds.

[0095] The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

[0096] Figure 4 A schematic diagram of the structure of a semi-supervised target detection device for complex remote sensing scenes provided by an exemplary embodiment of the present disclosure is shown. Figure 4 The semi-supervised target detection device 400 for complex remote sensing scenes includes: an acquisition module 401, a training module 402, a dynamic weighting module 403, an optimization module 404 and a target detection module 405.

[0097] The acquisition module 401 is used to acquire the student model and the teacher model.

[0098] The training module 402 is used to use the training set to train the student model and the teacher model respectively using a semi-supervised learning method based on pseudo labels and a semi-supervised learning method based on dense pseudo labels to obtain a first loss and a second loss. The first loss is the loss obtained after training using the semi-supervised learning method based on pseudo labels, and the second loss is the loss obtained after training using the semi-supervised learning method based on dense pseudo labels. The training set includes multiple remote sensing images for training.

[0099] The dynamic weighting module 403 is used to perform dynamic weighted summation on the first loss and the second loss to obtain a third loss.

[0100] The optimization module 404 is used to optimize the student model and the teacher model using the third loss.

[0101] The target detection module 405 is used to perform target detection on the remote sensing image based on the trained student model and teacher model.

[0102] Optionally, the dynamic weighting module 403 is also used to use a dense prediction consistency function to evaluate the credibility of the prediction results output by the teacher model; based on the credibility, the first loss and the second loss are weighted and summed to obtain a third loss.

[0103] Optionally, the dynamic weighting module 403 is further configured to determine the credibility using the following formula:

[0104] in, For credibility, is the number of sample points in the first positive sample regression region, For intersection-union ratio calculation, is the prediction result output by the teacher model, Output the prediction result for the student model at the i-th sample point in the first positive sample regression region, where the first positive sample regression region is the positive sample regression region determined by the semi-supervised learning method based on pseudo labels.

[0105] Optionally, the dynamic weighting module 403 is further configured to calculate the third loss using the following formula:

[0106] in, For the third loss, For the first loss, For the second loss.

[0107] Optionally, the training module 402 is also used to input the first remote sensing image into the student model after a strong data expansion process to obtain the prediction result and prediction feature map output by the student model, where the first remote sensing image is any unlabeled remote sensing image in the training set; input the first remote sensing image into the teacher model after a weak data expansion process to obtain the prediction result and prediction feature map output by the teacher model; based on the prediction result and prediction feature map output by the student model, as well as the prediction result and prediction feature map output by the teacher model, a first method is used to calculate the loss to obtain a first loss, and the first method is a method for calculating the loss in a semi-supervised learning method based on pseudo-labels; based on the prediction result and prediction feature map output by the student model, as well as the prediction result and prediction feature map output by the teacher model, a second method is used to calculate the loss to obtain a second loss, and the second method is a method for calculating the loss in a semi-supervised learning method based on dense pseudo-labels.

[0108] Optionally, the training module 402 is further configured to calculate the loss in the first manner according to the following formula to obtain a first loss:

[0109] in, For the first loss, is the number of sample points in the first positive sample regression region, which is the positive sample regression region determined by the semi-supervised learning method based on pseudo labels. is the total number of sample points in the first remote sensing image, represents the classification loss in target detection, represents the positioning loss in target detection, Output the predicted feature map for the student model at the jth sample point in the first remote sensing image. Output the predicted feature map for the student model at the i-th sample point in the first positive sample regression region. The prediction results output by the teacher model; And the second loss is obtained by calculating the loss in the second way according to the following formula:

[0110] in, For the second loss, is the number of sample points in the second positive sample regression region, which is the positive sample regression region determined by the semi-supervised learning method based on dense pseudo-labels. Output the predicted feature map for the teacher model at the jth sample point in the first remote sensing image, Output the predicted feature map for the student model at the kth sample point in the second positive sample regression region. Output the predicted feature map for the teacher model at the kth sample point in the second positive sample regression region.

[0111] It should be noted that: when the semi-supervised target detection device for complex remote sensing scenes provided in the above embodiment performs target detection, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the semi-supervised target detection device for complex remote sensing scenes provided in the above embodiment and the embodiment of the semi-supervised target detection method for complex remote sensing scenes belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0112] The division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present disclosure may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0113] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, mobile phone, or communication device, etc.) or a processor (processor) to execute all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0114] Figure 5 Schematic diagram of the structure of the computer device provided by the embodiment of the present disclosure. Figure 5 As shown, the computer device 500 includes: a processor 501 and a memory 502 .

[0115] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0116] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement the semi-supervised target detection method for complex remote sensing scenes provided in the embodiments of the present disclosure.

[0117] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the computer device 500, and the computer device 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0118] The embodiments of the present disclosure also provide a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer device, the computer device is enabled to execute the semi-supervised target detection method for complex remote sensing scenes provided in the embodiments of the present disclosure.

[0119] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the semi-supervised target detection method for complex remote sensing scenes provided in the embodiments of the present disclosure.

[0120] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A semi-supervised target detection method for complex remote sensing scenes, characterized in that: The method comprises: Get the student model and the teacher model; The student model and the teacher model are trained using a training set using a pseudo-label-based semi-supervised learning method and a dense pseudo-label-based semi-supervised learning method, respectively, to obtain a first loss and a second loss, wherein the first loss is a loss obtained after training using the pseudo-label-based semi-supervised learning method, and the second loss is a loss obtained after training using the dense pseudo-label-based semi-supervised learning method, wherein the training set includes a plurality of remote sensing images for training; Performing a dynamic weighted summation on the first loss and the second loss to obtain a third loss; Adopting the third loss to optimize the student model and the teacher model; Based on the trained student model and the teacher model, target detection is performed on the remote sensing image.

2. The method according to claim 1, characterized in that: The dynamically weighted summing of the first loss and the second loss to obtain a third loss includes: Using a dense prediction consistency function to evaluate the credibility of the prediction results output by the teacher model; Based on the credibility, a weighted sum is performed on the first loss and the second loss to obtain a third loss.

3. The method according to claim 2, characterized in that The dense prediction consistency function is used to evaluate the credibility of the prediction results output by the teacher model, including: The degree of credibility is determined using the following formula: in, For the stated degree of credibility, is the number of sample points in the first positive sample regression region, For intersection-union ratio calculation, is the prediction result output by the teacher model, Output a prediction result for the student model at the i-th sample point in the first positive sample regression region, where the first positive sample regression region is a positive sample regression region determined by the pseudo-label-based semi-supervised learning method.

4. The method according to claim 3, characterized in that The step of performing weighted summation of the first loss and the second loss based on the credibility to obtain a third loss includes: The third loss is calculated using the following formula: in, For the third loss, For the first loss, This is the second loss.

5. The method according to any one of claims 1 to 3, characterized in that: The student model and the teacher model are trained using a training set using a semi-supervised learning method based on pseudo labels and a semi-supervised learning method based on dense pseudo labels, respectively, to obtain a first loss and a second loss, including: Inputting a first remote sensing image into the student model after a strong data expansion process to obtain a prediction result and a prediction feature map output by the student model, wherein the first remote sensing image is any unlabeled remote sensing image in the training set; Inputting the first remote sensing image into the teacher model after a weak data expansion process to obtain a prediction result and a prediction feature map output by the teacher model; Based on the prediction results and prediction feature graphs output by the student model, and the prediction results and prediction feature graphs output by the teacher model, a first method is used to calculate the loss to obtain a first loss, where the first method is a method for calculating the loss in the pseudo-label-based semi-supervised learning method; Based on the prediction results and prediction feature graphs output by the student model, as well as the prediction results and prediction feature graphs output by the teacher model, a second method is used to calculate the loss to obtain a second loss. The second method is a method for calculating the loss in the semi-supervised learning method based on dense pseudo-labels.

6. The method according to claim 5, characterized in that The prediction result and prediction feature graph output by the student model, as well as the prediction result and prediction feature graph output by the teacher model, are used to calculate the loss in a first manner to obtain a first loss, including: The first loss is obtained by calculating the loss in the first manner according to the following formula: in, For the first loss, is the number of sample points in the first positive sample regression region, where the first positive sample regression region is the positive sample regression region determined by the pseudo-label-based semi-supervised learning method, is the total number of sample points in the first remote sensing image, represents the classification loss in target detection, represents the positioning loss in target detection, Outputting a predicted feature map for the student model at the j-th sample point in the first remote sensing image, Outputting a prediction feature map for the student model at the i-th sample point in the first positive sample regression region, The prediction result output by the teacher model; The prediction result and prediction feature graph output by the student model, and the prediction result and prediction feature graph output by the teacher model, are used to calculate the loss in a second manner to obtain a second loss, including: The second method is used to calculate the loss according to the following formula to obtain the second loss: in, For the second loss, is the number of sample points in the second positive sample regression region, where the second positive sample regression region is the positive sample regression region determined by the semi-supervised learning method based on dense pseudo-labels, Outputting a prediction feature map for the teacher model at the j-th sample point in the first remote sensing image, Outputting a prediction feature map for the student model at the kth sample point in the second positive sample regression region, Output a prediction feature map for the teacher model at the kth sample point in the second positive sample regression region.

7. A semi-supervised target detection device for complex remote sensing scenes, characterized in that: The device comprises: The acquisition module is used to obtain the student model and the teacher model; A training module, used for using a training set to train the student model and the teacher model respectively using a semi-supervised learning method based on pseudo labels and a semi-supervised learning method based on dense pseudo labels, to obtain a first loss and a second loss, wherein the first loss is the loss obtained after training using the semi-supervised learning method based on pseudo labels, and the second loss is the loss obtained after training using the semi-supervised learning method based on dense pseudo labels, wherein the training set includes a plurality of remote sensing images for training; A dynamic weighting module, used for dynamically weighting and summing the first loss and the second loss to obtain a third loss; An optimization module, configured to optimize the student model and the teacher model using the third loss; The target detection module is used to perform target detection on remote sensing images based on the trained student model and the teacher model.

8. A computer device, characterized in that: The computer device comprises: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images