A semi-supervised remote sensing target detection method and device based on detector decoupling

By adopting a detector decoupling method in semi-supervised remote sensing object detection, combining labelless and labeled data, and using feature extraction and classification detectors, the problem of poor model performance in remote sensing data is solved, and detection accuracy and adaptability are improved .

CN118570454BActive Publication Date: 2024-11-01RESEARCH INSTITUTE OF TSINGHUA UNIVERSITY IN SHENZHEN
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411053612.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-01
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

The existing semi-supervised object detection technology in complex remote sensing data has poor model performance due to the small number of categories, small in-class gaps and large inter-class gaps, especially the pseudo-performance of the target aspect ratio and rotation characteristics in remote sensing images. The label quality is reduced, resulting in accumulated errors and affecting model performance.

Method used

The semi-supervised remote sensing object detection method based on detector decoupling is adopted. By combining label-free samples and labeled samples remote sensing images, a feature extractor, a region generation module, a region pooling module, a regression detector and a student classification detector are used. Train the model through feature enhancement and exponential moving average to avoid degradation of pseudo-label quality and improve model performance.

Benefits of technology

It improves the detection accuracy of the object detection model, reduces error accumulation, adapts to the characteristics of remote sensing images, and improves the performance of the model under complex data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570454B_ABST
    Figure CN118570454B_ABST
Patent Text Reader

Abstract

The present invention provides a semi-supervised remote sensing target detection method and device based on detector decoupling, which relates to the technical field of remote sensing image processing. The method comprises: inputting the acquired remote sensing image to be detected into a target detection model to obtain the detection result output by the target detection model; the target detection model is obtained by training based on the weights corresponding to the aspect ratio of the real detection frame of at least one sample target in the unlabeled sample remote sensing image, the labeled sample remote sensing image, and the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected. When training the target detection model, the weights corresponding to the aspect ratio of the real detection frame of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and the labeled sample remote sensing image are combined, so that the performance of the target detection model is better, thereby improving the accuracy of target detection in the remote sensing image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a semi-supervised remote sensing target detection method and device based on detector decoupling. Background Art

[0002] With the increasing use of high-resolution satellites, it has become easier to obtain original remote sensing images. However, manual labeling of remote sensing images is not only time-consuming, labor-intensive, and costly, especially for special objects such as remote sensing images, which have many, dense, and large size variations. Therefore, Semi-supervised Object Detection (SSOD) technology uses a small amount of labeled data and a large amount of unlabeled data to optimize the model, which has become a reliable direction.

[0003] The effect of existing semi-supervised target detection depends largely on the quality of pseudo-labels. The errors caused by low-quality pseudo-labels will be gradually amplified during the self-training process. This is a systematic error caused by the general semi-supervised target detection framework. However, in complex remote sensing data, due to the small number of categories in the remote sensing target detection dataset, small intra-class gaps, large inter-class gaps, and a large number of targets on a single image, a small amount of data is sufficient to fine-tune the backbone network in the better teacher model and student model, while the detectors in the teacher model and student model may not be fully trained. The errors caused by the detectors in the unsupervised training process are back-propagated to the backbone network, resulting in a decrease in the performance of the backbone network, which leads to poor performance of the model. Summary of the invention

[0004] The present invention provides a semi-supervised remote sensing target detection method and device based on detector decoupling, which are used to solve the problem of poor performance of the model.

[0005] The present invention provides a semi-supervised remote sensing target detection method based on detector decoupling, comprising:

[0006] Acquire a remote sensing image to be detected;

[0007] The remote sensing image to be detected is input into a target detection model to obtain a detection result output by the target detection model; the target detection model is trained based on weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target.

[0008] According to a semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention, the target detection model includes a feature extractor, a region generation module, a region pooling module, a regression detector and a student classification detector;

[0009] Inputting the remote sensing image to be detected into the target detection model to obtain the detection result output by the target detection model includes:

[0010] Inputting the remote sensing image to be detected into the feature extractor to obtain a feature image output by the feature extractor;

[0011] Inputting the feature image into the region generation module to obtain a region image corresponding to each target in the feature image output by the region generation module;

[0012] Inputting each of the regional images into the regional pooling module to obtain the regional features corresponding to each of the targets output by the regional pooling module; the dimensions of the regional features corresponding to each of the targets are the same;

[0013] Inputting the regional features corresponding to each of the targets into the regression detector to obtain the detection frame coordinates of each of the targets output by the regression detector;

[0014] The regional features corresponding to each of the targets are input into the student classification detector to obtain the classification type of each of the targets output by the student classification detector.

[0015] According to a semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention, the target detection model is trained based on the following steps:

[0016] Acquire a training data set; the training data set includes a plurality of unlabeled sample remote sensing images and a plurality of labeled sample remote sensing images;

[0017] The unlabeled sample remote sensing images and the labeled sample remote sensing images are used to train the initial detection model to obtain the target detection model.

[0018] According to a semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention, the initial detection model is trained using each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images to obtain the target detection model, including:

[0019] Inputting each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images into an initial feature extractor respectively, and obtaining a first sample feature image corresponding to each of the unlabeled sample remote sensing images and a second sample feature image corresponding to each of the labeled sample remote sensing images output by the initial feature extractor;

[0020] Inputting each of the first sample feature images and each of the second sample feature images into an initial region generation module respectively, and obtaining a first sample region image corresponding to each of the sample targets in each of the first sample feature images and a second sample region image corresponding to each of the sample targets in each of the second sample feature images output by the initial region generation module;

[0021] Inputting each of the first sample region images and each of the second sample region images into an initial region pooling module respectively, obtaining first sample region features corresponding to each of the first sample region images and second sample region features corresponding to each of the second sample region images output by the initial region pooling module; the dimensions of each of the first sample region features are the same, and the dimensions of each of the second sample region features are the same;

[0022] The object detection model is determined based on each of the first sample region features and each of the second sample region features.

[0023] According to a semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention, the target detection model is determined based on each of the first sample region features and each of the second sample region features, including:

[0024] Using at least one feature enhancement method, perform feature enhancement on each of the first sample region features to obtain a plurality of enhanced first sample region features;

[0025] Inputting each of the enhanced first sample region features and each of the second sample region features into an initial regression detector, obtaining first sample detection frame coordinates of each of the sample targets corresponding to each of the enhanced first sample region features and second sample detection frame coordinates of each of the sample targets corresponding to each of the second sample region features output by the initial regression detector;

[0026] Inputting each of the enhanced first sample region features and each of the second sample region features into an initial student classification detector, obtaining a first classification score of each of the sample targets corresponding to each of the enhanced first sample region features and a second classification score of each of the sample targets corresponding to each of the second sample region features output by the initial student classification detector;

[0027] Inputting each of the second sample region features into an initial teacher classification detector to obtain a pseudo label of each of the sample targets output by the initial teacher classification detector; the structure of the initial teacher classification detector is the same as that of the initial student classification detector, and the parameters of the initial teacher classification detector are updated based on the exponential moving average of the parameters of the initial student classification detector;

[0028] Determine a total loss function value based on the first sample detection frame coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection frame of each sample target, the second sample detection frame coordinates of each sample target, the real detection frame coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target;

[0029] Based on the total loss function value, the target detection model is determined.

[0030] According to a semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention, the total loss function value is determined based on the first sample detection frame coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection frame of each sample target, the second sample detection frame coordinates of each sample target, the real detection frame coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target, including:

[0031] Determine an unsupervised regression loss value based on the weights corresponding to the first sample detection frame coordinates of each of the sample targets and the aspect ratios of the real detection frames of each of the sample targets;

[0032] Determining a supervised regression loss value based on the second sample detection frame coordinates of each of the sample targets and the real detection frame coordinates of each of the sample targets;

[0033] Determining an unsupervised classification loss value based on the first classification score of each of the sample targets and the pseudo label of each of the sample targets;

[0034] Determine a supervised classification loss value based on the second classification score of each of the sample objects corresponding to each of the second sample region features and the real detection frame coordinates of each of the sample objects;

[0035] Adding the unsupervised regression loss value and the unsupervised classification loss value to obtain an unsupervised loss value; the unsupervised loss value is used to update the parameters of the regression detector and the student classification detector;

[0036] Adding the supervised regression loss value and the supervised classification loss value to obtain a supervised loss value; the supervised loss value is used to update the parameters of the feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector;

[0037] The unsupervised loss value and the supervised loss value are added to obtain the total loss function value.

[0038] According to a semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention, the target detection model is determined based on the total loss function value, including:

[0039] Based on the total loss function value, update the parameters of the initial feature extractor, the initial region generation module, the initial region pooling module, the initial regression detector, the initial student classification detector and the initial teacher classification detector, and repeat the step of determining the total loss function value until the target number of iterations is reached or the total loss function value tends to be stable;

[0040] The finally updated parameters are assigned to the target detection model to obtain the target detection model.

[0041] The present invention also provides a semi-supervised remote sensing target detection device based on detector decoupling, comprising:

[0042] A first acquisition module is used to acquire a remote sensing image to be detected;

[0043] A detection module is used to input the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; the target detection model is trained based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target.

[0044] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the semi-supervised remote sensing target detection method based on detector decoupling as described above is implemented.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described semi-supervised remote sensing target detection methods based on detector decoupling.

[0046] The present invention provides a semi-supervised remote sensing target detection method and device based on detector decoupling, which obtains a remote sensing image to be detected; inputs the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; the target detection model is trained based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target. Since the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image are combined when training the target detection model, the problem of reducing the quality of pseudo labels due to the rotation of the remote sensing image and the change of the aspect ratio, which in turn leads to poor model performance, is avoided, so that the performance of the trained target detection model is better, thereby improving the detection accuracy of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a schematic diagram of the intersection-over-union ratio of the predicted box and the real box provided by the prior art.

[0049] Figure 2 It is a flow chart of the semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention.

[0050] Figure 3 Schematic diagram of standard deviation of aspect ratios of sample targets of different categories provided by the present invention.

[0051] Figure 4 It is a training schematic diagram of the target detection model provided by the present invention.

[0052] Figure 5 It is a comparative schematic diagram of the aspect ratio distribution and detection accuracy of different sample targets provided by the present invention.

[0053] Figure 6 It is a structural schematic diagram of a semi-supervised remote sensing target detection device based on detector decoupling provided by the present invention.

[0054] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In order to facilitate a clearer understanding of the various embodiments of the present application, relevant background knowledge is first introduced.

[0057] Existing semi-supervised target detection technologies are mostly aimed at ordinary target detection datasets. Semi-supervised target detection technologies mainly include two points: self-training and consistency regularization.

[0058] The self-training process is divided into the following steps: 1. Initial training: use labeled data to train an initial model; 2. Predict unlabeled data: use the initial model to predict unlabeled data; 3. Select high-confidence predictions: select high-confidence prediction data from the predictions; 4. Expand the training set: add high-confidence prediction data (with pseudo-labels) to the training set; 5. Iterative training: use the expanded training set to re-iterate the training model; 6. Convergence and termination: stop iteration when the performance of the model no longer improves significantly or reaches the predetermined number of iterations.

[0059] The process of consistency regularization is divided into the following steps: 1. Training model: Use labeled data to train the basic model; 2. Data perturbation: Apply random perturbations to unlabeled data; 3. Consistency prediction: Use the basic model to predict the perturbed unlabeled data; 4. Calculate consistency loss: Evaluate the prediction consistency of the model for the perturbed unlabeled data; 5. Combine supervised loss: Combine the consistency loss with the supervised loss to form a total loss function; 6. Model update: Use the total loss function to update the model parameters; 7. Iterative process: Repeat the above process until the model converges.

[0060] Semi-supervised object detection technology combines self-training and consistency regularization. The process is as follows: 1. Randomly initialize to obtain two models with the same parameters, namely the teacher model and the student model (Fast Regional Convolutional Neural Networks (Faster R-Convolutional Neural Networks, Faster R-CNN) structure); 2. Use labeled images to train the student model and the teacher; 3. The unlabeled images are subjected to strong transformation and weak transformation respectively. The images after weak transformation are input into the teacher model to obtain the prediction results, and the pseudo labels are obtained after screening. The student model predicts the images after strong transformation. The pseudo labels generated by the teacher model supervise the training of the student model, and back-propagate to update the parameters of the student model. The teacher model does not participate in back-propagation. The parameter update of the teacher model is obtained by the exponential sliding average of the student model; 4. Repeat the above process until the loss function converges.

[0061] In recent years, there have been many innovations based on the above semi-supervised object detection framework, for example, adding window jittering or uncertainty prediction modules in the pseudo-label selection process to improve the accuracy of pseudo-labels, or adding focal loss or soft weights in the consistency regularization process to reduce error accumulation.

[0062] However, in complex remote sensing data, simply migrating the general semi-supervised target detection framework to the detection of remote sensing images will not produce good results. Due to the small number of categories in the remote sensing target detection dataset, small intra-class gaps, large inter-class gaps, and large number of targets in a single image, a small amount of data is enough to fine-tune the backbone network in the teacher model and the student model, while the detectors in the teacher model and the student model may not be fully trained. The errors caused by the detectors in the unsupervised training process are back-propagated to the backbone network, resulting in a decrease in the performance of the backbone network, which leads to poor performance of the model.

[0063] In addition, the aspect ratio and size of objects in remote sensing images vary significantly, and for objects with significant aspect ratios, the quality of pseudo-labels in the self-training process will be further reduced due to the addition of a rotation mechanism. Figure 1 It is a schematic diagram of the intersection-over-union ratio of the predicted box and the real box provided by the prior art, such as Figure 1 As shown in the figure, the dotted box is the predicted box (pseudo label) of the model predicted target (ellipse), and the solid box is the ground truth (GT) box (real label) of the target. The predicted box can get a higher intersection over union (IOU) by calculating with the GT box, but the dotted box is still a pseudo label box of very low quality. In the process of multiple iterations of self-training, this low-quality pseudo label box will further amplify the negative impact of the detector error, resulting in poor model performance.

[0064] Based on the above-described problems, the present invention provides a semi-supervised remote sensing target detection method based on detector decoupling, which reduces the error accumulation caused by target rotation and type characteristics of remote sensing data sets to improve the performance of the model.

[0065] Combine the following Figure 2-Figure 5 A semi-supervised remote sensing target detection method based on detector decoupling of the present invention is described.

[0066] Figure 2 is a flow chart of the semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention, such as Figure 2 As shown, the method includes steps 201 and 202; wherein,

[0067] Step 201: Acquire a remote sensing image to be detected.

[0068] It should be noted that the semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention can be applied to the scenario of remote sensing satellite image detection. The executor of the method can be a semi-supervised remote sensing target detection device based on detector decoupling, such as an electronic device, or a control module in the semi-supervised remote sensing target detection device based on detector decoupling for executing the semi-supervised remote sensing target detection method based on detector decoupling.

[0069] Specifically, a remote sensing image to be detected is acquired by photographing a remote sensing image by an airplane or a satellite, and the remote sensing image to be detected includes at least one target.

[0070] Step 202, input the remote sensing image to be detected into the target detection model to obtain the detection result output by the target detection model; the target detection model is trained based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target.

[0071] Specifically, the target detection model is trained based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing images, the labeled sample remote sensing images, and at least one sample target in the labeled sample remote sensing images. The target detection model is used to detect at least one target in the remote sensing image to be detected; the detection results include the detection frame coordinates and classification type of each target.

[0072] After the target detection model is trained, the remote sensing image to be detected is input into the target detection model to obtain the detection result output by the target detection model.

[0073] The semi-supervised remote sensing target detection method based on detector decoupling provided by the present invention is obtained by acquiring a remote sensing image to be detected; inputting the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; the target detection model is obtained by training based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target. Since the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image are combined when training the target detection model, the problem of reducing the quality of pseudo labels due to the rotation of the remote sensing image and the change of the aspect ratio, which in turn leads to poor model performance, is avoided, so that the performance of the trained target detection model is better, thereby improving the detection accuracy of the target detection model.

[0074] Optionally, the target detection model includes a feature extractor, a region generation module, a region pooling module, a regression detector and a student classification detector; the specific implementation process of the above step 202 includes:

[0075] (1) Inputting the remote sensing image to be detected into the feature extractor to obtain a feature image output by the feature extractor.

[0076] Specifically, the feature extractor is used to extract the features of the remote sensing image to be detected. The remote sensing image to be detected is input into the feature extractor, and a feature image output by the feature extractor can be obtained.

[0077] (2) Inputting the feature image into the region generation module to obtain a region image corresponding to each target in the feature image output by the region generation module.

[0078] Specifically, the region generation module is used to generate a region image corresponding to each target in the feature image, wherein the region generation module may be a region proposal network (RPN), and the size of the region image is smaller than the size of the feature image.

[0079] By inputting the feature image into the region generation module, the region image corresponding to each target in the feature image output by the region generation module can be obtained.

[0080] (3) Inputting each of the regional images into the regional pooling module to obtain the regional features corresponding to each of the targets output by the regional pooling module; the dimensions of the regional features corresponding to each of the targets are the same.

[0081] Specifically, the regional pooling module is used to unify the dimensions of each regional image so that the dimensions of each regional image are the same. By inputting each regional image into the regional pooling module, the regional features (i.e., regional proposal boxes) corresponding to each target output by the regional pooling module can be obtained; wherein, the dimensions of the regional features corresponding to each target are the same, for example, the dimensions of the regional features corresponding to each target are 256. 7 7.

[0082] (4) Inputting the regional features corresponding to each of the targets into the regression detector to obtain the detection frame coordinates of each of the targets output by the regression detector.

[0083] Specifically, the regression detector is used to detect each target and obtain the detection frame coordinates. The region features corresponding to each target are input into the regression detector, and the detection frame coordinates of each target output by the regression detector can be obtained.

[0084] (5) Inputting the regional features corresponding to each of the targets into the student classification detector to obtain the classification type of each of the targets output by the student classification detector.

[0085] Specifically, the student classification detector is used to detect the classification score of each target and obtain the classification type of each target based on the classification score. The regional features corresponding to each target are input into the student classification detector, and the classification type of each target output by the student classification detector can be obtained.

[0086] In an embodiment of the present application, a remote sensing image to be detected is input into a feature extractor to obtain a feature image output by the feature extractor; the feature image is input into a region generation module to obtain a region image corresponding to each target in the feature image output by the region generation module; each region image is input into a region pooling module to obtain a region feature corresponding to each target output by the region pooling module; the dimension of the region feature corresponding to each target is the same; the region feature corresponding to each target is input into a regression detector to obtain the detection frame coordinates of each target output by the regression detector; the region feature corresponding to each target is input into a student classification detector to obtain the classification type of each target output by the student classification detector. By decoupling the feature extractor from the regression detector and the student classification detector, the problem of inaccurate detection frame caused by the rotation problem of the remote sensing image to be detected is solved, thereby improving the accuracy of target detection in the remote sensing image to be detected.

[0087] Optionally, the target detection model is trained based on the following steps:

[0088] Acquire a training data set; the training data set includes a plurality of unlabeled sample remote sensing images and a plurality of labeled sample remote sensing images; use each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images to train an initial detection model to obtain the target detection model.

[0089] Specifically, in remote sensing semi-supervised target detection, a training data set is first obtained, where the training data set includes multiple unlabeled sample remote sensing images and multiple labeled sample remote sensing images, that is, a set of labeled data and a set of unlabeled data , where x j represents the jth sample remote sensing image, y j Represents the category and coordinate label of the detection box of the jth sample remote sensing image, represents the number of labeled data, represents the amount of unlabeled data, The goal of semi-supervised remote sensing target detection is to use unlabeled data to assist model training.

[0090] The initial detection model can be further trained using the unlabeled sample remote sensing images and the labeled sample remote sensing images to obtain a final trained target detection model.

[0091] Optionally, the adopting of each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images to train an initial detection model to obtain the target detection model includes:

[0092] (a) Inputting each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images into an initial feature extractor respectively, and obtaining a first sample feature image corresponding to each of the unlabeled sample remote sensing images and a second sample feature image corresponding to each of the labeled sample remote sensing images output by the initial feature extractor.

[0093] Specifically, by inputting each unlabeled sample remote sensing image into the initial feature extractor, a first sample feature image corresponding to each unlabeled sample remote sensing image output by the initial feature extractor can be obtained; by inputting each labeled sample remote sensing image into the initial feature extractor, a second sample feature image corresponding to each labeled sample remote sensing image output by the initial feature extractor can be obtained.

[0094] (b) Inputting each of the first sample feature images and each of the second sample feature images into an initial region generation module respectively, to obtain a first sample region image corresponding to each of the sample targets in each of the first sample feature images and a second sample region image corresponding to each of the sample targets in each of the second sample feature images output by the initial region generation module.

[0095] Specifically, by inputting each first sample feature image into the initial area generation module, a first sample area image corresponding to each sample target in each first sample feature image output by the initial area generation module can be obtained; by inputting each second sample feature image into the initial area generation module, a second sample area image corresponding to each sample target in each second sample feature image output by the initial area generation module can be obtained.

[0096] (c) Inputting each of the first sample area images and each of the second sample area images into an initial area pooling module respectively, and obtaining first sample area features corresponding to each of the first sample area images and second sample area features corresponding to each of the second sample area images output by the initial area pooling module; the dimensions of each of the first sample area features are the same, and the dimensions of each of the second sample area features are the same.

[0097] Specifically, by inputting each first sample area image into the initial area pooling module, the first sample area features corresponding to each first sample area image output by the initial area pooling module can be obtained; by inputting each second sample area image into the initial area pooling module, the second sample area features corresponding to each second sample area image output by the initial area pooling module can be obtained; wherein the dimensions of each first sample area feature are the same, and the dimensions of each second sample area feature are the same.

[0098] (d) Determining the object detection model based on each of the first sample region features and each of the second sample region features.

[0099] Specifically, based on each first sample region feature and each second sample region feature, a target detection model may be further determined.

[0100] Optionally, determining the target detection model based on each of the first sample region features and each of the second sample region features includes:

[0101] (d-1) Using at least one feature enhancement method to enhance each of the first sample region features, to obtain a plurality of enhanced first sample region features.

[0102] Specifically, the feature enhancement method may be a method of adding random Gaussian noise or a strong transformation method, wherein the strong transformation method is a rotation at a larger angle, application of random noise, color jittering, enlargement after cropping, or perspective transformations.

[0103] At least one feature enhancement method may be used to enhance the features of each first sample region to obtain multiple enhanced first sample region features. For example, by adding random Gaussian noise to enhance the features of each first sample region to obtain multiple enhanced first sample region features.

[0104] (d-2) Inputting each of the enhanced first sample area features and each of the second sample area features into an initial regression detector, and obtaining first sample detection frame coordinates of each of the sample targets corresponding to each of the enhanced first sample area features and second sample detection frame coordinates of each of the sample targets corresponding to each of the second sample area features output by the initial regression detector.

[0105] It should be noted that in R-CNN-based detectors, two different head structures (i.e., fully connected head and convolutional head) that share the same feature layer are usually used for classification and regression tasks, which will lead to multi-task conflicts between the convolutional layer used for classification and the fully connected layer used for regression, thereby hindering the generation of high-quality pseudo-labels. In order to solve this problem, the present application divides the detector into three independent branches, namely, a regression detector, a student classification detector, and a teacher classification detector. The classification detector is then converted into a simple classification task that is fully utilized in semi-supervised learning (SSL), so that a wide range of semi-supervised classification techniques can be migrated to semi-supervised detection tasks. Since the fully connected network performs better in classification tasks, the classification detector of the present application uses two fully connected layers to obtain classification prediction results, where the sizes of the two fully connected layers are 12544 respectively. 1024 vs 1024 Number of categories. Since convolutional networks perform better in regression tasks, the regression detector in this application uses two convolutional layers and one fully connected layer to obtain the final prediction result, where the size of the convolutional layer is 3 3 1024, the size of the fully connected layer is 1024 4.

[0106] Specifically, by inputting each enhanced first sample area feature into the initial regression detector, the first sample detection frame coordinates of each sample target corresponding to each enhanced first sample area feature output by the initial regression detector can be obtained. For example, by respectively inputting each enhanced first sample area feature obtained by adding different random Gaussian noises into the initial regression detector, the first sample detection frame coordinates of each sample target corresponding to each enhanced first sample area feature obtained by adding different random Gaussian noises distributed output by the initial regression detector can be obtained.

[0107] By inputting each second sample region feature into the initial regression detector, the second sample detection frame coordinates of each sample target corresponding to each second sample region feature output by the initial regression detector can be obtained.

[0108] (d-3) Inputting each of the enhanced first sample region features and each of the second sample region features into an initial student classification detector, and obtaining a first classification score for each of the sample targets corresponding to each of the enhanced first sample region features and a second classification score for each of the sample targets corresponding to each of the second sample region features output by the initial student classification detector.

[0109] Specifically, by inputting each enhanced first sample region feature into the initial student classification detector, the first classification score of each sample target corresponding to each enhanced first sample region feature output by the initial student classification detector can be obtained. For example, by inputting each enhanced first sample region feature obtained by a strong transformation into the initial student classification detector, the first classification score of each sample target corresponding to each enhanced first sample region feature output by the initial student classification detector can be obtained.

[0110] By inputting each second sample region feature into the initial student classification detector, a second classification score of each sample target corresponding to each second sample region feature output by the initial student classification detector can be obtained.

[0111] (d-4) Inputting each of the second sample area features into an initial teacher classification detector to obtain a pseudo label of each of the sample targets output by the initial teacher classification detector; the structure of the initial teacher classification detector is the same as that of the initial student classification detector, and the parameters of the initial teacher classification detector are updated based on the exponential moving average of the parameters of the initial student classification detector.

[0112] Specifically, the features of each second sample region are input into the initial teacher classification detector to obtain the third classification score of each sample target output by the initial teacher classification detector, and the third classification score is compared with the preset threshold, and the category corresponding to the third classification score greater than the preset threshold is used as the pseudo label of each sample target. 7 The second sample region feature (ROI feature map) of 7 is sent to the initial teacher classification detector to generate a pseudo label without box coordinates. The pseudo label will be used as the true label to supervise the initial student classification detector to be trained on the first sample region feature (ROI feature map) obtained in a strong transformation manner. At the same time, the initial student classification detector will pass the parameters to the initial teacher classification detector through EMA.

[0113] It should be noted that the structure of the initial teacher classification detector is the same as that of the initial student classification detector, and the parameters of the initial teacher classification detector are updated based on the exponential moving average (EMA) of the parameters of the initial student classification detector.

[0114] (d-5) Determine the total loss function value based on the first sample detection frame coordinates of each of the sample targets, the weight corresponding to the aspect ratio of the true detection frame of each of the sample targets, the second sample detection frame coordinates of each of the sample targets, the true detection frame coordinates of each of the sample targets, the first classification score of each of the sample targets, the second classification score of each of the sample targets and the pseudo label of each of the sample targets.

[0115] Specifically, based on the first sample detection box coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection box of each sample target, the second sample detection box coordinates of each sample target, the real detection box coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target, the total loss function value can be further determined.

[0116] (d-6) Determining the target detection model based on the total loss function value.

[0117] Specifically, based on the total loss function value, the target detection model can be further determined.

[0118] Optionally, determining the total loss function value based on the first sample detection frame coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection frame of each sample target, the second sample detection frame coordinates of each sample target, the real detection frame coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target includes:

[0119] 1) Determine an unsupervised regression loss value based on the weights corresponding to the first sample detection frame coordinates of each of the sample targets and the aspect ratios of the true detection frames of each of the sample targets.

[0120] It should be noted that better prediction performance means stronger anti-interference ability, and the unsupervised loss is optimized by using the prior aspect ratio of each category as a weight, which is calculated by the true detection box of each sample target (i.e., the true value label of the labeled data). Assuming that the distribution of labeled data is the same as that of unlabeled data, in general, the more accurate the model prediction, the more credible the sample, so the loss function weight should increase. However, the opposite conclusion is drawn in this application, and the loss function weight is positively correlated with the standard deviation of the aspect ratio.

[0121] Regarding regression tasks, the effectiveness of existing semi-supervised frameworks is unclear. Although some studies do not consider semi-supervised regression losses, they still achieve good results. But this ambiguity comes from the task inconsistency between classification and regression, and rotation will further aggravate the multi-task conflict caused by task inconsistency, and low-quality pseudo-labels will mislead the training direction of the regression detector. Therefore, in this application, a self-supervised framework is used to utilize unlabeled data instead of a semi-supervised method with less accurate pseudo-labels. Specifically, a random enhancement containing random Gaussian noise, random cropping, or brightening is set, hoping that the randomly enhanced detection box can produce better prediction results. Unsupervised regression loss function as follows:

[0122] (1)

[0123] in, represents a randomly enhanced sequence function, and Represents the coordinates of the detection boxes after two different enhancements r Offset value, that is, the coordinates of the two different enhanced detection boxes, r Indicates the offset value. The offset value will revise the coordinate value of the original region suggestion box to obtain the coordinate value of the final detection box. u indicates unlabeled data. represents the ratio of unlabeled data to labeled data, B represents the number of labeled data participating in training, and b represents the subscript of the count.

[0124] Specifically, the weight corresponding to the aspect ratio of the real detection box of each sample target is It is expressed by formula (2):

[0125] (2)

[0126] in, represents the width of the true detection box of the i-th category, represents the length of the true detection box of the i-th category, represents the standard error function, Represents a logarithmic function.

[0127] Figure 3 is a schematic diagram of the standard deviation of aspect ratios of sample targets of different categories provided by the present invention, such as Figure 3 As shown in the figure, the standard deviation of the aspect ratio of sample targets of 15 categories in the Dota-split dataset. Categories with larger standard deviations include large vehicles and ports, which usually have larger aspect ratios. Due to the rotation characteristics of remote sensing data, the aspect ratio of the horizontal detection box will be more dispersed for thinner targets, and the final detection results will be less accurate than those of other categories of targets.

[0128] After obtaining the weight corresponding to the aspect ratio of the true detection frame of each sample target, the unsupervised regression loss expressed by the above formula (1) can be optimized, and the optimized unsupervised regression loss value is expressed by formula (3): ;in:

[0129] (3)

[0130] Based on the weights corresponding to the coordinates of the first sample detection box of each sample target and the aspect ratio of the true detection box of each sample target, the above formula (3) can be used to determine the unsupervised regression loss value.

[0131] 2) Determine a supervised regression loss value based on the second sample detection frame coordinates of each of the sample targets and the real detection frame coordinates of each of the sample targets.

[0132] Specifically, based on the coordinates of the second sample detection frame of each sample target and the coordinates of the real detection frame of each sample target, the supervised regression loss value can be determined: , reg means regression, Represents the label, that is, the coordinates of the real detection box. The supervised regression loss value is used to train the regression detector, that is, the ability of the training detector to predict the coordinates of the detection box.

[0133] 3) Determine an unsupervised classification loss value based on the first classification score of each of the sample targets and the pseudo label of each of the sample targets.

[0134] Specifically, based on the first classification score of each sample target and the pseudo label of each sample target, the unsupervised classification loss value can be determined using formulas (4) and (5): ;in:

[0135] (4)

[0136] (5)

[0137] in, represents a pseudo label, represents the first classification score, y Represents the prediction result, represents the first sample region feature after enhancement, that is, the sample region feature after strong enhancement, It represents the first sample region feature after enhancement, that is, the sample region feature after weak enhancement.

[0138] 4) Determine a supervised classification loss value based on the second classification score of each of the sample objects corresponding to each of the second sample region features and the true detection frame coordinates of each of the sample objects.

[0139] Specifically, based on the second classification score of each sample target corresponding to each second sample region feature and the real detection frame coordinates of each sample target, the supervised classification loss value can be determined: , cls represents classification, that is, classification prediction loss function.

[0140] 5) Adding the unsupervised regression loss value and the unsupervised classification loss value to obtain an unsupervised loss value; the unsupervised loss value is used to update the parameters of the regression detector and the student classification detector.

[0141] Specifically, the unsupervised regression loss value and unsupervised classification loss Add together to get the unsupervised loss value , that is, it can be expressed as:

[0142] (6)

[0143] It should be noted that during the training process, the unsupervised loss value is used to update the parameters of the regression detector and the student classification detector, that is, only the parameters of the regression detector and the student classification detector are updated during back propagation, without updating the parameters of the feature extractor.

[0144] 6) Adding the supervised regression loss value and the supervised classification loss value to obtain a supervised loss value; the supervised loss value is used to update the parameters of the feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector.

[0145] Specifically, there will be a supervised regression loss value And supervised classification loss value Add together to get the supervised loss value , that is, it can be expressed as:

[0146] (7)

[0147] It should be noted that during the training process, supervised loss values ​​are used to update the parameters of the feature extractor, region generation module, region pooling module, regression detector, and student classification detector, that is, the parameters of all modules are updated during back propagation. In view of the small intra-class gap and large inter-class gap of remote sensing images, the feature extractor is only trained on the labeled data set, that is, the separated feature extractors (backbone network) are integrated into a shared network trained only by labeled data, and the coupled classification detector and regression detector are separated, and the classification detector and regression detector are optimized and trained.

[0148] 7) Adding the unsupervised loss value and the supervised loss value to obtain the total loss function value.

[0149] Specifically, the final training objective includes an unsupervised loss value and a supervised loss value. By adding the unsupervised loss value and the supervised loss value, the total loss function value can be obtained.

[0150] Optionally, determining the target detection model based on the total loss function value includes:

[0151] Based on the total loss function value, update the parameters of the initial feature extractor, the initial region generation module, the initial region pooling module, the initial regression detector, the initial student classification detector and the initial teacher classification detector, and repeat the step of determining the total loss function value until the target number of iterations is reached or the total loss function value tends to be stable; assign the final updated parameters to the target detection model to obtain the target detection model.

[0152] Specifically, according to the determined total loss function value, it is judged whether the total loss function is regionally stable. When the total loss function value changes, the parameters of the regression detector and the student classification detector can be updated based on the unsupervised loss value, and the parameters of the feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector can be updated based on the supervised loss value, and the steps of determining the total loss function value are repeated until the target number of iterations is reached or the total loss function value tends to be stable; wherein the target number of iterations is the maximum number of iterations, for example, the target number of iterations is 1000. The finally updated parameters are assigned to the target detection model, that is, the parameters of the finally updated feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector are assigned to the target detection model, thereby obtaining the target detection model.

[0153] Figure 4 It is a training diagram of the target detection model provided by the present invention, such as Figure 4 As shown, this application proposes a new semi-supervised remote sensing target detection framework, namely Splitting Detector (SLD). The following improvements are made to the existing semi-supervised target detection framework: 1. To address the problem of inaccurate detection frames caused by rotation, the Faster R-CNN detector and feature extractor are decoupled, and the regression and classification detectors are independently trained in an unsupervised manner. 2. In view of the fact that the intra-class gap of remote sensing images is small and the inter-class gap is large, the feature extractor is only trained on labeled data sets. The specific steps include:

[0154] Step 1: Obtain a training data set, where the training data set includes a plurality of unlabeled sample remote sensing images and a plurality of labeled sample remote sensing images. For example, the number of labeled sample remote sensing images is n.

[0155] Step 2: Input each unlabeled sample remote sensing image and each labeled sample remote sensing image into the initial feature extractor respectively, and obtain the first sample feature image corresponding to each unlabeled sample remote sensing image and the second sample feature image corresponding to each labeled sample remote sensing image output by the initial feature extractor.

[0156] Step 3: Input each first sample feature image and each second sample feature image into the initial region generation module respectively, and obtain the first sample region image corresponding to each sample target in each first sample feature image and the second sample region image corresponding to each sample target in each second sample feature image output by the initial region generation module.

[0157] Step 4: Input each first sample region image and each second sample region image into the initial region pooling module (i.e., ROI pooling module) respectively, and obtain the first sample region features (i.e., features corresponding to the region proposal box) corresponding to each first sample region image output by the initial region pooling module and the second sample region features (i.e., features corresponding to the region proposal box) corresponding to each second sample region image; wherein the dimensions of the first sample region features are the same, and the dimensions of the second sample region features are the same.

[0158] Step 5: Use two different feature enhancement methods of adding random Gaussian noise (feature enhancement 1 and feature enhancement 2) to enhance the features of each first sample region, and obtain multiple enhanced first sample region features. Use a feature enhancement method of strong transformation (feature enhancement 3) to enhance the features of each first sample region, and obtain multiple enhanced first sample region features.

[0159] Step 6: Input each enhanced first sample region feature to the initial regression detector to obtain the first sample detection frame coordinates of each sample target corresponding to each enhanced first sample region feature output by the initial regression detector.

[0160] Step 7: Input each second sample region feature into the initial regression detector to obtain the second sample detection frame coordinates of each sample target corresponding to each second sample region feature output by the initial regression detector.

[0161] Step 8: Input each enhanced first sample region feature into the initial student classification detector to obtain the first classification score of each sample target corresponding to each enhanced first sample region feature output by the initial student classification detector.

[0162] Step 9: Input each second sample region feature into the initial student classification detector to obtain a second classification score of each sample target corresponding to each second sample region feature output by the initial student classification detector.

[0163] Step 10: Input each second sample region feature (i.e., the feature remains unchanged) into the initial teacher classification detector to obtain a pseudo label of each sample target output by the initial teacher classification detector. The structure of the initial teacher classification detector is the same as that of the initial student classification detector, and the parameters of the initial teacher classification detector are updated based on the exponential moving average of the parameters of the initial student classification detector.

[0164] Step 11: weights corresponding to the first sample detection frame coordinates of each sample target and the aspect ratio of the real detection frame of each sample target (for example, , …, ), the unsupervised regression loss value is determined by the above formula (3); based on the first classification score of each sample target and the pseudo label of each sample target, the unsupervised classification loss value is determined by formula (4); based on the second sample detection frame coordinates of each sample target and the real detection frame coordinates of each sample target, the supervised regression loss value is determined; based on the second classification score of each sample target corresponding to each second sample region feature and the real detection frame coordinates of each sample target, the supervised classification loss value is determined.

[0165] Step 12: Add the unsupervised regression loss value and the unsupervised classification loss value to obtain the unsupervised loss value; the unsupervised loss value is used to update the parameters of the regression detector and the student classification detector; add the supervised regression loss value and the supervised classification loss value to obtain the supervised loss value; the supervised loss value is used to update the parameters of the feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector; finally, add the unsupervised loss value and the supervised loss value to obtain the total loss function value.

[0166] Step 13. When the total loss function value does not tend to be stable, update the parameters of the regression detector and the student classification detector based on the unsupervised loss value, and update the parameters of the feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector based on the supervised loss value, and repeat the step of determining the total loss function value until the target number of iterations is reached or the total loss function value tends to be stable; assign the final updated parameters to the target detection model to obtain the target detection model.

[0167] It should be noted that, unlike common datasets, the difference between instances of the same category in remote sensing images is small, the size of instances of the same category varies greatly, and the aspect ratio changes of detection frames of different categories also vary greatly. Therefore, although the number of remote sensing data categories is small, the detection difficulty of different categories varies greatly. However, it is found that there is a strong correlation between the detection difficulty and the degree of change in the aspect ratio of detection frames of different categories (this is because the higher the aspect ratio of the target, the more it is affected by rotation). Figure 5 is a schematic diagram comparing the aspect ratio distribution and detection accuracy of different sample targets provided by the present invention, such as Figure 5 As shown in the figure, the different sample targets are ports and storage tanks. When only 10% of the data is used to train the model, the detection performance of storage tanks, which are more prone to aspect ratio changes, is better than that of ports. The same pattern also exists for other categories of sample targets.

[0168] This application will be compared in detail with supervised baselines and state-of-the-art semi-supervised object detection methods, including Soft-Teacher and Unbiased-Teacher V2. The training dataset is DOTA_split. Table 1 shows the comparison results of different supervised detection methods. In order to make a fair comparison, this application uses the same structure in the detector, where the feature extractor can be ResNet-50 or (Feature Pyramid Network, FPN), and the region generation network is RPN. The mean average precision (mAP) is used as the evaluation criterion, where the mean average precision is shown in formulas (8) and (9):

[0169] (8)

[0170] (9)

[0171] Where n represents the number of sample targets, P represents the proportion of true objects detected in all positive predictions, and R represents the proportion of true objects detected in all GTs.

[0172] Under different experimental settings, the performance of the SLD of this application on the DOTA_ split dataset is far superior to the most advanced semi-supervised object detection methods. Specifically, the method provided by this application is 1.1, 1.4, and 2.1 higher than UnbiasedV2 in terms of mAP, and improves the accuracy of supervised learning by 15.1, 14.3, and 12.5mAP. Where P is the proportion of true objects detected in all positive predictions, and R is the proportion of true objects detected in all GTs.

[0173] Table 1. Comparison results of different supervised detection methods

[0174]

[0175] Compared with the SOTA semi-supervised detection method, the method provided in this application has higher detection accuracy and detects more true values. In addition, the method provided in this application has less accumulated errors, which greatly reduces the false detection rate, which is of great significance in the field of remote sensing.

[0176] The method provided by the present application is an improved semi-supervised detection framework that is more adapted to the characteristics of remote sensing images. The existing parallel dual-branch semi-supervised detection framework is changed into a unified feature extractor and an independent parallel regression detector and a student detector. On the one hand, it can reduce the error accumulation caused by low-quality pseudo labels and improve the performance of the model; on the other hand, the method of the present application provides a bridge between semi-supervised target detection and semi-supervised classification, so that a large number of semi-supervised classification methods can be embedded in the semi-supervised detection framework of the present application, making it more widely used.

[0177] The following is a description of the semi-supervised remote sensing target detection device based on detector decoupling provided by the present invention. The semi-supervised remote sensing target detection device based on detector decoupling described below and the semi-supervised remote sensing target detection method based on detector decoupling described above can be referenced to each other.

[0178] Figure 6 is a schematic diagram of the structure of a semi-supervised remote sensing target detection device based on detector decoupling provided by the present invention, such as Figure 6 As shown, the semi-supervised remote sensing target detection device 600 based on detector decoupling includes a first acquisition module 601 and a detection module 602; wherein,

[0179] The first acquisition module 601 is used to acquire a remote sensing image to be detected;

[0180] The detection module 602 is used to input the remote sensing image to be detected into the target detection model to obtain the detection result output by the target detection model; the target detection model is trained based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target.

[0181] The semi-supervised remote sensing target detection device based on detector decoupling provided by the present invention obtains a remote sensing image to be detected; inputs the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; the target detection model is trained based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target. Since the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image are combined when training the target detection model, the problem of reducing the quality of pseudo labels due to the rotation of the remote sensing image and the change of the aspect ratio, which in turn leads to poor model performance, is avoided, so that the performance of the trained target detection model is better, thereby improving the detection accuracy of the target detection model.

[0182] Optionally, the target detection model includes a feature extractor, a region generation module, a region pooling module, a regression detector and a student classification detector; the detection module 602 is specifically used for:

[0183] Inputting the remote sensing image to be detected into the feature extractor to obtain a feature image output by the feature extractor;

[0184] Inputting the feature image into the region generation module to obtain a region image corresponding to each target in the feature image output by the region generation module;

[0185] Inputting each of the regional images into the regional pooling module to obtain the regional features corresponding to each of the targets output by the regional pooling module; the dimensions of the regional features corresponding to each of the targets are the same;

[0186] Inputting the regional features corresponding to each of the targets into the regression detector to obtain the detection frame coordinates of each of the targets output by the regression detector;

[0187] The regional features corresponding to each of the targets are input into the student classification detector to obtain the classification type of each of the targets output by the student classification detector.

[0188] Optionally, the semi-supervised remote sensing target detection device 600 based on detector decoupling further includes:

[0189] A second acquisition module is used to acquire a training data set; the training data set includes a plurality of unlabeled sample remote sensing images and a plurality of labeled sample remote sensing images;

[0190] The training module is used to train the initial detection model using the unlabeled sample remote sensing images and the labeled sample remote sensing images to obtain the target detection model.

[0191] Optionally, the training module is specifically used to:

[0192] Inputting each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images into an initial feature extractor respectively, and obtaining a first sample feature image corresponding to each of the unlabeled sample remote sensing images and a second sample feature image corresponding to each of the labeled sample remote sensing images output by the initial feature extractor;

[0193] Inputting each of the first sample feature images and each of the second sample feature images into an initial region generation module respectively, and obtaining a first sample region image corresponding to each of the sample targets in each of the first sample feature images and a second sample region image corresponding to each of the sample targets in each of the second sample feature images output by the initial region generation module;

[0194] Inputting each of the first sample region images and each of the second sample region images into an initial region pooling module respectively, obtaining first sample region features corresponding to each of the first sample region images and second sample region features corresponding to each of the second sample region images output by the initial region pooling module; the dimensions of each of the first sample region features are the same, and the dimensions of each of the second sample region features are the same;

[0195] The target detection model is determined based on each of the first sample region features and each of the second sample region features.

[0196] Optionally, the training module is further used to:

[0197] Using at least one feature enhancement method, perform feature enhancement on each of the first sample region features to obtain a plurality of enhanced first sample region features;

[0198] Inputting each of the enhanced first sample region features and each of the second sample region features into an initial regression detector, obtaining first sample detection frame coordinates of each of the sample targets corresponding to each of the enhanced first sample region features and second sample detection frame coordinates of each of the sample targets corresponding to each of the second sample region features output by the initial regression detector;

[0199] Inputting each of the enhanced first sample region features and each of the second sample region features into an initial student classification detector, obtaining a first classification score of each of the sample targets corresponding to each of the enhanced first sample region features and a second classification score of each of the sample targets corresponding to each of the second sample region features output by the initial student classification detector;

[0200] Inputting each of the second sample region features into an initial teacher classification detector to obtain a pseudo label of each of the sample targets output by the initial teacher classification detector; the structure of the initial teacher classification detector is the same as that of the initial student classification detector, and the parameters of the initial teacher classification detector are updated based on the exponential moving average of the parameters of the initial student classification detector;

[0201] Determine a total loss function value based on the first sample detection frame coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection frame of each sample target, the second sample detection frame coordinates of each sample target, the real detection frame coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target;

[0202] Based on the total loss function value, the target detection model is determined.

[0203] Optionally, the training module is further used to:

[0204] Determine an unsupervised regression loss value based on the weights corresponding to the first sample detection frame coordinates of each of the sample targets and the aspect ratios of the real detection frames of each of the sample targets;

[0205] Determining a supervised regression loss value based on the second sample detection frame coordinates of each of the sample targets and the real detection frame coordinates of each of the sample targets;

[0206] Determining an unsupervised classification loss value based on the first classification score of each of the sample targets and the pseudo label of each of the sample targets;

[0207] Determine a supervised classification loss value based on the second classification score of each of the sample objects corresponding to each of the second sample region features and the real detection frame coordinates of each of the sample objects;

[0208] Adding the unsupervised regression loss value and the unsupervised classification loss value to obtain an unsupervised loss value; the unsupervised loss value is used to update the parameters of the regression detector and the student classification detector;

[0209] Adding the supervised regression loss value and the supervised classification loss value to obtain a supervised loss value; the supervised loss value is used to update the parameters of the feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector;

[0210] The unsupervised loss value and the supervised loss value are added to obtain the total loss function value.

[0211] Optionally, the training module is further used to:

[0212] Based on the total loss function value, update the parameters of the initial feature extractor, the initial region generation module, the initial region pooling module, the initial regression detector, the initial student classification detector and the initial teacher classification detector, and repeat the step of determining the total loss function value until the target number of iterations is reached or the total loss function value tends to be stable;

[0213] The finally updated parameters are assigned to the target detection model to obtain the target detection model.

[0214] Figure 7 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a semi-supervised remote sensing target detection method based on detector decoupling, the method comprising: obtaining a remote sensing image to be detected; inputting the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; the target detection model is obtained by training based on the weights corresponding to the aspect ratio of the real detection frame of the unlabeled sample remote sensing image, the labeled sample remote sensing image and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target.

[0215] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0216] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a semi-supervised remote sensing target detection method based on detector decoupling provided by the above-mentioned methods, the method comprising: acquiring a remote sensing image to be detected; inputting the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; the target detection model is trained based on weights corresponding to the aspect ratios of the real detection frames of unlabeled sample remote sensing images, labeled sample remote sensing images, and at least one sample target in the labeled sample remote sensing images, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each of the targets.

[0217] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0218] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semi-supervised remote sensing target detection method based on detector decoupling, It is characterized in that include: Acquire a remote sensing image to be detected; Inputting the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; The target detection model is obtained by training based on the weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image. The target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target; the target detection model includes a feature extractor, a region generation module, a region pooling module, a regression detector, and a student classification detector; The target detection model is trained based on the following steps: Acquire a training data set; the training data set includes a plurality of unlabeled sample remote sensing images and a plurality of labeled sample remote sensing images; Using each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images to train an initial detection model to obtain the target detection model; The method of using the unlabeled sample remote sensing images and the labeled sample remote sensing images to train the initial detection model to obtain the target detection model includes: Inputting each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images into an initial feature extractor respectively, and obtaining a first sample feature image corresponding to each of the unlabeled sample remote sensing images and a second sample feature image corresponding to each of the labeled sample remote sensing images output by the initial feature extractor; Inputting each of the first sample feature images and each of the second sample feature images into an initial region generation module respectively, and obtaining a first sample region image corresponding to each of the sample targets in each of the first sample feature images and a second sample region image corresponding to each of the sample targets in each of the second sample feature images output by the initial region generation module; Inputting each of the first sample region images and each of the second sample region images into an initial region pooling module respectively, obtaining first sample region features corresponding to each of the first sample region images and second sample region features corresponding to each of the second sample region images output by the initial region pooling module; the dimensions of each of the first sample region features are the same, and the dimensions of each of the second sample region features are the same; Determining the target detection model based on each of the first sample region features and each of the second sample region features; The determining the target detection model based on each of the first sample region features and each of the second sample region features includes: Using at least one feature enhancement method, perform feature enhancement on each of the first sample region features to obtain a plurality of enhanced first sample region features; Inputting each of the enhanced first sample region features and each of the second sample region features into an initial regression detector, obtaining first sample detection frame coordinates of each of the sample targets corresponding to each of the enhanced first sample region features and second sample detection frame coordinates of each of the sample targets corresponding to each of the second sample region features output by the initial regression detector; Inputting each of the enhanced first sample region features and each of the second sample region features into an initial student classification detector, obtaining a first classification score of each of the sample targets corresponding to each of the enhanced first sample region features and a second classification score of each of the sample targets corresponding to each of the second sample region features output by the initial student classification detector; Inputting each of the second sample region features into an initial teacher classification detector to obtain a pseudo label of each of the sample targets output by the initial teacher classification detector; the structure of the initial teacher classification detector is the same as that of the initial student classification detector, and the parameters of the initial teacher classification detector are updated based on the exponential moving average of the parameters of the initial student classification detector; Determine a total loss function value based on the first sample detection frame coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection frame of each sample target, the second sample detection frame coordinates of each sample target, the real detection frame coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target; Based on the total loss function value, the target detection model is determined.

2. The semi-supervised remote sensing target detection method based on detector decoupling according to claim 1, It is characterized in that Inputting the remote sensing image to be detected into the target detection model to obtain the detection result output by the target detection model includes: Inputting the remote sensing image to be detected into the feature extractor to obtain a feature image output by the feature extractor; Inputting the feature image into the region generation module to obtain a region image corresponding to each target in the feature image output by the region generation module; Inputting each of the regional images into the regional pooling module to obtain the regional features corresponding to each of the targets output by the regional pooling module; the dimensions of the regional features corresponding to each of the targets are the same; Inputting the regional features corresponding to each of the targets into the regression detector to obtain the detection frame coordinates of each of the targets output by the regression detector; The regional features corresponding to each of the targets are input into the student classification detector to obtain the classification type of each of the targets output by the student classification detector.

3. The semi-supervised remote sensing target detection method based on detector decoupling according to claim 1, It is characterized in that The determining of the total loss function value based on the first sample detection frame coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection frame of each sample target, the second sample detection frame coordinates of each sample target, the real detection frame coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target includes: Determine an unsupervised regression loss value based on the weights corresponding to the first sample detection frame coordinates of each of the sample targets and the aspect ratios of the real detection frames of each of the sample targets; Determining a supervised regression loss value based on the second sample detection frame coordinates of each of the sample targets and the real detection frame coordinates of each of the sample targets; Determining an unsupervised classification loss value based on the first classification score of each of the sample targets and the pseudo label of each of the sample targets; Determine a supervised classification loss value based on the second classification score of each of the sample objects corresponding to each of the second sample region features and the real detection frame coordinates of each of the sample objects; Adding the unsupervised regression loss value and the unsupervised classification loss value to obtain an unsupervised loss value; the unsupervised loss value is used to update the parameters of the regression detector and the student classification detector; Adding the supervised regression loss value and the supervised classification loss value to obtain a supervised loss value; the supervised loss value is used to update the parameters of the feature extractor, the region generation module, the region pooling module, the regression detector and the student classification detector; The unsupervised loss value and the supervised loss value are added to obtain the total loss function value.

4. The semi-supervised remote sensing target detection method based on detector decoupling according to claim 1, It is characterized in that The determining the target detection model based on the total loss function value includes: Based on the total loss function value, update the parameters of the initial feature extractor, the initial region generation module, the initial region pooling module, the initial regression detector, the initial student classification detector and the initial teacher classification detector, and repeat the step of determining the total loss function value until the target number of iterations is reached or the total loss function value tends to be stable; The finally updated parameters are assigned to the target detection model to obtain the target detection model.

5. A semi-supervised remote sensing target detection device based on detector decoupling, It is characterized in that include: A first acquisition module is used to acquire a remote sensing image to be detected; A detection module, used for inputting the remote sensing image to be detected into a target detection model to obtain a detection result output by the target detection model; the target detection model is obtained by training based on weights corresponding to the aspect ratios of the real detection frames of the unlabeled sample remote sensing image, the labeled sample remote sensing image, and at least one sample target in the labeled sample remote sensing image, and the target detection model is used to detect at least one target in the remote sensing image to be detected; the detection result includes the detection frame coordinates and classification type of each target; The target detection model includes a feature extractor, a region generation module, a region pooling module, a regression detector and a student classification detector; The semi-supervised remote sensing target detection device based on detector decoupling also includes: A second acquisition module is used to acquire a training data set; the training data set includes a plurality of unlabeled sample remote sensing images and a plurality of labeled sample remote sensing images; A training module, used to train the initial detection model using the unlabeled sample remote sensing images and the labeled sample remote sensing images to obtain the target detection model; The training module is specifically used for: Inputting each of the unlabeled sample remote sensing images and each of the labeled sample remote sensing images into an initial feature extractor respectively, and obtaining a first sample feature image corresponding to each of the unlabeled sample remote sensing images and a second sample feature image corresponding to each of the labeled sample remote sensing images output by the initial feature extractor; Inputting each of the first sample feature images and each of the second sample feature images into an initial region generation module respectively, and obtaining a first sample region image corresponding to each of the sample targets in each of the first sample feature images and a second sample region image corresponding to each of the sample targets in each of the second sample feature images output by the initial region generation module; Inputting each of the first sample region images and each of the second sample region images into an initial region pooling module respectively, obtaining first sample region features corresponding to each of the first sample region images and second sample region features corresponding to each of the second sample region images output by the initial region pooling module; the dimensions of each of the first sample region features are the same, and the dimensions of each of the second sample region features are the same; Determining the target detection model based on each of the first sample region features and each of the second sample region features; The training module is also used for: Using at least one feature enhancement method, perform feature enhancement on each of the first sample region features to obtain a plurality of enhanced first sample region features; Inputting each of the enhanced first sample region features and each of the second sample region features into an initial regression detector, obtaining first sample detection frame coordinates of each of the sample targets corresponding to each of the enhanced first sample region features and second sample detection frame coordinates of each of the sample targets corresponding to each of the second sample region features output by the initial regression detector; Inputting each of the enhanced first sample region features and each of the second sample region features into an initial student classification detector, obtaining a first classification score of each of the sample targets corresponding to each of the enhanced first sample region features and a second classification score of each of the sample targets corresponding to each of the second sample region features output by the initial student classification detector; Inputting each of the second sample region features into an initial teacher classification detector to obtain a pseudo label of each of the sample targets output by the initial teacher classification detector; the structure of the initial teacher classification detector is the same as that of the initial student classification detector, and the parameters of the initial teacher classification detector are updated based on the exponential moving average of the parameters of the initial student classification detector; Determine a total loss function value based on the first sample detection frame coordinates of each sample target, the weight corresponding to the aspect ratio of the real detection frame of each sample target, the second sample detection frame coordinates of each sample target, the real detection frame coordinates of each sample target, the first classification score of each sample target, the second classification score of each sample target and the pseudo label of each sample target; Based on the total loss function value, the target detection model is determined.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the semi-supervised remote sensing target detection method based on detector decoupling as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the semi-supervised remote sensing target detection method based on detector decoupling as described in any one of claims 1 to 4 is implemented.