Remote sensing image target detection method based on single-sample contrast feature changes

Abstract

This invention discloses a remote sensing image target detection method based on single-sample contrast feature changes. Belonging to the field of target detection technology, the method includes: acquiring an original remote sensing image set; preprocessing the original remote sensing image set to obtain an augmented image set; establishing a multi-head contrastive learning (MBR) training feature encoding network; inputting the augmented image set into the MBR training feature encoding network to obtain a trained feature extraction network; acquiring a remote sensing image to be tested and a conditional image; constructing a conditional target detection framework, wherein the conditional target detection framework includes the trained feature extraction network; and inputting the remote sensing image to be tested and the conditional image into the conditional target detection framework to obtain several targets in the remote sensing image that share the same interest category as the conditional image. This invention can extract same-category and rotation-invariant features from the remote sensing image and the conditional image, reducing the model's data dependency and improving its transferability.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, and in particular relates to a remote sensing image target detection method based on single-sample contrast feature changes. Background Technology

[0002] With the development of remote sensing technology over the past decade, object detection has become increasingly important as a technology closely related to many practical applications of remote sensing. The ability to effectively detect objects in remote sensing images is crucial for fields such as remote sensing-based urban traffic, large-scale precision agriculture, and geographic information systems. However, current deep learning-based remote sensing image object detection technologies are highly data-dependent; their model performance largely depends on the size of the training dataset and the diversity of images and objects. However, most current remote sensing image datasets suffer from small image sizes and a limited number of object categories. Furthermore, conventional object detection methods can only detect object categories that have appeared in the training dataset, significantly reducing the transferability and practicality of the model.

[0003] Unlike camera-captured images, remote sensing images suffer from the problem of random target rotation. The same target rotating at different angles often interferes with the model. Therefore, it is of great significance to construct a remote sensing image target detection model with low data dependence, good transferability, and minimal impact from target rotation in remote sensing images. Summary of the Invention

[0004] The purpose of this invention is to provide a remote sensing image target detection method based on single-sample contrast feature changes, so as to solve the problems existing in the prior art.

[0005] To achieve the above objectives, this invention provides a remote sensing image target detection method based on single-sample contrast feature changes, comprising:

[0006] Obtain the original remote sensing image set, and preprocess the original remote sensing image set to obtain the augmented image set;

[0007] A multi-head contrastive learning feature encoding network is established, and the augmented image set is input into the multi-head contrastive learning feature encoding network to obtain a trained feature extraction network.

[0008] Acquire a remote sensing image to be tested and a conditional image, and construct a conditional target detection framework, wherein the conditional target detection framework includes the trained feature extraction network; input the remote sensing image to be tested and the conditional image into the conditional target detection framework to obtain several targets in the remote sensing image to be tested that have the same interest category as the conditional image.

[0009] Optionally, the process of preprocessing the original remote sensing image set to obtain the augmented image set includes:

[0010] Obtain the original remote sensing image set and category labels, and based on the original remote sensing image set and the category labels, obtain the image set with category classification;

[0011] Based on the random rotation module, the image sets of each category are randomly rotated to obtain augmented image sets of each category.

[0012] Optionally, a multi-head contrastive learning feature encoding network is established. The process of training the multi-head contrastive learning feature encoding network to obtain a trained feature extraction network includes:

[0013] The augmented image set is input into the multi-head contrastive learning training feature encoding network, and the original feature space is obtained after feature extraction.

[0014] Based on the three mapping heads, the original feature space is projected to obtain several feature spaces, including: rotation-invariant feature space, same-category invariant feature space, and rotation and same-category invariant feature space.

[0015] Based on the aforementioned feature spaces, the loss value for each feature space is obtained respectively;

[0016] Based on the loss values ​​of each feature space, a total loss value is obtained. Based on the total loss value, the network is trained to obtain a trained feature extraction network.

[0017] Optionally, the conditional target detection framework includes the trained feature extraction network, the Transformer feature change module, and the detection head;

[0018] The process of target detection in the remote sensing image to be tested includes: inputting the remote sensing image to be tested and the conditional image; obtaining the features of the remote sensing image to be tested and the features of the conditional image based on the trained feature extraction network; inputting the features of the remote sensing image to be tested and the features of the conditional image into the Transformer feature transformation module to obtain the interest object enhancement features of the remote sensing image to be tested; inputting the interest object enhancement features of the remote sensing image to be tested into the detection head to obtain several targets in the remote sensing image to be tested that have the same interest category as the conditional image.

[0019] Optionally, the Transformer feature transformation module includes an encoding module and a decoding module;

[0020] The process of performing feature transformation on the remote sensing image under test based on the Transformer feature transformation module to obtain the enhanced features of the object of interest in the remote sensing image under test includes:

[0021] The conditional image features are input into the encoding module, and the encoded conditional image features are obtained based on the multi-scale deformable self-attention mechanism.

[0022] The encoded conditional image features and the remote sensing image features to be tested are input into the decoding module, and the enhanced features of the object of interest in the remote sensing image to be tested are obtained based on the multi-scale deformable mutual attention mechanism.

[0023] Optionally, the detection head includes the Faster-RCNN detection framework;

[0024] The Faster-RCNN detection framework includes: a candidate box generation layer, a RoI Pooling layer, and an R-CNN layer.

[0025] Optionally, the process of obtaining several targets in the remote sensing image to be tested that have the same interest category as the conditional image based on the Faster-RCNN detection framework includes:

[0026] The enhanced features of the object of interest in the remote sensing image to be tested are input into the candidate box generation layer, and the feature map corresponding to the candidate box is obtained based on the sliding window.

[0027] The feature map corresponding to the candidate box is input into the RoI Pooling layer to obtain a feature map of the same specification.

[0028] The feature maps of the same specifications are input into the R-CNN prediction head to obtain several targets in the remote sensing image to be tested that are of the same interest category as the conditional image.

[0029] The R-CNN layer includes a first fully connected layer and a second fully connected layer; the first fully connected layer is used to predict the object detection probability, and the second fully connected layer is used to predict the bounding box position.

[0030] The technical effects of this invention are as follows:

[0031] 1. This invention utilizes a conditional target detection model to achieve target detection in remote sensing images. The model's applicability is expanded from the target categories appearing in the training dataset to the target categories corresponding to objects of interest in the conditional image. This eliminates the need to train the model on a large-scale image dataset containing such objects and manually labeled data when it is desired to detect a certain type of object in a remote sensing image. This reduces the model's data dependency and improves its transferability.

[0032] 2. This invention utilizes multi-head contrastive learning to train the encoding network, enabling the network to extract same-category and rotation-invariant features from remote sensing images and conditional images; and utilizes a Transformer feature transformation network to enhance the features of targets in remote sensing images that are of the same category as the object of interest. Attached Figure Description

[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This invention provides a remote sensing image target detection method based on single-sample comparative feature changes.

[0035] Figure 2 This is the multi-head comparison learning module in this embodiment of the invention;

[0036] Figure 3 This is the variable multi-scale multi-head mutual attention module in the embodiments of the present invention. Detailed Implementation

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0039] Example 1

[0040] This embodiment provides a remote sensing image target detection method based on single-sample contrast feature changes. It utilizes a conditional target detection network to detect targets in the remote sensing image that belong to the same category as the object of interest in the conditional image. A contrastive learning method is used to train a feature extraction network, enabling it to output category-specific and rotation-invariant features. Target detection is then completed based on the enhanced remote sensing image features output by the Transformer. The method includes the following steps:

[0041] Step 1: Process and augment the remote sensing image data to construct a remote sensing image sequence that includes the rotated image;

[0042] Step 2: Use the augmented dataset to train the feature encoding network using a contrastive learning approach so that it outputs features of the same class and rotation invariant features;

[0043] Step 3: Utilize a conditional target detection framework to detect all targets in the remote sensing image that belong to the same category as the object of interest in the conditional image;

[0044] Step 4: Use the Transformer feature transformation module to transform the remote sensing image features based on the conditional image;

[0045] Step 5: Using the modified remote sensing image features described above, the detection head detects all targets in the remote sensing image that belong to the same category as the object of interest in the conditional image.

[0046] In step 1, a main target image dataset including rotated images is constructed using an existing remote sensing image dataset. The images in the new dataset are derived from the targets within the bounding box of the remote sensing image dataset and are scaled to a uniform size. The images in the new dataset have their own category labels, and the rotation augmentation mode corresponding to each image is obtained after random rotation operation.

[0047] In step 2, a feature encoding network is trained using multi-head contrastive learning to ensure that its outputs have class-invariance and rotation invariance, thereby generating a feature space within the encoding network. Then, three mapping heads are used to project the image feature representation into three feature spaces. The middle respectively make ,{ },{ The features of the three types of sample pairs should be as similar as possible, among which, Represents image samples, Represents the category to which the image belongs. and Represents different image indexes within a certain category. This represents the image after rotation transformation.

[0048] In step 3, a conditional target detection framework is used to perform target detection in the remote sensing image:

[0049]

[0050] in, The remote sensing image to be detected. A conditional image containing objects of interest. The detection framework used. This refers to the set of targets detected by the model.

[0051] In step 4, the Transformer feature transformation module is used to transform the remote sensing image features based on the conditional image:

[0052] )

[0053] )

[0054] in, For the encoding module in Transformer, For the decoding module in Transformer, , The conditional image and remote sensing image features are output by the feature extraction network. These are the conditional image features after encoder processing. Enhance the features of objects of interest in the remote sensing image processed by the Transformer module.

[0055] In step 5, based on remote sensing image features The detection head is used to detect targets of interest in remote sensing images.

[0056] { ... }= DeteHead ( )

[0057] in, For the detection head based on Faster-RCNN, detect the target Includes confidence level and detection frame position and range information .

[0058] Example 2

[0059] like Figure 1-3 As shown, this embodiment provides a remote sensing image target detection method based on single-sample contrast feature changes, including:

[0060] like Figure 1 The diagram shown is an overall flowchart of a remote sensing image target detection method based on single-sample contrast feature changes according to the present invention. Taking the detection of a small car in a remote sensing image as an example, the process specifically includes the following steps:

[0061] Step 1: Construct a main target image dataset using the existing remote sensing image dataset. The images in the newly constructed dataset are obtained by scaling the targets within the bounding boxes of the images in the remote sensing image dataset. Let the image set for each category be denoted as . ,in, Index the image categories in the dataset. Represents the target category of the set. This represents the target quantity of the set. To obtain the rotated images, a random rotation module is used to randomly rotate the images by 90 degrees, 180 degrees, and 270 degrees, resulting in a set of images for each type of rotation. .

[0062] Step 2: Train the feature encoding network using multi-head contrastive learning to ensure its outputs are class-invariant and rotation-invariant. This generates the feature space within the encoding network. Then, three mapping heads are used to project the image feature representation into three feature spaces. The middle will respectively include { },{ },{ } are marked as positive sample pairs, where, For each training session, an image index is used for a specific anchor. Index any other image in the same category as this image. In the three feature spaces respectively... In the feature space, similar targets should be as close as possible to each other. In the feature space, the current target and its randomly rotated augmented mode should be as close as possible. V 3. In the feature space, the random rotation augmented modes corresponding to the current target and other similar targets are made as close as possible. This is achieved by labeling the random rotation augmented modes of a certain image and other similar images as positive sample pairs, resulting in the following loss functions:

[0063]

[0064]

[0065]

[0066]

[0067] in, This serves as the index for negative samples in each training iteration. .

[0068] During the comparative learning process, an impulse mechanism is used to update the encoding and store all sample encodings. A schematic diagram of the module is shown below. Figure 2 .

[0069] Step 3: Perform target detection in remote sensing images using a conditional target detection framework:

[0070]

[0071] in, The input is the remote sensing image to be detected. A conditional image containing objects of interest. The detection framework used. Let be the set of targets detected by the model. Furthermore, based on the basic requirements of single-sample conditional object detection, let be the set of categories to which the objects of interest belong during training. During the testing process, the set of categories to which the objects of interest belonged was: Both must meet the following conditions. .

[0072] Step 4: The Transformer feature transformation module transforms the remote sensing image features based on the conditional image to obtain a remote sensing image feature map that highlights only objects of interest within the same category as those in the conditional image. This invention uses a classic Transformer network containing Encoder and Decoder modules to implement feature transformation.

[0073] )

[0074] )

[0075] in, For the encoding module in Transformer, For the decoding module in Transformer, , The conditional image and remote sensing image features are output by the feature extraction network. These are the conditional image features after encoder processing. Enhance the features of objects of interest in the remote sensing image processed by the Transformer module.

[0076] For the Encoder module, the input is the multi-scale features of the remote sensing image. To obtain the multi-scale features, this invention first uses the output layer of the ResNet50 feature extraction network. The feature is convolved with a kernel size of 1. A convolution operation with a stride of 2 produces a new output layer, denoted as . and will to The layer serves as the multi-scale input to the Encoder, denoted as . Employing a deformable multi-scale attention mechanism, the feature map is first linearly transformed to obtain a multi-head... The diagram illustrates that when performing sparse attention operations on points in a feature map, the relative position of each point is first used as a multi-head reference at various scales. The reference point in the graph is used to calculate the sparse attention operation for several points based on its feature values. The offsets and weights on the graph ultimately link each scale to the head. Value integration completes the attention operation. (This is done using any point on the feature map.) For example, the multi-scale deformable self-attention operation is as follows:

[0077]

[0078] In the above formula, The eigenvectors on the feature map correspond to the classic Transformer structure. Used to calculate the participants in this attention operation Relative to reference point offset and their weights , satisfy . For each The mapping function from the reference position to each actual position on the graph. For feature maps to Feature transformation of the graph The connection matrix is ​​the output of the attention operation for each head. It is an index for each attention head. Indicates the number of attention heads. For each index, The total number of features involved in each attention operation.

[0079] During the decoding stage, the same operation is performed on the remote sensing image features output by the ResNet50 network to obtain... After the layer, to Layer features are input into the Decoder. During the decoding stage, there is simultaneous self-attention between multi-scale features of the remote sensing image and mutual attention between multi-scale features of the remote sensing image and the encoder output. The implementation of self-attention is the same as in the Encoder. When performing mutual attention operations... Generated from feature encoding of remote sensing images, The image is obtained through a linear transformation of the encoder's output, and then through... get The offset of the points participating in the mutual attention operation on the graph and weight Deformable mutual attention operations, such as Figure 3 As shown.

[0080] Step 5: Based on remote sensing image features This invention uses a detection head to detect targets of interest in remote sensing images. It employs Faster-RCNN as the detection framework, which mainly includes a Region Proposal Network (RPN) layer, an RoI Pooling layer, and an R-CNN probability prediction and bounding box regression prediction head.

[0081] The remote sensing image transformation features output by the Transformer are used as input to the RPN layer. After generating anchors through a sliding window, the anchors with the largest IoU and those with IoU greater than 0.7 corresponding to each actual bounding box are marked as positive samples, and those with IoU less than 0.3 are marked as negative samples.

[0082]

[0083] in, This is a label indicating whether the Anchor is a positive sample.

[0084] The optimized loss function for the bounding box and score is:

[0085]

[0086] in, This refers to the index of the bounding box in the mini-batch. For Anchor, It includes the probability score of the target. The coordinates of the Anchor. These are the actual bounding box coordinates corresponding to the Anchor.

[0087] Next, the feature maps corresponding to the candidate boxes output by the RPN are input into the RoI Pooling layer to unify their size before being fed into the R-CNN prediction head. The R-CNN layer flattens the features and then predicts the bounding box positions and probabilities through two fully connected layers. The fully connected layer used for probability prediction has two units corresponding to the object to be detected, other object classes, and the background, respectively. The fully connected layer used for bounding box position prediction has four units corresponding to the four coordinates of the bounding box.

[0088] This embodiment uses DIOR, a large-scale benchmark dataset for target detection in optical remote sensing images proposed by Northwestern Polytechnical University, as the implementation sample. It contains a total of 23,463 remote sensing images and 190,288 target instances. Based on the single-sample conditional target detection requirements, the entire dataset is divided into four subsets: 30% for contrastive learning pre-training, 50% as the training set, 10% as the test set, and 10% as the validation set. The proposed method is used to train and test the model.

[0089] In this embodiment of the invention, the mean precision (mAP) is used as the evaluation metric for the detection performance of the model. mAP is obtained by calculating the area under the precision and recall curves for different IoU thresholds. The precision (Precious) is calculated using the following formula:

[0090]

[0091] in, The number of correctly labeled sample frames, The number of incorrectly labeled sample frames. This represents the total number of sample boxes output by the model.

[0092] The formula for calculating recall is:

[0093]

[0094] in, This represents the number of targets to be detected that were not selected by the sample frame. This represents the actual number of targets to be detected in the remote sensing image.

[0095] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A remote sensing image target detection method based on single-sample contrast feature changes, characterized in that, Includes the following steps: Obtain the original remote sensing image set, and preprocess the original remote sensing image set to obtain the augmented image set; A multi-head contrastive learning feature encoding network is established, and the augmented image set is input into the multi-head contrastive learning network. The process of training a feature encoding network to obtain a trained feature extraction network includes: inputting the augmented image set into the multi-head contrastive learning training feature encoding network, extracting features to obtain an original feature space; projecting the original feature space onto three mapping heads to obtain several feature spaces, including: rotation-invariant feature space, class-invariant feature space, and rotation and class-invariant feature space; obtaining the loss value of each feature space based on the several feature spaces; obtaining the total loss value based on the loss values ​​of each feature space; and training the network based on the total loss value to obtain a trained feature extraction network. The rotation-invariant feature space is the feature space of the current target and its class-like targets; the class-invariant feature space is the feature space of the current target and its rotation augmented modal; and the rotation and class-invariant feature space is the feature space of the current target and its class-like target rotation augmented modal. Acquire a remote sensing image to be tested and a conditional image, and construct a conditional target detection framework, wherein the conditional target detection framework includes the trained feature extraction network; input the remote sensing image to be tested and the conditional image into the conditional target detection framework to obtain several targets in the remote sensing image to be tested that have the same interest category as the conditional image.

2. The remote sensing image target detection method based on single-sample contrast feature changes according to claim 1, characterized in that, The process of preprocessing the original remote sensing image set to obtain the augmented image set includes: Obtain the original remote sensing image set and category labels, and based on the original remote sensing image set and the category labels, obtain the image set with category classification; Based on the random rotation module, the image sets of each category are randomly rotated to obtain augmented image sets of each category.

3. The remote sensing image target detection method based on single-sample contrast feature changes according to claim 1, characterized in that, The conditional target detection framework includes the trained feature extraction network, the Transformer feature transformation module, and the detection head; The process of performing target detection on the remote sensing image to be tested includes: inputting the remote sensing image to be tested and the conditional image, and obtaining the features of the remote sensing image to be tested and the features of the conditional image based on the trained feature extraction network; The features of the remote sensing image to be tested and the features of the conditional image are input into the Transformer feature transformation module to obtain the interest object enhancement features of the remote sensing image to be tested; the interest object enhancement features of the remote sensing image to be tested are input into the detection head to obtain several targets in the remote sensing image to be tested that have the same interest category as the conditional image.

4. The remote sensing image target detection method based on single-sample contrast feature changes according to claim 3, characterized in that, The Transformer feature transformation module includes an encoding module and a decoding module; The process of performing feature transformation on the remote sensing image under test based on the Transformer feature transformation module to obtain the enhanced features of the object of interest in the remote sensing image under test includes: The conditional image features are input into the encoding module, and the encoded conditional image features are obtained based on the multi-scale deformable self-attention mechanism. The encoded conditional image features and the remote sensing image features to be tested are input into the decoding module, and the enhanced features of the object of interest in the remote sensing image to be tested are obtained based on the multi-scale deformable mutual attention mechanism.

5. The remote sensing image target detection method based on single-sample contrast feature changes according to claim 4, characterized in that, The detection head includes the Faster-RCNN detection framework; The Faster-RCNN detection framework includes: a candidate box generation layer, a RoI Pooling layer, and an R-CNN layer.

6. The remote sensing image target detection method based on single-sample contrast feature changes according to claim 5, characterized in that, The process of obtaining several targets in the remote sensing image to be tested that have the same interest category as the conditional image based on the Faster-RCNN detection framework includes: The enhanced features of the object of interest in the remote sensing image to be tested are input into the candidate box generation layer, and the feature map corresponding to the candidate box is obtained based on the sliding window. The feature map corresponding to the candidate box is input into the RoI Pooling layer to obtain a feature map of the same specification. The feature maps of the same specifications are input into the R-CNN prediction head to obtain several targets in the remote sensing image to be tested that are of the same interest category as the conditional image. The R-CNN layer includes a first fully connected layer and a second fully connected layer; the first fully connected layer is used to predict the object detection probability, and the second fully connected layer is used to predict the bounding box position.

Images