A Weakly Supervised Target Detection Method for Remote Sensing Images with Optimized Feature Extraction

Abstract

A method for weakly supervised object detection in remote sensing images with optimized feature extraction is provided, comprising the following steps: S1: Obtaining a weakly supervised object detection dataset of remote sensing images, wherein all images are labeled at the image level; S2: Training an image multi-classification model using the remote sensing image weakly supervised object detection dataset with image level labels to provide pre-training weight parameters for the weakly supervised object detection model; S3: Constructing a remote sensing image feature refinement module and integrating it into the weakly supervised object detection model; S4: Initializing the parameters of the backbone network of the weakly supervised object detection model using the pre-training weight parameters obtained in step S2, and training the weakly supervised object detection model; S5: Providing the image to be detected to the trained weakly supervised object detection model, wherein the trained weakly supervised object detection model outputs the object detection result for the image to be detected.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to a method for target detection in remote sensing images under weakly supervised learning. Background Technology

[0002] Object detection is a popular area in computer vision and digital image processing. It enables the automatic detection of object categories and locations, reducing the consumption of manpower and capital, and has significant practical implications. Compared to more basic image classification tasks, object detection involves an additional regression task: it not only needs to use algorithms to determine the presence of objects in an image but also to mark their locations within the image, performing regression prediction on the marked bounding boxes. In recent years, due to the widespread application of deep learning, object detection algorithms have developed rapidly.

[0003] However, typical fully supervised object detection models are trained on large datasets with precise manual annotations. These methods require each training image to have fine-grained, high-quality annotations. In remote sensing images, there are often multiple object instances in a single image, each potentially belonging to different categories, all requiring manual annotation. Some instances are even difficult to discern with the naked eye due to their small size, environmental influences, or image distortion, further increasing the difficulty and cost of dataset annotation. Therefore, to reduce the resource consumption associated with annotating large-scale remote sensing object detection datasets, researchers have begun to focus on how to achieve good performance in object detection tasks using more easily obtainable and annotated coarse-grained labels. A typical coarse-grained task setting is where the dataset labels only have image-level category information, lacking instance-level location information. This setting allows for direct research using existing image classification datasets and also enables rapid annotation on new datasets, significantly reducing the annotation cycle and accelerating deployment. Currently, coarse-grained datasets with only image-level annotations have become a major focus of weakly supervised object detection research.

[0004] However, due to the characteristics of remote sensing images, such as high background noise, dense instances, numerous small targets, and arbitrary orientations, current conventional weakly supervised target detection methods often fail to achieve ideal feature extraction results, and their final detection performance still lags significantly behind that of fully supervised target detection methods. Therefore, further research on weakly supervised target detection methods for remote sensing images and optimization of their feature extraction is of great significance. Summary of the Invention

[0005] This invention addresses the problem of poor feature extraction performance in weakly supervised target detection of remote sensing images by designing an optimized feature extraction method for weakly supervised target detection in remote sensing images.

[0006] The specific contents of this invention are as follows:

[0007] A method for weakly supervised target detection in remote sensing images with optimized feature extraction includes the following steps:

[0008] The first step is to obtain a remote sensing image weakly supervised target detection dataset, in which all images are labeled at the image level;

[0009] The second step is to train an image multi-classification model using image-level labels, which will provide pre-training weight parameters for the weakly supervised object detection model.

[0010] The third step is to construct a remote sensing image feature refinement module and integrate it into the weakly supervised target detection model;

[0011] The fourth step is to initialize the backbone network parameters of the weakly supervised object detection model using the pre-trained weights obtained in step S2, and to improve feature extraction using the remote sensing image feature refinement module, and to train the weakly supervised object detection network.

[0012] In step two, the ResNet network is used for image multi-class classification training. Compared with the usual multi-class classification training, the training needs to be stopped earlier and only two rounds are required to reduce overfitting and facilitate further fine-tuning in the subsequent weakly supervised object detection task.

[0013] In step three, the integrated weakly supervised object detection model structure includes a backbone network, a remote sensing image feature refinement module, a candidate region processing module, and a detector head. The backbone network extracts image features, generates feature maps, and feeds these feature maps into the remote sensing image feature refinement module to optimize feature extraction. Subsequent processing and prediction operations are then performed on the optimized feature maps. The backbone network adopts the same structure as the backbone network for classification tasks, such as ResNet, so the backbone network parameters obtained from pre-training in step two can be used directly. The candidate region processing module normalizes the features of the pre-extracted candidate boxes to obtain fixed-length feature vectors. The detector head feeds the fixed-length feature vectors into two branches: classification prediction and regression prediction, and outputs the final predicted instance location, size, and category information. Classification prediction is implemented using a fully connected layer with an output dimension equal to the number of categories, where each dimension represents the prediction probability of each category. Regression prediction is implemented using a fully connected layer with an output dimension of 4, where the four dimensions are (x, y, w, h), corresponding to the predicted x and y coordinates and the width and height parameters w and h, respectively.

[0014] In step three, the remote sensing image feature refinement module consists of a channel-level attention module and a spatial-level attention module; the channel-level attention module can be represented as...

[0015] G c(F)=σ(LN1(AvgPool(F))+LN2(MaxPool(F)))

[0016] Where F is the input feature map to be optimized, AvgPool represents global average pooling, MaxPool represents max pooling, LN1 and LN2 each represent a set of linear layers for feature mapping, and σ represents the activation function that maps values ​​to a continuous interval of 0 to 1; the spatial hierarchical attention module can be represented as...

[0017] G s (F)=σ(Conv 7 ([AvgPool(F);MaxPool(F)]))

[0018] Among them, Conv 7 This indicates a convolutional layer using a 7x7 kernel.

[0019] Compared with the prior art, the present invention has the following innovative features:

[0020] 1. To address the problem of poor feature extraction performance in remote sensing images, two optimization strategies were designed.

[0021] 2. It fully incorporates the characteristics of weakly supervised tasks, namely, only image-level labels, and utilizes the development in the field of image classification to generate pre-trained weights for training of downstream remote sensing weakly supervised target detection tasks.

[0022] 3. A remote sensing image feature refinement module was designed to address the characteristics of high noise levels and small target size in remote sensing images. This module can effectively remove background noise and optimize the feature extraction effect for small targets.

[0023] The method execution flow provided in the embodiments of this application can be run on devices such as personal computers, servers, embedded computing devices, and cloud computing platforms. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0025] Figure 1 This is a flowchart of a remote sensing image weakly supervised target detection method with optimized feature extraction according to the present invention.

[0026] Figure 2 The image shows a comparison of feature heatmaps on the DOTA dataset. The left image is the original image, the middle image is the heatmap without this method, and the right image is the heatmap with this method. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0028] According to an embodiment of this application, the data used is derived from the DOTA remote sensing image dataset (https: / / captain-whu.github.io / DOTA / dataset.html).

[0029] 1. Obtain the DOTA remote sensing image dataset. Remove instance annotations from the training set, retaining only image-level labels.

[0030] 2. Using ResNet as the backbone network, a multi-classification model of remote sensing images is trained using image-level labels; the iterative training is stopped early after two training rounds, which can provide the remote sensing image pre-training weight parameters of the ResNet backbone network for the weakly supervised object detection model, and will not make the model too focused on the prominent features of objects, which is convenient for the localization of object detection tasks.

[0031] 3. Construct a remote sensing image feature refinement module, consisting of a channel-level attention module and a spatial-level attention module, wherein the channel-level attention module is in the form of G. c (F) = σ(LN1(AvgPool(F)) + LN2(MaxPool(F))), where the feature map F is processed by global average pooling and max pooling, and then fed into two linear mapping layers. Finally, a sigmoid activation function is used to generate a channel-wise mask between 0 and 1. The spatial level attention module is in the form of G. s (F)=σ(Conv 7 ([AvgPool(F);MaxPool(F)])), the feature map F is concatenated together in the last dimension after global average pooling and max pooling, and then after a 7*7 convolutional layer, the sigmoid activation function is used to generate a mask between 0 and 1 in the spatial plane.

[0032] 4. The remote sensing image feature refinement module is integrated into the weakly supervised object detection model. The integrated weakly supervised object detection model includes a backbone network, a remote sensing image feature refinement module, a candidate region processing module, and a detector head. The backbone network adopts the same ResNet50 structure as the backbone network for the classification task (https: / / arxiv.org / abs / 1512.03385); the candidate region processing uses RoI Align (https: / / arxiv.org / pdf / 1703.06870) to normalize the features of the pre-extracted candidate boxes, obtaining a fixed-length 128-dimensional feature vector; the detector head uses a multi-instance learning detector head, which includes a classification branch and a detection branch. It predicts and normalizes the fixed-length feature vector to obtain classification and detection scores, multiplies them to obtain image-level label information, and calculates the cross-entropy loss between the image-level labels of the training data.

[0033] 5. Initialize the backbone network of the weakly supervised object detection model using the ResNet50 weights obtained in step 2, and train it using a model that integrates a remote sensing image feature refinement module. Iterate training until the model converges on the validation set. In step 4, both the trained weakly supervised object detection model and the multi-class model trained in step 2 use image-level labeled training data.

[0034] Experimental results on the DOTA remote sensing image dataset show that the detection performance after applying this method is improved by 2.1% compared to the previous method. The feature maps are visualized as follows: Figure 2 As shown, the middle example is the feature extraction effect of the original weakly supervised object detection method, and the right side is the feature extraction effect after applying this method. It can be seen that the feature extraction capability of the model can be significantly improved, achieving the expected results.

[0035] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application. Clearly, those skilled in the art can make various alterations and variations to this application without departing from its spirit and scope. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for weakly supervised object detection of remote sensing images with optimized feature extraction, characterized in that, The method comprises the following steps: S1: obtaining a remote sensing image weakly supervised target detection dataset, wherein the labels of all images are image-level labels; S2: training an image multi-classification model using the remote sensing image weakly supervised target detection dataset with image-level labels to provide pre-training weight parameters of the weakly supervised target detection model; S3: constructing a remote sensing image feature refinement module integrated into the weakly supervised target detection model; the structure of the integrated weakly supervised target detection model comprises a backbone network, a remote sensing image feature refinement module, a candidate region processing module and a detection head; the remote sensing image feature refinement module comprises a channel-level attention module and a spatial-level attention module; the backbone network extracts image features from an input image to generate a feature map, and the feature map is sent to the remote sensing image feature refinement module to optimize feature extraction, and subsequent processing and prediction operations are performed on the optimized feature map; the backbone network adopts the same network structure as the image multi-classification model; the candidate region processing module normalizes the features of pre-extracted candidate boxes to obtain fixed-length feature vectors, and provides the fixed-length feature vectors to the detection head; the detection head sends the fixed-length feature vectors to a classification prediction branch and a regression prediction branch respectively to output the position, size and category information of the final predicted target instance; the classification prediction branch uses a fully connected layer with an output dimension of the number of categories to implement, and each dimension represents the prediction probability of each category; the regression prediction branch uses a fully connected layer with an output dimension of 4 to implement, and the four dimensions are (x, y, w, h), which correspond to the predicted x, y coordinates and width and height w, h parameters; S4: initializing the parameters of the backbone network of the weakly supervised target detection model using the pre-training weight parameters obtained in step S2, improving feature extraction of the feature map output by the backbone network using the remote sensing image feature refinement module, and training the weakly supervised target detection model; S5: providing a to-be-detected image to the trained weakly supervised target detection model, and the trained weakly supervised target detection model outputs a target detection result of the to-be-detected image.

2. The method of claim 1, wherein, The model for training image multi-classification in step S2 is trained using a ResNet network for image multi-classification tasks, wherein only 2 rounds of training are performed to stop training in advance to reduce overfitting.

3. The method of claim 2, wherein, The output of the channel-level attention module is , ； Wherein, F is the feature map to be optimized input to the remote sensing image feature refining module, AvgPool represents global average pooling, MaxPool represents maximum pooling, , Each represents a set of linear layers for feature mapping, represents an activation function that maps values to the continuous interval of 0~1, and the output generated by the channel-level attention module is multiplied by the original feature map as a weight and sent to the spatial-level attention module for further refinement to obtain a weighted feature map ; the output of the spatial-level attention module is , ； wherein, represents a convolutional layer with a 7*7 size convolution kernel, and the spatial level attention module outputs are regarded as weights and are multiplied by ; the remote sensing image feature refining module outputs a pre-extraction candidate frame.

4. The method of claim 3, wherein, The detection head comprises a classification branch and a detection branch, which predict and normalize the fixed-length feature vectors output by the candidate region processing module to obtain classification scores and detection scores, multiply the classification scores and the detection scores to obtain image-level label information, and calculate the cross-entropy loss between the image-level label information and the image-level label.

5. An information processing apparatus comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein The computer program is executed by the processor to implement the method of any one of claims 1 to 4.

Images