A self-supervised pre-training method suitable for remote sensing target detection tasks

Abstract

This invention relates to a self-supervised pre-training method suitable for remote sensing target detection tasks, belonging to the technical field of pre-training for remote sensing target detection tasks. The method uses the real target bounding boxes in the detection task dataset as a basis for random multi-scale target cropping. Small-scale targets in the cropped target-level image slices are rescaled and stitched together to ensure that small-scale target information is effectively preserved in high-occlusion-rate image reconstruction tasks, thereby improving the detection performance of small-scale targets. This invention effectively avoids the problem in high-occlusion-rate scene-level image reconstruction tasks where small-scale targets are completely occluded, preventing them from obtaining good representation learning during pre-training and thus affecting detection performance. Furthermore, by aligning the image reconstruction task with the target detection task, this method enhances the promoting effect of the pre-trained model on remote sensing target detection tasks.

Description

Technical Field

[0001] This invention relates to a pre-training method for image reconstruction tasks, specifically a self-supervised pre-training method suitable for remote sensing target detection tasks, belonging to the field of pre-training technology for remote sensing target detection. Background Technology

[0002] Pre-training methods for image reconstruction tasks have made rapid progress in computer vision. These methods can acquire task-independent generalized representations through self-supervised model pre-training on large amounts of unlabeled data. Specifically, image reconstruction involves randomly dividing an image into several patches, randomly masking some of these patches according to a set masking ratio, and then training an autoencoder to reconstruct the pixel information of the masked portions. Image reconstruction has proven to be an excellent self-supervised pre-training proxy task in natural scenes. The representations acquired through this proxy task can be effectively transferred to various downstream tasks and improve fine-tuning performance.

[0003] Given the significant success of image reconstruction technology in natural scenes, some researchers have begun applying image reconstruction-based pre-training methods to the field of remote sensing. This includes using image reconstruction tasks to train base models with extremely large-scale parameters on unlabeled remote sensing datasets to improve the performance of various downstream tasks in remote sensing. Additionally, some researchers have attempted to prepare extremely large datasets for pre-training image reconstruction-based models by collecting remote sensing image data from various data sources. This ensures that the models learn data knowledge encompassing diverse remote sensing scenes during the pre-training phase, enabling compatibility with various downstream tasks. Furthermore, some researchers have proposed a cross-domain knowledge transfer method. This method aims to transfer knowledge learned from existing large-scale natural scene datasets to various downstream remote sensing tasks through continuous pre-training, avoiding the tedious work of collecting and organizing large-scale remote sensing datasets. This approach is simple and effective.

[0004] However, current image reconstruction tasks in the remote sensing field are all based on scene-level image data. When the random masking rate of the image reconstruction task is set high, the complex details of the remote sensing scene and the densely arranged small-scale targets are easily lost during the masking process. This makes it impossible to learn the corresponding target-level representation knowledge through the reconstruction task during the pre-training stage. Obviously, this creates a serious task misalignment problem with the detection task that requires classifying and locating remote sensing targets of various scales, thus limiting the role of the pre-trained model in promoting downstream detection tasks and affecting detection performance. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings and defects of existing technologies, and to solve the problem that the pre-training process for scene-level image reconstruction cannot learn target-level representation knowledge, thus limiting the performance improvement of the pre-trained model on downstream detection tasks. This invention creatively proposes a self-supervised pre-training method suitable for remote sensing target detection tasks, thereby further improving the performance of detection tasks.

[0006] The method of the present invention is achieved through the following technical solution.

[0007] A self-supervised pre-training method suitable for remote sensing target detection tasks, the method comprising the following steps:

[0008] Step 1: Using the image reconstruction task, perform self-supervised pre-training of the autoencoder network based on the Vision Transformer feature extraction network on a natural scene dataset with millions of data points to obtain model parameters with domain-level generalization knowledge.

[0009] The autoencoder based on the Vision Transformer feature extraction network consists of two parts: an encoder and a decoder. The encoder uses a 12-layer Transformer Block network structure with a multi-head attention mechanism, while the decoder uses an 8-layer Transformer Block network structure with a multi-head attention mechanism.

[0010] Step 2: Perform random multi-scale target cropping based on the real target bounding boxes in the downstream target detection task dataset to obtain target-level task-related image data;

[0011] Step 3: Based on the target-level task-related image data obtained in Step 2, first divide the image data into small-scale target image data and non-small-scale target image data according to the image size. Then, uniformly rescale the small-scale target image data and perform random stitching operation.

[0012] Step 4: On the non-small-scale target image data obtained in Step 3 and the small-scale target image data after random stitching, sample resampling is performed to address the imbalance of class samples in the multi-class target detection dataset, thereby achieving a balance of multi-class samples.

[0013] Step 5: Use the non-small-scale target image data and small-scale target image data after sample resampling in Step 4 to update the model parameters with domain-level generalization knowledge in Step 1, and complete the self-supervised pre-training suitable for remote sensing target detection tasks.

[0014] In step 2, the method for random multi-scale target cropping is as follows:

[0015] Step 2.1: Based on the ground truth annotations of the training set of the object detection task dataset, obtain the center point coordinates (x, y), target width w, and height of all bounding boxes;

[0016] Step 2.2: Using the center point (x,y) of the target as a reference, the target width w and height are randomly scaled by a factor of α to obtain a rectangular region. The obtained rectangular region is then cropped to obtain target-level image data. The specific formula is shown in the following formula (1):

[0017]

[0018] In equation (1), I scene This represents the original remote sensing image at the scene level; N represents the number of true bounding boxes on this scene-level image. Indicates the coordinates of the center point of the i-th annotation box; w i and i Let represent the width and height of the i-th annotation box; α represents the random scaling factor, which ranges from [3,5]; I object This represents the final target-level image data; Crop(·) represents the cropping operation; i = 1, 2, 3, ..., N;

[0019] In step 3, the method for dividing the target image data into small-scale and non-small-scale target image data based on image size is as follows:

[0020] Step 3.1: Filter small-scale target image data;

[0021] First, set a size threshold β for small-scale target images, i.e., β = 80×80 resolution. Based on the size threshold β, filter the target-level image data obtained in step 2. Use non-small-scale target image data larger than the threshold directly for processing in step 4; process small-scale target image data smaller than the threshold in steps 3.2 and 3.3.

[0022] The method for uniformly rescaling and randomly stitching small-scale target image data is as follows:

[0023] Step 3.2: Rescale the small-scale target.

[0024] Because the small-scale target image data is too small to meet the input size of the pre-training network during the pre-training process, it is rescaled. At the same time, considering that pixel distortion will occur when interpolating small targets to a resolution of 224×224, in order to avoid the distortion problem affecting the pre-training process of the image reconstruction task, the small-scale target image data is uniformly scaled to a moderate scale, namely 112×112 resolution.

[0025] Step 3.3: Perform random stitching on the rescaled small-scale targets.

[0026] Because small-scale targets have small width and height values, even with multi-scale context information acquisition using the α random scaling factor in step 2.2, the context information for small-scale targets is still limited. Therefore, in order to further enrich the context information of small targets, and considering how to reasonably adapt the 112×112 resolution to the 224×224 input resolution of the pre-training process, a random stitching method is adopted. Specifically, four images of the same category are randomly selected and stitched together in a 2x2 row and 2x2 column combination.

[0027] In step 4, the method for sample resampling is as follows:

[0028] Because multi-class object detection datasets suffer from severe class imbalance, directly referencing the ground truth bounding boxes to obtain object-level image data does not eliminate this imbalance. This imbalance can easily lead to overlearning of certain classes with a high proportion of data during pre-training, resulting in inappropriate representation learning. Therefore, to avoid the impact of data distribution on the pre-training process, data resampling is used to balance the class samples. Specific steps include:

[0029] Step 4.1: Based on the obtained target-level image data, count the number of image data in each category and calculate the average number of samples in each category, as shown in the following formula (2).

[0030]

[0031] In equation (2), C represents the number of categories; N j N represents the number of target-level images in the j-th category; avq Represents the average value of the category samples, j = 1, 2, 3, ..., C;

[0032] Step 4.2: For quantities greater than N avg Perform sample downsampling on categories with fewer than N; avg Oversampling is performed on each category to ensure that the number of samples in each category remains balanced.

[0033] Beneficial effects

[0034] Compared with existing technologies, this method has the following advantages:

[0035] (1) This method can effectively solve the problem that the pre-trained model's performance improvement on downstream detection tasks is limited due to the task misalignment between existing scene-level image reconstruction tasks and target-level detection tasks. In particular, it solves the problem that dense small targets in remote sensing target detection tasks are easily completely occluded in image reconstruction tasks, resulting in the inability to be reconstructed. It ensures that the information of small target objects is effectively learned in the pre-training stage, thereby improving the detection performance of small-scale targets.

[0036] (2) This method can significantly improve the performance of remote sensing target detection tasks. At the same time, compared with the pre-training method of collecting large-scale datasets, this method is simple and effective and has great practical application value.

[0037] (3) This invention relates to a self-supervised pre-training method suitable for remote sensing target detection tasks, belonging to the technical field of pre-training for remote sensing target detection tasks. First, in order to effectively learn target-level image information with multi-scale contextual content, this method performs random multi-scale target cropping based on the real target bounding boxes in the detection task dataset. Then, the small-scale targets in the cropped target-level image slices are rescaled and stitched together to ensure that the small-scale target information is effectively preserved in image reconstruction tasks with high masking rates, thereby improving the detection performance of small-scale targets. In addition, considering the problem of class imbalance in the cropped target-level image slices, this method further resamples the target-level image slices by category to ensure that the representation information of each category of targets is learned fairly and reasonably during the pre-training process based on the image reconstruction task. In summary, this method effectively avoids the problem in scene-level image reconstruction tasks with high occlusion rates where small-scale targets in remote sensing scenes are completely occluded, preventing them from obtaining good representation learning during pre-training and thus affecting detection performance. Furthermore, by aligning the image reconstruction task with the target detection task, this method enhances the role of the pre-trained model in promoting remote sensing target detection. Attached Figure Description

[0038] Figure 1 This is the overall flowchart of this method;

[0039] Figure 2 This is a schematic diagram of the preprocessing process for target-level images;

[0040] Figure 3 This is a schematic diagram illustrating model fine-tuning in an object detection task. Detailed Implementation

[0041] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0042] The method of the present invention firstly employs a multi-scale target random cropping method to obtain target-level image slices, thereby effectively learning target-level representation information containing contextual content at different scales during the pre-training stage.

[0043] Then, a rescaling and stitching method is used for the small targets to ensure that small-scale target information can be effectively captured and learned under the high masking rate of the image reconstruction task. In addition, the stitching operation not only avoids the pixel distortion problem of small target images, but also further enriches the contextual information of small targets.

[0044] Finally, the samples in the multi-class dataset were resampled, which effectively solved the problem that the severely imbalanced number of classes caused overlearning of the representation knowledge of some class samples during the pre-training process.

[0045] Example

[0046] A self-supervised pre-training method suitable for remote sensing target detection tasks, the overall process is as follows: Figure 1 As shown, the specific steps include:

[0047] Step 1: Use an autoencoder based on the Vision Transformer network structure to pre-train on the ImageNet dataset, which contains millions of data points of natural scenes, for a self-supervised agent task of image reconstruction. Save the model from this training process, denoted as W. N .

[0048] Step 2: Perform random multi-scale target cropping based on the real target bounding boxes of a specific downstream target detection task dataset to obtain target-level task-related image data.

[0049] This invention provides a specific implementation method, such as... Figure 2 As shown in (a), the steps include:

[0050] Step 2.1: Based on the target annotation information in the training set of the target detection task dataset, obtain the center point coordinates (x, y), target width w, and height of all bounding boxes.

[0051] Step 2.2: Using the center point (x,y) of the target as a reference, randomly scale the target width W and height h by a factor of α, and then crop this area to obtain target-level image data. The specific formula is shown in the following formula (1):

[0052]

[0053] In Equation 1, I scene Represents the original scene-level image; N represents the number of true bounding boxes on this scene-level image; Indicates the coordinates of the center point of the i-th annotation box; w i and h i Let represent the width and height of the i-th annotation box; α represents the random scaling factor, which ranges from [3,5]; I object This represents the final target-level image data; Crop(·) represents the cropping operation; i = 1, 2, 3, ..., N;

[0054] Step 3: Based on the target-level image data obtained in Step 2, divide it into small-scale targets and non-small-scale targets according to the image size. Then, uniformly scale the small-scale target image data and perform random stitching.

[0055] This invention provides a specific implementation method, such as... Figure 2 As shown in (b) and (c), the steps include:

[0056] Step 3.1: Filter small-scale target image data.

[0057] First, set a size threshold β for small-scale target images, i.e., β = 80×80 resolution. Based on the size threshold β, filter the target-level image data obtained in step 2. Use non-small-scale target image data larger than the threshold for processing in step 4; process small-scale target image data smaller than the threshold in steps 3.2 and 3.3.

[0058] Step 3.2: Rescale the small-scale target.

[0059] Because the small-scale target image data is too small to meet the input resolution of the pre-training network during the pre-training process, it is rescaled. At the same time, considering that pixel distortion will occur when interpolating small targets to the pre-training input resolution, the small-scale target image data is uniformly scaled to a moderate resolution to avoid the distortion problem affecting the image reconstruction pre-training process.

[0060] The following example illustrates this:

[0061] First, the input size in the pre-training stage is 224×224 resolution. In order to avoid pixel distortion problems caused by directly rescaling small-scale target image data with a resolution of less than 80×80 to 224×224 resolution, the small-scale target images are scaled to a moderate resolution of 112×112.

[0062] Step 3.3: Perform image stitching on the rescaled small-scale target image.

[0063] Because small-scale targets have small width and height values, even with multi-scale context information acquisition using the α random scaling factor in step 2.2, the context information for small-scale targets is still limited. To enrich the context information of small-scale targets, and considering the need to reasonably transform the 112×112 resolution image to the 224×224 pre-training input resolution, a random stitching method was adopted.

[0064] The following example illustrates this:

[0065] First, the small-scale target image data obtained in step 3.2 is divided into categories. Within each category, four target-level images are randomly selected and stitched together to form a 2x2 composite image with a resolution of 224×224. The reason for stitching within categories is to avoid learning unreasonable contextual information during pre-training due to significant differences in context between different categories. For example, stitching ship and aircraft targets together would result in the ship appearing in an unreasonable airport area, thus affecting representation learning during the pre-training stage.

[0066] Step 4: Based on the target-level image data obtained in Step 3, sample resampling is performed to address the class imbalance problem in the multi-class target detection dataset, so as to achieve class sample balance during the pre-training process.

[0067] Because multi-class object detection datasets suffer from severe class imbalance, directly referencing the ground truth bounding boxes to obtain object-level image data does not eliminate this imbalance. This imbalance can easily lead to overlearning of certain classes with a high proportion of data during pre-training, resulting in inappropriate representation learning. Therefore, to avoid the impact of data distribution on the pre-training process, data resampling is employed to balance the class samples.

[0068] This invention provides a specific implementation method, such as... Figure 2 As shown in (d), the steps include:

[0069] Step 4.1: On the target-level image data obtained in Step 2 and Step 3, count the number of image data of each category and calculate the average number of samples, as shown in the following formula (2).

[0070]

[0071] In equation (2), C represents the number of categories; N j N represents the number of target-level images in the j-th category; avg This represents the average value of the category samples. j = 1, 2, 3, ..., C

[0072] Step 4.2: For quantities greater than N avgPerform sample downsampling on categories with fewer than N; avg Oversampling is performed on each category to ensure that the number of samples in each category remains balanced.

[0073] Step 5: While maintaining the same pre-training method as in Step 1, train the pre-trained model W from Step 1... N Continue loading, and then perform a second stage of pre-training on the target-level image data processed in step 4 to obtain the pre-trained model W. R .

[0074] Since the second-stage pre-training process and the model fine-tuning process in step 6 are adjacent, in this invention, the target-level image reconstruction pre-training method is only used for the second-stage pre-training process.

[0075] Furthermore, since the pre-training methods used in steps 1 and 5 are the same, this invention provides a specific implementation method:

[0076] Step 5.1: Perform non-overlapping slicing on the image, and then perform random masking.

[0077] First, the input size of the Vision Transformer-based autoencoder network is specified as 3×224×224.

[0078] Then, the input image is passed through a convolutional layer with a kernel size of 16×16, a stride of 16, and an output feature dimension of 768 to obtain a feature layer with a resolution of 14×14×768.

[0079] Secondly, the feature layer is expanded into a 196×768 feature vector in the spatial dimension by using the Flatten operation.

[0080] Finally, by randomly masking 75% of the 196 dimensions using random numbers, the features of the unmasked portion are obtained, with dimensions of 49×768, denoted as X. ～M .

[0081] Step 5.2: Use the Vision Transformer as the encoder, and then patch the unmasked areas using the X block. ～M The input is fed into the encoder part of the autoencoder, and after passing through the encoder, the intermediate implicit feature R is obtained.

[0082] The following example illustrates this.

[0083] First, construct a class token with a dimension of 1×768, and then associate it with X. ～M The features are then concatenated to form a 50×768 feature vector.

[0084] Then, the feature vector with a dimension of 50×768 is input into a 12-layer Transformer Block network with a multi-head attention mechanism.

[0085] Finally, an intermediate implicit feature R with a dimension of 50×768 is obtained through feature extraction.

[0086] Step 5.3: Use the learnable feature vector to replace the feature representation of the masked part, and then concatenate it with the intermediate implicit feature R.

[0087] Since the encoder stage only utilizes image information from the unmasked regions, the decoder stage needs to restore the feature representations of the masked regions in order to reconstruct the original image, thereby ensuring that the spatial dimensions of the final prediction remain aligned with the original image.

[0088] The following example illustrates this.

[0089] First, the intermediate implicit feature R is input into a fully connected layer with an input dimension of 768 and an output dimension of 512, resulting in a feature vector of dimension 50×512.

[0090] Then, a learnable feature vector R of 147×512 is randomly initialized. M As a substitute feature for the masked area.

[0091] Finally, and R M The features R are then stitched together to form a corresponding complete image. all Its dimensions are 197×512.

[0092] Step 5.4: Transfer R all The image is input into the decoder and reconstructed using the decoder.

[0093] The following example illustrates this.

[0094] The decoder uses an 8-layer Transformer Block network structure with a multi-head attention mechanism, where the number of heads is 16.

[0095] R from the above steps all The input is fed into the decoder, and the feature vector R is obtained through feature extraction. r Then R r Feed a fully connected layer with an input dimension of 512 and an output dimension of 768, and remove the class token dimension to obtain a feature vector with a dimension of 196×768. Finally, through dimension rearrangement, obtain a feature vector with a dimension of 3×224×224, denoted as Z.

[0096] The supervision signal for this reconstruction task is the original pixel information of the masked patch block, and the loss function used is the L2 distance function, as shown in equation (3):

[0097]

[0098] In equation (3), M represents the number of pixels, and y i Represents the pixel values ​​of the original image. L2 represents the pixel value predicted by the autoencoder, and L2(·) represents the L2 distance function.

[0099] Step 6: Take the model W obtained in Step 5 R It is embedded as part of the skeleton network in the object detection network framework, and then the model parameters are fine-tuned in a supervised manner on data with labeled information.

[0100] This invention provides a specific implementation method, such as... Figure 3 As shown, it includes the following steps:

[0101] Step 6.1: Convert the autoencoder model W based on the Vision Transformer network structure R The encoder parameters are extracted separately and used as model parameters for subsequent fine-tuning, denoted as F.

[0102] Step 6.2: Embed the model parameters F and the corresponding network structure as a skeleton network into the object detection network.

[0103] The following example illustrates the model fine-tuning process:

[0104] First, the input image size is normalized to 3×512×512, and then data augmentation is performed.

[0105] Then, we will use the Mask R-CNN object detector as an example. The Mask R-CNN network structure consists of a skeleton network, a neck network, and a head network. During model fine-tuning, only the encoder's model parameters F need to be loaded into the skeleton network.

[0106] Secondly, the skeleton network with pre-trained model parameters is combined with the remaining network layers to complete the construction of the object detector.

[0107] Finally, the target is detected using a constructed target detector.

[0108] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make several modifications without departing from the principles of the present invention, and these modifications should also be considered to fall within the scope of protection of the present invention.

Claims

1. A self-supervised pre-training method suitable for remote sensing target detection tasks, characterized in that... The steps of this method include: Step 1: Perform self-supervised pre-training on the autoencoder network based on the Vision Transformer feature extraction network on a natural scene dataset to obtain model parameters with domain-level generalization knowledge. Step 2: Perform random multi-scale target cropping on the real target bounding boxes of the target detection task dataset to obtain target-level task-related image data; Step 3: Based on the target-level task-related image data obtained in Step 2, first divide the image data into small-scale target image data and non-small-scale target image data according to the image size. Then, uniformly rescale the small-scale target image data and perform random stitching operation. Step 4: On the non-small-scale target image data obtained in Step 3 and the small-scale target image data after random stitching, sample resampling is performed to address the imbalance in the number of class samples in the multi-class target detection dataset, thereby achieving a balance in the number of multi-class samples. Step 5: Use the non-small-scale target image data and small-scale target image data after sample resampling in Step 4 to update the model parameters with domain-level generalization knowledge in Step 1, and complete the self-supervised pre-training suitable for remote sensing target detection tasks.

2. The self-supervised pre-training method for remote sensing target detection tasks according to claim 1, characterized in that: In step 1, the autoencoder network based on the Vision Transformer feature extraction network is pre-trained on a natural scene dataset with millions of data points using an image reconstruction task, so as to obtain model parameters with domain-level generalization knowledge. The autoencoder based on the Vision Transformer feature extraction network consists of two parts: an encoder and a decoder. The encoder uses a 12-layer Transformer Block network structure with a multi-head attention mechanism, while the decoder uses an 8-layer Transformer Block network structure with a multi-head attention mechanism.

3. The self-supervised pre-training method for remote sensing target detection tasks according to claim 2, characterized in that: In step 2, the method for random multi-scale target cropping is as follows: Step 2.1: Based on the ground truth annotations of the training set of the object detection task dataset, obtain the center point coordinates (x, y), target width w, and height of all bounding boxes; Step 2.2: Using the center point (x,y) of the target as a reference, randomly scale the target width w and height by a factor of α to obtain a rectangular region. Crop the obtained rectangular region to obtain target-level image data, as shown in the following formula (1): In equation (1), I scene This represents the raw remote sensing image at the scene level; N represents the number of ground truth bounding boxes on this scene-level image. w represents the coordinates of the center point of the i-th annotation box; i and i Let represent the width and height of the i-th annotation box; α represents the random scaling factor, which ranges from [3,5]; I object This represents the final target-level image data; Crop(·) represents the cropping operation; i = 1, 2, 3, ..., N.

4. The self-supervised pre-training method for remote sensing target detection tasks according to claim 3, characterized in that: In step 3, the method for dividing the target image data into small-scale and non-small-scale target image data based on image size is as follows: Set a size threshold β for small-scale target images, i.e., β = 80×80 resolution, and filter the target-level image data obtained in step 2 according to the size threshold β.

5. A self-supervised pre-training method for remote sensing target detection tasks according to claim 3, characterized in that: In step 3, the method for uniformly rescaling and randomly stitching small-scale target image data is as follows: Small-scale target image data are uniformly scaled to 112×112 resolution, and four images of the same category are randomly selected and stitched together in a 2-row, 2-column arrangement.

6. A self-supervised pre-training method for remote sensing target detection tasks according to claim 5, characterized in that: In step 4, the method for sample resampling is as follows: Step 4.1: Based on the acquired target-level image data, count the number of image data in each category and calculate the average number of samples in each category, as shown in the following formula (2): In equation (2), C represents the number of categories; N j N represents the number of target-level images in the j-th category; avg This represents the average number of samples in each category; j = 1, 2, 3, ..., C; Step 4.2: For quantities greater than N avg Perform sample downsampling on categories with fewer than N; avg Oversampling is performed on each category to ensure that the number of samples in each category remains balanced.

Images