A Contrastive Self-Supervised Learning Method for Remote Sensing Image Representation
By constructing a self-supervised training dataset from heterogeneous satellite images and combining it with a teacher-student network model, the semantic inconsistency problem caused by random augmentation in self-supervised learning of remote sensing images is solved, thereby improving the self-supervised learning performance of remote sensing images and the detection accuracy of downstream tasks.
Patent Information
- Application Number
- CN202210522221.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-13
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
General self-supervised learning methods do not perform well on remote sensing images. Random data augmentation leads to irreversible damage to foreground target information, affecting the performance of self-supervised learning.
We construct self-supervised training samples for remote sensing images using heterogeneous satellite imagery, augment the data by utilizing differences in sensor type, shooting angle, and access time, and combine this with a teacher-student network model for self-supervised training to ensure semantic consistency and diversity of the views.
It effectively improves the performance of self-supervised learning of remote sensing images, reduces the dependence on manual annotation, and improves the detection accuracy of downstream tasks.
Smart Images

Figure CN114972934B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer vision, and in particular relates to a self-supervised learning method for remote sensing images. Background Art
[0002] Benefiting from the vigorous development of deep learning, remote sensing image automatic interpretation technology has made great progress. Models based on deep neural networks have greatly increased the efficiency and quality of automatic interpretation, and have also achieved amazing milestones in remote sensing scene classification, semantic segmentation, target detection and other tasks. Generally speaking, deep learning-based methods often require a large amount of training data to supervise the training of the network in order to achieve good results, but the annotation of remote sensing images is time-consuming, labor-intensive and extremely costly, which to some extent limits the performance of remote sensing image interpretation.
[0003] The phenomenon that image resources are abundant but the cost of annotation is high in computer vision tasks has promoted the development of self-supervised learning. The goal of self-supervised learning is to mine potential semantic association information from a large amount of unlabeled data to obtain a general visual representation, which can effectively alleviate the requirement for the amount of labeled data. At present, a mainstream self-supervised learning method is based on contrastive learning. Different augmented views are generated by randomly cropping and flipping the original image, and the network is trained to generate representation vectors with consistent semantic expressions in the feature space for these augmented views, thus achieving self-supervised learning.
[0004] However, when applying general self-supervised learning methods to remote sensing images, good results are often not achieved. Unlike natural images, which are usually centered on the target and have low scene complexity, the scenes in remote sensing images are more complex, and the foreground targets are densely distributed. When general contrastive self-supervised learning methods perform random augmentation, it is very easy to cause irreversible damage to the foreground target information. The non-common parts between views can hardly be restored by prediction methods during the contrastive learning process, which also results in poor image representation effects from self-supervised learning and has a negative impact on downstream visual tasks. Therefore, it is necessary to design a contrastive self-supervised learning method for remote sensing image representation. Summary of the invention
[0005] After research, it was found that self-supervised learning in the general vision field does not work well on remote sensing images. The views formed by random data augmentation are prone to semantic deviations. Simply reducing the intensity of data augmentation will lose the diversity of augmented views and impair the performance of self-supervised learning.
[0006] To solve the above technical problems, the present invention adopts heterogeneous satellite images to construct remote sensing image self-supervised training samples, and utilizes the imaging differences caused by differences in sensor models, shooting angles, access times, etc. of heterogeneous satellites as natural augmentation, while ensuring the semantic consistency of augmented views and balancing the diversity of different views.
[0007] The technical solution adopted by the present invention is: a comparative self-supervised learning method for remote sensing image representation, comprising the following steps:
[0008] Step 01: Obtain heterogeneous and registered high-resolution optical remote sensing satellite images, use sliding windows of different sizes to randomly crop multi-source image pairs of different sizes synchronously, generate registered remote sensing image pairs, and at the same time, filter data according to the size of pixel variance in the image, retaining image pairs with pixel variance greater than 20 for self-supervised training of remote sensing images;
[0009] Step 02: Using the heterogeneously registered image pair, randomly select a square area with an area of 30% to 100% within the image range, and crop the two images simultaneously with the area as the boundary to obtain a pair of views. Then, the two views are independently randomly rotated and enhanced, and finally uniformly scaled to the same size to generate a pair of views in the same scene.
[0010] Step 03: Utilize the paired views and send them into the teacher-student self-supervised learning network for training. In both the teacher network and the student network, firstly extract the feature maps of the views through the backbone network, and construct a feature pyramid according to the feature map pyramid network. And use the feature projector with the same structure to generate the projection vector of the corresponding level from the features of each level of the feature pyramid. Then, arbitrarily select the projection vectors of different levels of the two views in the teacher network and the student network, and use the feature predictor in the student network to predict the projection vector of the teacher network according to the projection vector of the student network, so as to realize self-supervised training.
[0011] Furthermore, in step 03, the student network includes a feature extractor f θ , feature projector g θ and feature predictor p θ , the teacher network includes a feature extractor f with the same architecture as the student network ε and feature projector g ε , where θ and ε represent the weight parameters of the student network and the teacher network respectively, and all feature projectors and feature predictors are composed of multi-layer perceptrons.
[0012] Furthermore, the feature extractor extracts image features through the backbone network and constructs a feature pyramid based on the feature map pyramid network. The backbone networks include ResNet and VGG. Taking ResNet as an example, the outputs of the last four stages of ResNet {C2, C3, C4, C5} are taken, and they are superimposed and fused from top to bottom after 1×1 convolution to obtain the features {M2, M3, M4, M5}, which are then subjected to 3×3 convolution to finally obtain multi-level feature maps {P2, P3, P4, P5}.
[0013] Furthermore, the specific implementation process of step 03 is as follows;
[0014] According to the generated view v i and v′ i , respectively input into the teacher network and the student network, and then pass through the feature extractor f θ and f ε Extract multi-level feature maps and Then through the feature projector g θ and g ε Get the feature projection vector and Afterwards, by random sampling, the feature projection vector of one level of the multi-level feature projection vectors extracted by the student and teacher networks is selected, denoted as z i and z′ i , the feature predictor p in the student network θ Used to project the vector z according to the characteristics of the student network i To predict the feature projection vector z′ of the teacher network i , through the prediction between the feature projection vectors at different levels in the two branches, the scale invariance of remote sensing images can be learned during the training process; the output of the student network is denoted as p θ (z i ), the loss function is defined as the mean square error between the prediction vector output by the student network and the projection vector output by the teacher network, that is,
[0015] Furthermore, the parameters θ of the student network are updated through network training as the loss function is back-propagated, and the parameters ε of the teacher network are updated through the exponential moving average of the student network parameters θ. Given a decay rate τ∈[0,1], the update formula of ε is: ε←τε+(1-τ)θ.
[0016] Furthermore, in step 01, tile map service interfaces of different satellite maps are used to obtain precisely aligned multi-source high-resolution satellite images, including Google Satellite Maps, Bing Maps, ARCGIS, and Tiandi Maps.
[0017] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: the present invention constructs a remote sensing self-supervised learning training data set by collecting heterogeneous satellite images, uses synchronous random cropping to ensure semantic consistency between paired views, and relies on the imaging differences caused by differences in sensor models, shooting angles, access times, etc. of heterogeneous satellites to constitute natural data augmentation, thereby balancing the diversity of different views; combined with an asynchronous teacher-student network model, comparative self-supervised learning of remote sensing images is achieved. Compared with the general comparative self-supervised learning method, the method disclosed in the present invention can effectively avoid semantic inconsistencies caused by random augmentation, thereby damaging the performance of self-supervised learning representation of remote sensing images. The present invention can be effectively applied to the interpretation task of remote sensing images based on deep learning, effectively reducing the reliance on manual annotation of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A flow chart of a remote sensing image comparison self-supervised learning method according to an embodiment of the present invention;
[0019] Figure 2 This is a comparative self-supervised learning network framework diagram in an embodiment of the present invention;
[0020] Figure 3 It is a framework diagram of the feature pyramid network in the implementation example of the present invention. DETAILED DESCRIPTION
[0021] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0022] like Figure 1 As shown, the process of the method of the present invention mainly includes the following steps:
[0023] Step 01: Obtain heterogeneous and registered high-resolution optical remote sensing satellite images, use sliding windows of different sizes to randomly crop multi-source image pairs of different sizes synchronously, generate registered remote sensing image pairs, and at the same time, filter data according to the size of pixel variance in the image, retaining image pairs with pixel variance greater than 20 for self-supervised training of remote sensing images;
[0024] Specifically, use the tile map service interface of different satellite maps to obtain accurately aligned multi-source high-resolution satellite images, including but not limited to Google Satellite Maps, Bing Maps, ARCGIS, Tiandi Map, etc. The selected satellite images need to meet two conditions: first, they must have high semantic richness, and satellite image data from all over the world should be widely obtained to ensure that the self-supervised model can fully learn remote sensing features; second, there must be more prominent foreground targets in the image, and it should not contain too many areas with low information density such as mountainous areas and open spaces. Specifically, based on the GPS information of some public landmarks, select typical remote sensing scene areas containing common land objects to download large-size multi-source satellite images, including but not limited to urban residential areas, airports, ports, railway stations and power stations.
[0025] Use sliding windows of different sizes to randomly crop large-size multi-source remote sensing image pairs synchronously, including but not limited to 800×800, 1200×1200, and 2000×2000 crop windows. Considering that the aforementioned remote sensing images may contain single images without foreground, such as part of the sea surface and large tracts of fields and vegetation, which are useless in self-supervised learning, this method simply screens the data according to the size of the pixel variance in the image, retaining image pairs with a pixel variance greater than 20 to ensure that the image contains a certain amount of information. Finally, heterogeneous remote sensing image pairs of different sizes are obtained as remote sensing image self-supervised training datasets.
[0026] Step 02: Using the heterogeneously registered image pair, randomly select a square area with an area of 30% to 100% within the image range, and crop the two images simultaneously with the area as the boundary to obtain a pair of views. Then, the two views are independently randomly rotated and enhanced, and finally uniformly scaled to the same size to generate a pair of views in the same scene.
[0027] Specifically, for heterogeneous and registered image pairs I i and I′ i , randomly select a square area with an area of 30% to 100% in the image range, and crop the two images at the same time with this area as the boundary to obtain a pair of views x i and x′ i , cropping in the same area with the same size can preserve the same scene and ensure the semantic consistency of the two views. For the cropped view pair, the two views independently select the rotation enhancement angle α in the range of [0°, 360°) i and α′ i , taking the center position of the view as the rotation axis, apply the rotation transformation respectively, crop the area beyond the original image boundary during the rotation process, and fill the blank area with zero. Finally, scale it to 224×224 pixels to get the final view pair v i and v′i .
[0028] Step 03: Use the paired views to send them to the teacher-student self-supervised learning network for training. In both the teacher network and the student network, the feature maps of the views are first extracted through the backbone network, and the feature pyramid is constructed according to the feature map pyramid network (FPN). The feature projector with the same structure is used to generate the projection vector of the corresponding level from the features of each level of the feature pyramid. After that, the projection vectors of different levels of the two views in the teacher network and the student network are arbitrarily selected, and the projection vector of the teacher network is predicted according to the projection vector of the student network using the feature predictor in the student network, so as to achieve self-supervised training.
[0029] Specifically, use Figure 2 The teacher-student network model shown in the figure, the student network includes a feature extractor f θ , feature projector g θ and feature predictor p θ , the teacher network includes a feature extractor f with the same architecture as the student network ε and feature projector g ε , where θ and ε represent the weight parameters of the student network and the teacher network respectively. All feature projectors and feature predictors are composed of multi-layer perceptrons. The feature extractor extracts image features through the backbone network and constructs a feature pyramid based on the feature graph pyramid network. The choice of backbone network includes but is not limited to ResNet, VGG, etc. Taking ResNet as an example, the feature pyramid network is as follows Figure 3 As shown in the figure, the outputs of the last four stages of ResNet {C2, C3, C4, C5} are taken, and they are superimposed and fused from top to bottom after 1×1 convolution respectively, and the features {M2, M3, M4, M5} are obtained, and then they are subjected to 3×3 convolution to finally obtain the multi-level feature maps {P2, P3, P4, P5}.
[0030] According to the above generated view v i and v i ′ , respectively input into the teacher network and the student network, and then pass through the feature extractor f θ and f ε Extract multi-level feature maps and Then through the feature projector g θ and g ε Get the feature projection vector and Afterwards, by random sampling, the feature projection vector of one level of the multi-level feature projection vectors extracted by the student and teacher networks is selected, denoted as zi and z i ′ , the feature predictor p in the student network θ Used to project the vector z according to the characteristics of the student network i To predict the feature projection vector z of the teacher network i ′ , through the prediction between the feature projection vectors at different levels in the two branches, the scale invariance of remote sensing images can be learned during the training process. The output of the student network is denoted as p θ (z i ). The loss function is defined as the mean square error between the prediction vector output by the student network and the projection vector output by the teacher network, that is,
[0031] The parameters θ of the student network are updated through network training as the back propagation of the loss function, and the parameters ε of the teacher network are updated through the exponential moving average of the student network parameters θ. Given a decay rate τ∈[0,1], the update formula for ε is: ε←τε+(1-τ)θ.
[0032] In order to verify the optimization effect of the present invention on remote sensing image representation, the present application verifies the effect on the downstream remote sensing target detection task, and selects the DOTA dataset for verification. First, according to the dataset construction scheme of the present invention, a remote sensing image self-supervised learning training set of the same size as the ImageNet-1k classification dataset is constructed, including 1.28 million unlabeled remote sensing images with sizes of 800×800, 1200×1200, and 2000×2000; then the remote sensing image feature extractor trained by the self-supervised learning method of the present invention is migrated to the downstream detection task for fine-tuning, and the accuracy of the results of the detection task reflects the effectiveness of the method of the present invention. This application compares the self-supervised training model using the method of the present invention and the general self-supervised learning method on the dataset constructed by the present invention, as well as the existing ImageNet-1k dataset supervised pre-training model, and the performance of the migration to the downstream detection task, and the results are shown below.
[0033] Algorithm Description Downstream detection task mAP (%) ImageNet-1k supervised pre-training 63.7 General Contrastive Self-Supervised Methods (Random Data Augmentation) 51.4 This application 67.0
[0034] In summary, the comparative self-supervised learning method for remote sensing image representation described in the present invention mainly constructs a remote sensing self-supervised learning training data set by collecting heterogeneous satellite images, uses synchronous random cropping to ensure the semantic consistency between paired views, and relies on the imaging differences caused by differences in sensor models, shooting angles, access times, etc. of heterogeneous satellites to constitute natural data augmentation, balancing the diversity of different views; combined with an asynchronous teacher-student network model, comparative self-supervised learning of remote sensing images is achieved. The final experiment verifies that the method described in the present invention can improve the defects of the general comparative self-supervised learning method on remote sensing images, and surpasses the method of supervised pre-training using the ImageNet-1k data set. The present invention can be effectively applied to the interpretation task of remote sensing images based on deep learning, effectively reducing the reliance on manual annotation of remote sensing images.
[0035] It should be understood that parts not elaborated in detail in this specification belong to the prior art.
[0036] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.
Claims
1. A comparative self-supervised learning method for remote sensing image representation, characterized in that: The following steps are involved: Step 01, obtain heterogeneous and registered high-resolution optical remote sensing satellite images, use sliding windows of different sizes to synchronously and randomly crop multi-source image pairs of different sizes to generate registered remote sensing image pairs, and at the same time, filter data according to the size of pixel variance in the image, retaining image pairs with pixel variance greater than M for self-supervised training of remote sensing images; Step 02, using the heterogeneously registered image pair, randomly select a square area with an area of P1% to P2% within the image range, and crop the two images simultaneously with the area as the boundary to obtain a pair of views, and then perform random rotation enhancement on the two views independently, and finally uniformly scale them to the same size to generate a pair of views under the same scene; Step 03, using the paired views, they are sent to the teacher-student self-supervised learning network for training. In the teacher network and the student network, the feature maps of the views are first extracted respectively through the backbone network, and a feature pyramid is constructed according to the feature map pyramid network, and a feature projector with the same structure is used to generate a projection vector of the corresponding level from the features of each level of the feature pyramid; then, the projection vectors of different levels of the two views in the teacher network and the student network are arbitrarily selected, and the feature predictor in the student network is used to predict the projection vector of the teacher network according to the projection vector of the student network, so as to achieve self-supervised training.
2. The method for comparative self-supervised learning of remote sensing image representation according to claim 1, characterized in that: In step 03, the student network includes the feature extractor f θ , feature projector g θ and feature predictor p θ , the teacher network includes a feature extractor f with the same architecture as the student network ε and feature projector g ε , where θ and ε represent the weight parameters of the student network and the teacher network respectively, and all feature projectors and feature predictors are composed of multi-layer perceptrons.
3. The method for comparative self-supervised learning of remote sensing image representation according to claim 2, characterized in that: The feature extractor extracts image features through the backbone network and constructs a feature pyramid based on the feature map pyramid network. The backbone networks include ResNet and VGG. Taking ResNet as an example, the outputs of the last four stages of ResNet {C2, C3, C4, C5} are taken, and they are superimposed and fused from top to bottom after 1×1 convolution to obtain the features {M2, M3, M4, M5}, which are then subjected to 3×3 convolution to finally obtain multi-level feature maps {P2, P3, P4, P5}.
4. The method for comparative self-supervised learning of remote sensing image representation according to claim 2, characterized in that: The specific implementation process of step 03 is as follows; According to the generated view v i and v′ i , respectively input into the teacher network and the student network, and then pass through the feature extractor f θ and f ε Extract multi-level feature maps and Then through the feature projector g θ and g ε Get the feature projection vector and Afterwards, by random sampling, the feature projection vector of one level of the multi-level feature projection vectors extracted by the student and teacher networks is selected, denoted as z i and z′ i , the feature predictor p in the student network θ Used to project the vector z according to the characteristics of the student network i To predict the feature projection vector z′ of the teacher network i , through the prediction between the feature projection vectors at different levels in the two branches, the scale invariance of remote sensing images can be learned during the training process; the output of the student network is denoted as p θ (z i ), the loss function is defined as the mean square error between the prediction vector output by the student network and the projection vector output by the teacher network, that is, 5. The method for comparative self-supervised learning of remote sensing image representation according to claim 4, characterized in that: The parameters θ of the student network are updated through network training as the back propagation of the loss function, and the parameters ε of the teacher network are updated through the exponential moving average of the student network parameters θ. Given a decay rate τ∈[0,1], the update formula of ε is: ε←τε+(1-τ)θ.
6. The method for comparative self-supervised learning of remote sensing image representation according to claim 1, characterized in that: In step 01, use the tile map service interfaces of different satellite maps to obtain accurately aligned multi-source high-resolution satellite images, including Google Satellite Maps, Bing Maps, ARCGIS, and Tiandi Maps.
7. The method for comparative self-supervised learning of remote sensing image representation according to claim 1, characterized in that: The value of M is 20.
8. The method for comparative self-supervised learning of remote sensing image representation according to claim 1, characterized in that: P1 is 30 and P2 is 100.
