Remote sensing image self-supervision pre-training method based on multi-view consistency constraint
The self-supervised pre-training method for remote sensing images using multi-scale cropping and dual alignment loss functions addresses the issues of insufficient multi-scale feature modeling and weak local consistency in self-supervised representation learning of remote sensing images, thereby improving the robustness and stability of remote sensing images and enhancing task performance in remote sensing scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIMEI UNIV
- Filing Date
- 2026-01-27
- Publication Date
- 2026-04-24
AI Technical Summary
Existing self-supervised representation learning methods for remote sensing images lack modeling of multi-scale characteristics and lack consistency constraints between local views, making it difficult for models to learn robust multi-scale invariant representations and resulting in insufficient stability of local features, which affects the performance of downstream tasks in remote sensing scenarios.
We employ a label-free self-distillation method for vision Transformers, generating multi-scale views through a multi-scale cropping strategy. We then train the method using a dual alignment loss function, which includes local alignment to global loss and local alignment to local loss. This strengthens the feature consistency constraints between local views and generates more discriminative remote sensing representations.
It significantly enhances the multi-scale invariance learning capability of remote sensing images, improves the model's robustness to complex ground structures and semantic stability of local regions, and enhances the performance of tasks such as remote sensing classification, detection, and segmentation.
Smart Images
Figure CN121921673A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent analysis of remote sensing images and computer vision technology, and relates to a method based on multi-view... Figure 1 The self-supervised pre-training method for remote sensing images with consistency constraints can be applied to remote sensing intelligent interpretation tasks such as remote sensing scene classification, target detection, semantic segmentation, and multi-temporal change detection. Background Technology
[0002] Currently, self-supervised representation learning for remote sensing images often employs a self-distillation framework based on contrastive learning or a teacher-student structure. This approach generates two global views and several local views from the same image, using the teacher network output as a supervisory signal to constrain the student network to maintain semantic consistency between local and global views, thereby achieving unlabeled feature learning. In these methods, view generation strategies are primarily based on random cropping and data augmentation, while consistency constraints are limited to the unidirectional relationship of aligning local views with global views.
[0003] When the existing technologies mentioned above are directly applied to remote sensing images, the following technical problems and defects still exist: (1) Existing methods lack modeling of the multi-scale characteristics of remote sensing images.
[0004] Remote sensing images contain objects with extremely large scale differences (the size difference between buildings, roads, farmland, water bodies, etc. can be tens of times). However, the local view generated by existing methods has a limited scale range, and all local views come from different random locations. This makes it impossible to effectively simulate the observation changes of the same ground feature area at different scales, resulting in the model having difficulty learning robust multi-scale invariant representations.
[0005] (2) There is a lack of consistency constraints between local views, and local features are unstable.
[0006] Existing self-supervised frameworks only align "local to global" views, without constraining the internal consistency between multiple local views. When local views come from the same geographic area but have different clipping scales or enhancement perturbations, their features often exhibit semantic shifts, resulting in a loose local feature space and insufficient stability of the model, which affects the performance of downstream tasks (such as object detection and change detection).
[0007] In summary, existing self-supervised pre-training methods still have technical shortcomings in terms of insufficient multi-scale modeling and weak local consistency. There is an urgent need to design new multi-view construction and consistency learning mechanisms in remote sensing scenarios to improve the scale robustness and local feature stability of the model. Summary of the Invention
[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-view based... Figure 1 A self-supervised pre-training method for remote sensing images with consistency constraints.
[0009] To achieve the objectives of this invention, the following technical solutions are adopted.
[0010] A multi-view based Figure 1 A self-supervised pre-training method for remote sensing images with consistency constraints is proposed. This method employs a label-free self-distillation approach oriented towards the visual Transformer to perform self-supervised pre-training on remote sensing images, thereby obtaining a remote sensing representation pre-trained model. The process of obtaining the remote sensing representation pre-trained model includes the following steps: S1. Perform multi-scale cropping strategy and data augmentation operation sequentially on the input remote sensing image to generate a multi-scale view set; the multi-scale view set includes a global view and a local view, wherein: the local view includes a local view used to learn scale-invariant features; S2. The teacher network takes a global view as input, and the student network takes both a global view and a local view as input. A dual-alignment loss function is used to train the student network to obtain a pre-trained remote sensing representation model adapted to downstream tasks. The dual-alignment loss function consists of a local-aligned global loss function and a local-aligned local loss function. The local alignment global loss function is obtained by minimizing the cross-entropy between the probability distributions of the teacher network output and the probability distributions of the student network output. It is used to constrain the consistency of local and global view features in the spatial representation of the same remote sensing image, i.e., feature consistency constraint. Furthermore, by extracting the global view features of the teacher network, the global view features of the student network, and the local view features, a feature carrier is provided for calculating the local alignment local loss function. The local alignment loss function, under the condition of imposing feature consistency constraints on any two local views, is obtained by calculating the cross-entropy between the feature distributions of any two local views and adding a symmetric inverse term. It is used to constrain the uniformity of features of each local view within the same remote sensing image.
[0011] Furthermore, the multi-scale cropping strategy employs both a global view cropping strategy and a local view cropping strategy to crop the input remote sensing image, generating two global views and ten local views, wherein: The global view cropping strategy randomly crops twice within a large scale range to obtain two large-scale images, and adjusts the size of the two large-scale images to the same preset size to generate two global views; The local view cropping strategy randomly crops six times at different spatial locations within a small scale range to generate six small-scale images. The size of each of the six small-scale images is then adjusted to six different preset sizes to generate six local views. Then, another small-scale image is randomly cropped within the small scale range, and the size of that small-scale image is adjusted to four different preset sizes to generate four local views for learning scale-invariant features.
[0012] Furthermore, the data augmentation operation includes a global view. Enhanced operations, global view Enhancement operations and local view enhancement operations, where: Global view Enhanced action: Random horizontal flip Color fluctuation Random grayscale Strong Gaussian blur Enhancement formula: ; Global view Enhanced action: Random horizontal flip Color fluctuation Random grayscale Weak Gaussian blur Enhancement formula: ; Local view enhancement: Random horizontal flip Color fluctuation Random grayscale Gaussian blur Enhancement formula: .
[0013] Furthermore, the large-scale range is 32%-100%; the small-scale range is 5%-32%.
[0014] Furthermore, the image after the data augmentation operation can maintain the semantics of the remote sensing image and still represent the same ground feature or the same scene.
[0015] Furthermore, the multi-scale view set V={vg1,vg2,v1,v2,…,v10}, where: vg1 and vg2 are global views, and v1 to v10 are local views.
[0016] Furthermore, the construction process of the dual alignment loss function includes the following steps: S71, Define Student Network Teacher Network Student Network Teacher Network Output K-dimensional probability distribution and The output K-dimensional probability distribution and After normalization using the softmax function, we obtain:
[0017] in: It is a temperature parameter that controls the sharpness of the output distribution. The calculation formula is similar and uses temperature parameters. In a fixed teacher network Under the premise of minimizing student network parameters The cross-entropy loss is used to match these two distributions:
[0018] Where: cross-entropy loss function It is used to provide a unified standard of measurement; S72. Input all global and local views from the multi-scale view set V into the student network, and only input the global view into the teacher network. By minimizing the cross-entropy between the probability distributions output by the teacher network and the probability distributions output by the student network, establish a local alignment global loss function:
[0019] And extract the global view features of the teacher network and the global and local view features of the student network: ; ; .
[0020] S73, For any two partial views Apply feature consistency constraints and calculate the cross-entropy between the feature distributions of any two local views:
[0021] And add a symmetrical reverse term to ensure symmetry:
[0022] S74. Construct a dual alignment loss function for training the student network based on the loss functions of local alignment to global and local alignment to local:
[0023] In the formula: This is a hyperparameter used to balance the contribution weights of local alignment to global and local alignment to local loss.
[0024] Furthermore, the hyperparameter λ ranges from 0.1 to 0.5.
[0025] Furthermore, the downstream tasks include scene classification, object detection, semantic segmentation, and multi-temporal change detection.
[0026] Compared with the prior art, the present invention has the following advantages: 1. Based on the traditional DINO multi-view input structure, this invention proposes a multi-scale resampling mechanism for local views: in addition to the original 6 randomly cropped local views, 4 additional local images from the same spatial location but at different scales are added, enabling the model to learn stable semantic features under scale changes within a fixed area, thereby significantly enhancing the multi-scale invariance learning ability unique to remote sensing images. 2. This invention is the first to propose a local-local feature consistency constraint in remote sensing scenarios. It applies an alignment loss to any two local views, enabling the model to maintain consistency not only between the global and local views, but also to enhance the semantic stability between local regions. It is particularly suitable for remote sensing images with large differences in the scale of ground features, and improves the robustness of the model to complex ground feature structures. 3. By introducing multi-scale local view combination and local-local consistency alignment, this invention can generate more discriminative remote sensing representations while maintaining cross-scale stability, enabling the pre-trained model to perform better in tasks such as remote sensing classification, detection, segmentation and change detection. Attached Figure Description
[0027] Figure 1 This is a schematic diagram of the workflow of the present invention; Figure 2 The process of constructing multi-scale views; Figure 3 This refers to the network model training process. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0029] As an embodiment of the present invention, such as Figures 1 to 3 As shown, a multi-view-based Figure 1 A self-supervised pre-training method for remote sensing images with consistency constraints includes the following steps: S1, Multi-scale view construction, such as Figure 2 As shown; For input remote sensing images Two global views are generated according to a multi-scale strategy. and ten partial views : S11, Global View Clipping: For an input image Randomly crop the image within the range of 32% to 100% to generate two global views, and then resize the two global views to... ; S12, Partial view clipping: Original partial view clipping: Randomly crop the images within a range of 5% to 32%, generating six randomly cropped images. Then, resize the first six images to six different sizes. , Newly added partial view clipping: One image is randomly cropped from 5% to 32% of the original image, and then the image is resized to four different sizes. This generates four local views that are spatially consistent but at different scales, enabling the model to directly learn scale-invariant features.
[0030] S2. Feature extraction from student-teacher networks; S21, Multi-scale view of student network coding; Student Network Encode all views and output the corresponding feature representations:
[0031] S22. Stability characteristics of teacher network coding; Teacher Network Then encode the global view:
[0032] S23. Update the teacher network using exponential moving average (EMA); Teacher network parameters are updated synchronously from the student network via EMA:
[0033] Where m is the momentum factor, used to maintain the smooth stability of teacher-side features.
[0034] S3, Consistency Constraints and Optimization, such as Figure 3 As shown; S31, Local-Global Consistency Constraint (DINO Standard Loss) The student network's prediction of the local view must be consistent with the teacher network's prediction of the global view.
[0035] After temperature normalization:
[0036] The consistency loss is:
[0037] S32, Local-Local Consistency Constraint For any two partial views :
[0038] The local-local consistency loss is:
[0039] This loss method ensures semantic consistency for the same land cover region at different scales, thus enabling stable local feature representations even under complex scale variations.
[0040] S33, Total Loss Function; The ultimate training objective of the model is:
[0041] Where: λ is the balance coefficient.
[0042] This invention is applied to the automatic identification of land use in remote sensing scenarios, used for intelligent identification and classification of land cover types in large-scale remote sensing imagery. In this application, optical remote sensing image data covering the target area is first acquired from a remote sensing satellite platform. The images can originate from Sentinel-2, Landsat, or other Earth observation satellite systems, and the acquisition phase does not rely on manual annotation information. The acquired remote sensing images are then preprocessed, and the processed images are used as input data.
[0043] Subsequently, unsupervised feature learning is performed using this invention. By generating global and multi-scale local views of the same remote sensing image during the pre-training stage, the model can simultaneously learn the overall spatial structure information of the image and local texture and semantic features at different scales, thereby constructing a general remote sensing feature extraction model. After feature learning is completed, the feature extraction model is deployed in a land use identification system to extract features from newly acquired remote sensing images. Combined with subsequent classification modules, it enables automatic identification and mapping of various land use types such as farmland, forest land, water bodies, and construction land. This application example is suitable for practical business scenarios such as land resource surveys, urban planning decision support, and regional ecological environment monitoring.
[0044] This invention can also be applied to urban area change monitoring scenarios, enabling automatic analysis and identification of surface change information in remote sensing images from different time periods. In this application, multiple remote sensing image data of the same geographical area collected at different times are acquired, and the images are registered, cropped, and spatially aligned to form temporally consistent remote sensing image pairs.
[0045] In the change monitoring system, this invention employs unsupervised feature modeling of single-period remote sensing images, enabling the model to learn stable structural features and local details at different spatial scales without relying on manual annotation. Subsequently, the feature extraction model is used to extract features from remote sensing images at different time phases, and the extracted features are input into the change analysis module to automatically identify and label areas in the images that have undergone changes. Through multi-scale feature representation, the system can simultaneously monitor large-scale urban structural changes as well as localized minor changes, such as new buildings, road expansions, or land use adjustments. This application example can be widely used in practical scenarios such as urban expansion monitoring, infrastructure construction supervision, and post-disaster change analysis.
[0046] Table 1: Performance results of KNN and linear detection methods on the EuroSAT land use classification task
[0047] To verify the effectiveness of the proposed multi-scale local cropping self-supervised representation learning method in remote sensing image feature learning, a comparative experiment was conducted based on the SeCo-100K remote sensing unlabeled dataset. This dataset contains approximately 100,000 multi-temporal remote sensing images from Sentinel-2. During pre-training, only the original image information was used without any manual annotations. All comparative methods were pre-trained under the same data scale and data source conditions to ensure experimental fairness. The DINO method based on a fixed-scale local cropping strategy, the DINO-TP method which incorporates a temporal perspective as positive samples, and the DINO-MC method were selected as comparison objects. These methods maintained consistency in network structure, loss function form, and optimizer settings, differing only in the local view cropping strategy and the construction method of positive samples. To verify the generality of the method, ResNet-50, Wide ResNet-50-2, Vision Transformer (ViT-Small), and Swin Transformer (Swin-Tiny) were used as backbone networks, and none of the backbone networks were loaded with any supervised pre-training weights during the pre-training phase. After completing self-supervised pre-training, the backbone network parameters were frozen, and the features were evaluated on the EuroSAT remote sensing land use classification dataset. The evaluation methods included KNN classification evaluation based on frozen features and linear probing evaluation using only trained linear classifiers. The evaluation metric was uniformly classification accuracy.
[0048] Table 1 summarizes the classification accuracy results of different self-supervised methods on the EuroSAT dataset under linear probing and KNN evaluation, reflecting the quality of the feature representations learned by each method. Experimental results show that, under the same pre-training data scale, this invention achieves the highest classification accuracy in most backbone network configurations. Compared with the DINO method using fixed-scale local cropping, it shows significant performance improvements on backbone networks such as ResNet-50, WRN-50-2, and ViT-Small. Furthermore, compared with the DINO-TP method which introduces temporal positive samples, it has advantages in feature stability and classification consistency. Further analysis shows that the multi-scale local cropping strategy enables the model to simultaneously perceive local semantic information at different spatial scales during the pre-training stage. This alleviates, to some extent, the impact of scale variations and semantic complexity in remote sensing images on the stability and discriminative power of feature learning, enhances the alignment ability between local regions, and thus significantly improves the feature discriminative performance in downstream classification tasks. Taking the WRN-50-2 backbone network as an example, the method of this invention uses only 100,000 unlabeled remote sensing images for pre-training, and its linear detection accuracy has surpassed some methods that use millions of data points for pre-training, demonstrating high data utilization efficiency. Furthermore, the results in Table 1 show that the method of this invention exhibits a consistent performance improvement trend across both convolutional neural network and Transformer network structures, with minimal performance fluctuations under different backbone networks. This indicates that the method does not depend on a specific network structure, possesses good versatility and stability, and demonstrates a more reliable performance advantage over time-positive sample alignment strategies in remote sensing land use classification tasks.
[0049] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A self-supervised pre-training method for remote sensing images based on multi-view consistency constraints, employing a label-free self-distillation method oriented towards vision Transformer to perform self-supervised pre-training of remote sensing images to obtain a remote sensing representation pre-trained model, characterized in that: The process of obtaining the remote sensing representation pre-trained model includes the following steps: S1. Perform multi-scale cropping strategy and data augmentation operation sequentially on the input remote sensing image to generate a multi-scale view set; the multi-scale view set includes a global view and a local view, wherein: the local view includes a local view used to learn scale-invariant features; S2. The teacher network takes a global view as input, and the student network takes both a global view and a local view as input. A dual-alignment loss function is used to train the student network to obtain a pre-trained remote sensing representation model adapted to downstream tasks. The dual-alignment loss function consists of a local-aligned global loss function and a local-aligned local loss function. The local alignment global loss function is obtained by minimizing the cross-entropy between the probability distributions of the teacher network output and the probability distributions of the student network output. It is used to constrain the consistency of local and global view features in the spatial representation of the same remote sensing image, i.e., feature consistency constraint. Furthermore, by extracting the global view features of the teacher network, the global view features of the student network, and the local view features, a feature carrier is provided for calculating the local alignment local loss function. The local alignment loss function, under the condition of imposing feature consistency constraints on any two local views, is obtained by calculating the cross-entropy between the feature distributions of any two local views and adding a symmetric inverse term. It is used to constrain the consistency of features of each local view within the same remote sensing image.
2. The self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 1, characterized in that: The multi-scale cropping strategy employs both a global view cropping strategy and a local view cropping strategy to crop the input remote sensing image, generating two global views and ten local views respectively. The global view cropping strategy randomly crops twice within a large scale range to obtain two large-scale images, and adjusts the size of the two large-scale images to the same preset size to generate two global views; The local view cropping strategy randomly crops six times at different spatial locations within a small scale range to generate six small-scale images. The size of each of the six small-scale images is then adjusted to six different preset sizes to generate six local views. Then, another small-scale image is randomly cropped within the small scale range, and the size of that small-scale image is adjusted to four different preset sizes to generate four local views for learning scale-invariant features.
3. The self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 2, characterized in that: The data augmentation operation includes a global view. Enhanced operations, global view Enhancement operations and local view enhancement operations, where: Global view Enhanced action: Random horizontal flip Color fluctuation Random grayscale Strong Gaussian blur Enhancement formula: ； Global view Enhanced action: Random horizontal flip Color fluctuation Random grayscale Weak Gaussian blur Enhancement formula: ； Local view enhancement: Random horizontal flip Color fluctuation Random grayscale Gaussian blur Enhancement formula: 。 4. The self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 3, characterized in that: The large-scale range is 32%-100%; the small-scale range is 5%-32%.
5. The self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 4, characterized in that: The image after the data augmentation operation can maintain the semantics of the remote sensing image and still represent the same ground feature or the same scene.
6. The self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 5, characterized in that: The multi-scale view set V={vg1,vg2,v1,v2,…,v10}, where vg1 and vg2 are global views, and v1 to v10 are local views.
7. The self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 6, characterized in that: The process of constructing the double alignment loss function includes the following steps: S71, Define Student Network Teacher Network Student Network Teacher Network Output K-dimensional probability distribution and The output K-dimensional probability distribution and After normalization using the softmax function, we obtain: ； in: It is a temperature parameter that controls the sharpness of the output distribution. The calculation formula is similar and uses temperature parameters. In a fixed teacher network Under the premise of minimizing student network parameters The cross-entropy loss is used to match these two distributions: ； Where: cross-entropy loss function It is used to provide a unified standard of measurement; S72. Input all global and local views from the multi-scale view set V into the student network, and only input the global view into the teacher network. By minimizing the cross-entropy between the probability distributions output by the teacher network and the probability distributions output by the student network, establish a local alignment global loss function: ； And extract the global view features of the teacher network and the global and local view features of the student network: ； ； ； S73, For any two partial views Apply feature consistency constraints and calculate the cross-entropy between the feature distributions of any two local views: ； And add a symmetrical reverse term to ensure symmetry: ; S74. Construct a dual alignment loss function for training the student network based on the loss functions of local alignment to global and local alignment to local: ； In the formula: This is a hyperparameter used to balance the contribution weights of local alignment to global and local alignment to local loss.
8. The self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 7, characterized in that: The value range of the hyperparameter λ is 0.1-0.
5.
9. A self-supervised pre-training method for remote sensing images based on multi-view consistency constraints according to claim 8, characterized in that: The downstream tasks include scene classification, object detection, semantic segmentation, and multi-temporal change detection.