Remote sensing image self-supervised representation learning method

Abstract

The invention discloses a remote sensing image self-supervised representation learning method. The method comprises the following steps: acquiring a remote sensing image data set; taking the whole remote sensing image data set as input, generating a token representing abstract semantic information in the image by using an unsupervised token, and taking the token as a prediction target of the model; in order to reduce ground feature information provided by a pre-training task, predicting reconstruction loss through a teacher model, constructing prediction loss in combination with the importance of the token, generating the pre-training task according to a prediction loss sequence and transmitting the pre-training task to a student model, and completing image reconstruction by the student model; and outputting the classified remote sensing image. According to the invention, the influence of noise in original pixels is reduced, and the model is prevented from being limited to learning low-level pixel features.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing technology, and in particular relates to a remote sensing image self-supervised representation learning method. Background Art

[0002] Self-supervised representation learning (SSL) is a new pre-training paradigm derived from traditional supervised representation learning, which is based on the strong dependence of supervised representation learning on high-quality labeled data and the problem of easy overfitting. This paradigm is to input a large amount of unlabeled data sets during the pre-training process, and then use a small amount of labeled data sets for fine-tuning according to the different downstream tasks. The purpose is to use only a small amount of labeled data to make the performance of the pre-trained model transferred to the downstream tasks reach or even exceed that of supervised learning. Obviously, self-supervised representation learning can free the model from the dependence on large-scale labeled data sets, making it possible to obtain a good-performing unlabeled interpretable model, which just echoes the current situation in the field of remote sensing where satellite technology is booming and unlabeled remote sensing image data sets are surging, so that self-supervised representation learning has achieved relatively superior performance in the fields of scene classification and target detection of remote sensing image data sets.

[0003] Self-supervised representation learning methods can be divided into two main categories: contrastive learning and generative learning. Contrastive learning is a discriminative method that aims to use a similarity metric to achieve the purpose of moving negative samples away and positive samples closer during training. Generative learning usually reconstructs the input image in the form of image-encoder encoding-latent feature vector-decoder decoding. Among them, mask image modeling has attracted widespread attention as the most widely used and representative method of generative learning with superior performance. This type of method allows the image to be partially masked, and the remaining unmasked image is input into the model as known information, prompting the model to predict the specific content of the masked image and achieve the purpose of image feature learning. Many existing examples, such as MAEBEiT, follow the paradigm of mask image modeling and have achieved great success.

[0004] However, since remote sensing images have higher redundancy than natural images and are of various scales and types of objects, existing mask image modeling methods still have the following problems: 1) Over-emphasis on pixel details. The prediction targets of the models mostly focus on the original pixel values, and the training targets excessively pursue the absolute similarity between the original pixels and the reconstructed pixels, which leads to the model learning focusing on learning the low-level features of local pixels in the image and being sensitive to noise in remote sensing images. 2) The pre-training tasks are mainly generated by random masking strategies, which still expose the instance features of objects in remote sensing images. Affected by the hints of the pre-training tasks, the model representation learning cannot learn the deeper features of the objects in the images. Summary of the invention

[0005] Based on the above problems, this application proposes a MIM pre-training framework for loss-guided generation of pre-training tasks. The framework consists of three parts: an unsupervised tokenizer for integrating high-level abstract features of images, a teacher model for guiding the construction of mask pre-training tasks, and a student model for image reconstruction. Most existing models in the remote sensing field use pixels as prediction targets, which are often easily affected by noise in the original image and limit the model to learning low-level pixel features. Therefore, the entire remote sensing image dataset is used as input, and an unsupervised tokenizer is used to generate tokens representing abstract semantic information in the image, and it is used as the prediction target of the model, thereby ensuring that the model learns high-level image features without losing low-level pixel details. Due to the multi-scale and multi-object features of remote sensing images, the redundant features of the image are greatly amplified. Even if the existing random masking strategy performs large-scale masking, it is unlikely to accurately mask the object patches. The remaining uncovered views of the instance cause the instance features to be exposed, that is, a reconstruction hint is given to the model. For the model, this type of pre-training task still retains a lot of object visual information, and the features learned by the model are greatly suppressed. In order to reduce the ground object information provided by the pre-training task, it is assumed that a higher reconstruction loss value represents a rich amount of information. The reconstruction loss is predicted by the teacher model, and the prediction loss is constructed based on the importance of the token. The pre-training tasks are generated by sorting the prediction losses and passed to the student model, which then completes the image reconstruction task.

[0006] Specifically, the remote sensing image self-supervised representation learning method disclosed in this application includes the following steps:

[0007] Obtain remote sensing image datasets;

[0008] The entire remote sensing image dataset is used as input, and an unsupervised tokenizer is used to generate tokens representing abstract semantic information in the image, which are used as the prediction target of the model.

[0009] In order to reduce the ground object information provided by the pre-training task, the reconstruction loss is predicted by the teacher model, and the prediction loss is constructed based on the importance of the token. The pre-training tasks are generated by sorting the prediction losses and passed to the student model, which then completes the image reconstruction.

[0010] Output classified remote sensing image.

[0011] Preferably, the unsupervised tokenizer uses a KMeans clustering method to generate discrete tokens and construct a codebook, and uses the discrete tokens as the reconstruction target of the model;

[0012] The teacher model and the student model have the same structure, both of which are composed of a teacher encoder fθt , student encoder f θs And two decoders Composition, of which The reconstruction loss used to predict image patches, d φ Used for pixel reconstruction of covered patches. The subscripts t and s represent the teacher model and the student model. Both the encoder and the decoder are composed of the ViTbackbone network.

[0013] Preferably, during the training process, the input remote sensing image I∈R H×W×C Divide into a series of non-overlapping patches (H, W) is the resolution of the remote sensing image, C is the number of channels, P is the number of original pixels contained in the patches, N = HW / P 2 , represents the number of patches;

[0014] Then the remote sensing image is used as the teacher model encoder f θt The input of the encoder f θt The output intermediate latent feature vector h t As The input of is used to predict the reconstruction difficulty of each patch and guide the generation of the covering strategy;

[0015] The student model uses the masked image patches generated by the teacher model as input to complete the image reconstruction. The teacher model and the student model work alternately, and the parameters θ of the student model are s The update is completed according to the back propagation of the loss function, and the parameters θ of the teacher model t The updated student model θ s Exponential Moving Average EMA obtains:

[0016] θ t <-mθ t +(1-m)θ s

[0017] Where m represents the momentum coefficient.

[0018] Preferably, the unsupervised clustering method KMeans is used as a tokenizer, the original image data set is segmented into non-overlapping patchesx as input, and the capacity of the generated codebook is regulated by the hyperparameter k; the KMeans method is a function K, then the codebookZ is expressed as:

[0019] Z=K(x),Z={z1,z2,z3……z q},q∈[1,k]

[0020] where codebook Z∈R K×D , D represents the feature dimension, and each codebook vector {z1, z2, z3…zq} contained therein represents a cluster center, and these cluster centers represent the semantic feature categories in the remote sensing image; accordingly, the discrete tokent corresponding to each patch of each image is expressed as:

[0021] t=argmin||xz i ||,z i ∈Z

[0022] argmin(.) represents the nearest neighbor search, that is, for each patch, the cluster center obtained by the nearest neighbor search is used as its corresponding discrete token, which is used as the prediction target of the subsequent model.

[0023] Preferably, the N×N image patches are flattened to a length of N 2 The sequence is used as the input of the teacher model and the encoder f θt , and encoding Complete the loss prediction work; then calculate the importance of each cluster center based on the discrete token, and calculate the covering priority of each patch based on the loss value and cover it;

[0024] Use N×N image patchesx as the teacher model encoder Input, get the potential feature vector output by the encoder Then use the decoder Perform loss prediction and predict the loss L pred Depend on The output value is calculated by taking the average value. The specific formula is as follows:

[0025]

[0026] Where Mean(.) represents the average operation; based on the predicted loss, the importance of each type of discrete token is calculated and used as an indicator of its importance in the current image. The specific formula for the indicator of patches that should be covered is:

[0027]

[0028] Represents the average value of the prediction loss corresponding to each type of token, which represents the importance of the current image.

[0029] Preferably, the covering strategy is to perform argsort(.) operation on the loss index T in descending order to obtain the reconstruction difficulty of the patches inside the remote sensing image, intercept a certain proportion of the patches according to the sorting for covering; and introduce partial randomness, that is, the overall covering is 75%, of which 25% is randomly covered, and the first 50% of the patches are intercepted according to the sorting for covering, which is specifically expressed as:

[0030]

[0031] λ = argsort(T)

[0032] Where λ represents the sequence of index T after argsort(.), RandomSubset represents random selection from the remaining patches, and the resulting matrix M∈{0,1} representing the coverage is an N×N two-dimensional matrix, where elements 0 and 1 represent uncovered and covered patches, respectively.

[0033] Preferably, after masking guided by the teacher model, the unmasked remote sensing image patches are used as the student model encoder The input is passed through an image reconstruction encoder d φs , whose output Represents the original pixels obtained by image reconstruction;

[0034] For each covered patch, use a softmax classifier to predict the corresponding discrete token where x M represents the masked image, W c ∈R k×D ,b c ∈R k Represents the weight of the linear layer in the sotmax classifier;

[0035] The pre-training goal is to maximize the log-likelihood of correctly classifying discrete tokens given a masked image:

[0036]

[0037] Where ν represents the entire training data set, x M Represents the masked image after masking according to the masking matrix M.

[0038] Preferably, for the student model loss prediction Set up an additional training objective for loss prediction:

[0039] For each patch in the remote sensing image, predict the relative size of the prediction loss value between each patch, and implicitly learn the ranking relationship of the prediction loss between patches;

[0040] Finally, the training objective is expressed as:

[0041]

[0042] in and is from the decoder in the student model The i-th and j-th output Represents the prediction loss value of the student model, where i, j = 1, 2, 3... N represents the different patches in the image, used to identify each patch; σ(z) = e z / (e z +1) represents the sigmoid activation function; and As two identifiers, they are used to identify the relative size between the prediction loss and the standard reconstruction loss. The specific formula is as follows:

[0043]

[0044] Among them, M i =M j =1 means that patches i and patches j are covered during training. is the loss of model image reconstruction;

[0045] The final training goal of the model combined with the goal of loss prediction and the goal of image reconstruction is finally expressed as:

[0046]

[0047] The training objective of the model forces the pre-training tasks generated by the model to imply less visual concepts of objects; The training goal is to enable the model to learn abstract semantic information.

[0048] Through this new pre-training framework, a basic pre-training model is obtained, and the representation learning ability of the model is verified in the downstream task of scene classification. This application evaluates the model representation learning performance on three public datasets for remote sensing image scene classification: Aerial Image Dataset (AID), UC-Merced Land Use (UCM), and NWPU-RESISC45 (NWPU45).

[0049] The beneficial effects of this application are as follows:

[0050] (1) The unsupervised method is used as a tokenizer, and the token of the high-level abstract features of the collection dataset is used as the prediction target. This is to reduce the impact of noise in the original pixels and avoid the model being limited to learning low-level pixel features.

[0051] (2) We proposed a masking strategy guided by prediction and reconstruction loss, and built a self-supervised representation learning framework in which a teacher model guides masking and a student model reconstructs images. This aims to reduce the problem that random masking strategies can easily retain the visual concept identity of objects in remote sensing images, giving too many hints to model feature learning.

[0052] (3) Verification on public datasets shows that the model of this application achieves a maximum improvement of 1.04% compared with MAE under the same conditions, verifying the effectiveness of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The framework diagram of this application, which includes a student network and a teacher network, and the teacher network is updated by the student network in an exponential moving average (EMA) manner;

[0054] Figure 2 The algorithm flow of this application. DETAILED DESCRIPTION

[0055] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0056] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0057] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0058] The technical solutions provided in the embodiments of the present application involve technologies such as machine learning and natural language processing of artificial intelligence, which are specifically introduced and explained through the following embodiments.

[0059] The method proposed in this application is a typical method for learning remote sensing image representation through generative self-supervision. The framework of the model is as follows Figure 1 As shown in Figure 3, it consists of two key parts, namely the prediction loss guided masking and the unsupervised tokenizer component.

[0060] The model of this application consists of an unsupervised tokenizer, a teacher model and a student model. The unsupervised tokenizer uses the KMeans clustering method to generate discrete tokens and build a codebook, and the discrete tokens are used as the reconstruction target of the model. The teacher model and the student model have the same structure, both of which are composed of an encoder (f θt ,f θs ) and two decoders Composition, including The reconstruction loss used to predict image patches, d φ It is used to reconstruct the pixels of the covered patches, and the subscripts t and s represent the teacher model and the student model. The encoder and decoder are both composed of the ViTbackbone network. During the training process, the input remote sensing image I∈R H×W×C Divide into a series of non-overlapping patches (H, W) is the resolution of the remote sensing image, C is the number of channels, P is the number of original pixels contained in the patches, N = HW / P 2 Represents the number of patches. Then the remote sensing image is used as the teacher model encoder f θt The input of the encoder f θt The output intermediate latent feature vector h t As The student model uses the mask image patches generated by the teacher model as input to complete the image reconstruction. The teacher model and the student model work alternately, and the parameter θ of the student model is s The update is completed according to the back propagation of the loss function, and the parameters θ of the teacher model t The updated student model θ s Exponential Moving Average (EMA) obtained.

[0061] θ t <-mθ t +(1-m)θ s (1)

[0062] Where m represents the momentum coefficient.

[0063] In the current MIM method (Masked image modeling is a representative method in generative self-supervised models and is the prior art in this field), the model focuses on the pixel consistency between the original image and the predicted image, and pays too much attention to the local pixel details of the image. However, in fact, remote sensing images contain rich semantic information of ground objects. Ignoring this information will affect the effect of model representation learning. In this application, an unsupervised clustering method is used to generate discrete tokens for each image patch. These tokens represent some features of the image that are more abstract than pixels, and use them as the prediction target of the model, thereby prompting the model to learn advanced image representations.

[0064] Specifically, in this application, the unsupervised clustering method KMeans is used as a tokenizer, the original image dataset is segmented into non-overlapping patchesx as input, and the capacity of the generated codebook is regulated by the hyperparameter k. Considering the KMeans method as a function K, the codebookZ can be expressed as:

[0065] Z=K(x),Z={z1,z2,z3……z q},q∈[1,k] (2)

[0066] where codebook Z∈R K×D , D represents the feature dimension (e.g. 768). The codebook vectors {z1, z2, z3…zq} contained therein represent a cluster center, and in remote sensing images, at a deeper level, these cluster centers can represent the semantic feature categories in remote sensing images. Accordingly, the discrete tokent corresponding to each patch of each image can be expressed as:

[0067] t=argmin||xzi ||,z i ∈Z (3)

[0068] argmin(.) represents the nearest neighbor search, that is, the cluster center obtained by the nearest neighbor search for each patch can be used as its corresponding discrete token, which can be used as the prediction target of the subsequent model.

[0069] In traditional MIM methods, the masking strategy of remote sensing images mostly adopts a certain proportion of random masking. However, the pre-training tasks generated by this method still show information redundancy, and for multi-scale and multi-object remote sensing images, even if a large proportion of masking is performed, the remaining parts can still reveal the basic appearance of the objects (i.e., the unity of visual concepts), making it impossible for the model to learn a more complete representation of the objects during feature learning. In this application, N×N image patches are flattened to a length of N. 2 The sequence is used as the input of the teacher model and the encoder f θt , and encoding Complete the loss prediction work. Then calculate the importance of each cluster center based on the discrete token, and calculate the covering priority of each patch based on the loss value and cover it.

[0070] Use N×N image patchesx as the teacher model encoder Input, get the potential feature vector output by the encoder Then we use the decoder Perform loss prediction and predict the loss L pred Depend on The output value is calculated by taking the average value. The specific formula is as follows:

[0071]

[0072] Where Mean(.) represents the average operation. However, since remote sensing images have the characteristics of many and cluttered objects, different objects have different importance in the image. Based on the prediction loss, the importance of each type of discrete token can be calculated and used as an indicator of its importance in the current image. Therefore, the specific formula for the indicator of patches that should be covered is:

[0073]

[0074] The average prediction loss corresponding to each type of token represents the importance of the current image. This is to make patches with higher overall prediction losses more prominent, rather than just highlighting a single patch.

[0075] The masking strategy used in the model is to perform argsort(.) operation on the loss index T in descending order to obtain the reconstruction difficulty of the patches inside the remote sensing image, and intercept a certain proportion of the patches according to the sorting for masking. However, considering that if the target object is completely covered, it is very unreasonable to reconstruct the target object based only on the background of the remote sensing image, so some randomness is introduced. The final implementation adopts an overall masking of 75%, of which 25% is randomly masked, and the first 50% of the patches are intercepted according to the sorting for masking, which can be specifically expressed as:

[0076]

[0077] λ=argsort(T) (6)

[0078] Where λ represents the sequence of index T after argsort(.), RandomSubset represents random selection from the remaining patches, and the resulting matrix M∈{0,1} representing the coverage is an N×N two-dimensional matrix, where elements 0 and 1 represent uncovered and covered patches, respectively.

[0079] In the current mainstream generative SSL methods, the masked content is often the same as the reconstructed content, which is the original pixel value. This can achieve as much consistency as possible at the pixel level, but at the same time, higher-level features, such as semantic information, will be neglected, and at the same time, the model will focus on learning detailed pixels. In this application, the tokens obtained using unsupervised clustering methods aggregate high-level abstract features, and the loss function that measures the accuracy of pixel prediction is converted to the loss of the accuracy of predicting tokens.

[0080] Specifically, after the teacher model guides the masking, the unmasked remote sensing image patches are used as the student model encoder The input is passed through an image reconstruction encoder d φs , whose output Represents the original pixels obtained by image reconstruction. For each covered patch, this application uses a softmax classifier to predict the corresponding discrete token where x M represents the masked image, W c ∈R k×D ,b c ∈R k Represents the weight of the linear layer in the sotmax classifier. The pre-training goal is to maximize the log-likelihood of correctly classifying discrete tokens given a masked image.

[0081]

[0082] Where ν represents the entire training data set, x M Represents the masked image after masking according to the masking matrix M.

[0083] Loss prediction for the student model This application sets an additional training objective for loss prediction. However, when we measure whether the prediction loss is accurate, we cannot measure the prediction loss after the argsort(.) operation. and reconstruction loss The actual distance between them, and the argsort(.) operation is not differentiable, so we transform the problem into a relative loss problem. Specifically, for each patch in the remote sensing image, we can predict the relative size of the prediction loss value between each patch, and implicitly learn the ranking relationship of the prediction loss between patches. Finally, the training objective can be expressed as:

[0084]

[0085] in is from the decoder in the student model Output Represents the prediction loss value of the student model, where i, j = 1, 2, 3... N represents the different patches in the image, used to identify each patch. σ(z) = e z / (e z +1) represents the sigmoid activation function. and As two identifiers, they are used to identify the relative size between the prediction loss and the standard reconstruction loss. The specific formula is as follows:

[0086]

[0087] Among them, M i =M j =1 means that patches i and patches j are covered during training.

[0088] In summary, the final training goal of the model combined with the goal of loss prediction and the goal of image reconstruction can be expressed as:

[0089]

[0090] The first training goal is to make the model loss prediction more accurate, and the generated pre-training tasks imply less visual concepts of objects. The second training goal is to enable the model to accurately learn abstract semantic information. The two together constitute the training goal of the model.

[0091] This application uses the remote sensing image scene classification task to evaluate the representation learning results of the model. This application is tested on the following three remote sensing image classification public datasets: Aerial Image Dataset (AID), UC-Merced LandUse (UCM) and NWPU-RESISC45 (NWPU45) to evaluate the performance of the proposed model.

[0092] The AID dataset is a sample dataset collected from Google Earth. It includes 30 categories such as airports and bare land. Each sample category contains 220 to 440 images. The dataset covers the entire world, and the spatial resolution is about 0.5 to 8 meters. The UCM dataset is from the United States Geological Survey (USGS). It includes 21 categories such as agriculture and aircraft. The dataset covers the United States, and the spatial resolution is about 1 foot. The NWPU45 dataset was extracted from Google Earth by remote sensing image experts. It includes 45 categories such as airplanes and airports. The resolution of most images is between 30 and 0.2 meters. The following table shows more details of the AID, UCM, and NWPU45 datasets.

[0093] Table 1. Detailed description of three public datasets

[0094] Dataset AID UCM NWPU45 Year 2017 2010 2017 Resolution 0.5～8m 1foot 0.2～30m Class Number 30 21 45 Size 600*600 256*256 256*256 Amount of Data for SSL Pretraining 8500 1980 25200

[0095] 4.2Baseline and Evaluation

[0096] In order to verify the performance of the proposed model, this application uses 7 mask modeling methods with different masking strategies as baseline methods:

[0097] The MAE structure is simple. It uses an asymmetric encoder structure to reconstruct the covered image and adopts a certain proportion of random covering strategy.

[0098] The SimMIM method is similar to the MAE structure, both of which use an asymmetric encoder structure, but its masking strategy is based on the MAE, and the size of the masked image patches is an integer multiple of the size of the segmented image blocks.

[0099] The CAE method randomly masks some patches and uses the tokens output from the discrete tokenizer (such as d-VAE) as the prediction target of the model.

[0100] ConvMAE uses the blockwise masking strategy to randomly mask a certain proportion of image blocks in the smallest scale view, and upsamples accordingly, applying the masking strategy to a larger scale convolution view.

[0101] RingMo redesigned the pre-training task to address the problem of dense objects in complex remote sensing scenes. It adopted an incomplete masking strategy, which randomly masks image patches while retaining some random pixels in the masked image patches, thereby effectively retaining the feature information of small targets.

[0102] Based on MAE, SelectiveMAE proposes a selective masking strategy, which allows the model to pay more attention to the areas with higher information content in the image by dynamically and selectively masking image blocks. Unlike the traditional MAE method, SelectiveMAE does not uniformly mask all image blocks, but makes weighted selections based on the importance of image blocks, thereby avoiding invalid learning of redundant or low-information areas.

[0103] In addition, we use top-1 accuracy as an evaluation indicator of model performance, which represents the proportion of samples whose predicted categories completely match the true categories in the test dataset to the total number of samples, considering only the category with the highest probability predicted by the model.

[0104] In the data enhancement stage, the present application crops the original image to a size of 224×224 (RandomResizedCrop) and performs a random flip (RandomHorizontalFlip) operation.

[0105] In the model of this application, the teacher model and the student model use the same model architecture, and the encoder f θt / f θs And decoder All use the conventional ViT architecture, with ViT-base as the default.

[0106] In the pre-training stage, 600 epochs were trained uniformly with a batch size of 32. The Adam W optimizer

[45] was used, the weight decay was 0.05, the learning rate was warmed up for 40 epochs, the initial learning rate was 1.5e-4, and then decayed according to the cosine law. In the scene classification fine-tuning stage, the batch size was 32 and 300 epochs were trained uniformly. The quantitative results of the performance analysis are shown in Table 2 below.

[0107] Table 2. Comparison with the sota model on three scene classification datasets

[0108]

[0109]

[0110] In the performance analysis table, on the UC-Merced Land Use dataset, MAE performed best compared to other models, and the classification accuracy of the model of this application was second best. Our model framework achieved the best scene classification results on the other two datasets, which also verified the effectiveness of this application. It can be observed from the table that several other methods performed poorly on the UCM dataset, while the accuracy increased significantly on other datasets, which also means that the size of the dataset will significantly affect the model's ability to represent learning, and the backbone network transformer used in these methods is more suitable for large datasets. MAE reduces the impact of pre-training datasets on its model's ability to represent learning due to its simpler model structure, concise image processing steps and smaller amount of data. This application compensates for the negative impact of some dataset sizes due to the use of more accurate masking strategies and more abstract prediction targets. This is also a research direction that can be further deepened and improved in the future.

[0111] This application proposes a model framework for feature learning of remote sensing image mask modeling. First, starting from the construction of remote sensing image pre-training tasks, in order to solve the instance information leakage problem caused by random masking, combined with the teacher-student model framework, the teacher model calculates the prediction loss based on the image patches, and the student model completes the masking strategy. Secondly, by converting the prediction target into a discrete token obtained by clustering, the prediction target of the model is transformed from low-level pixel information to higher-level abstract information, avoiding the representation learning bias caused by over-emphasizing detailed pixel information. This application uses the trained basic model to fine-tune the downstream scene classification task, compares several state-of-the-art image mask modeling models, and verifies the superiority of the proposed framework.

[0112] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0113] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0114] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0115] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment, and any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A remote sensing image self-supervised representation learning method, characterized in that: The following steps are involved: Obtain remote sensing image datasets; The entire remote sensing image dataset is used as input, and an unsupervised tokenizer is used to generate tokens representing abstract semantic information in the image, which are used as the prediction target of the model. In order to reduce the ground object information provided by the pre-training task, the reconstruction loss is predicted by the teacher model, and the prediction loss is constructed based on the importance of the token. The pre-training tasks are generated by sorting the prediction losses and passed to the student model, which then completes the image reconstruction. Output classified remote sensing image.

2. The remote sensing image self-supervised representation learning method according to claim 1, characterized in that: The unsupervised tokenizer uses the KMeans clustering method to generate discrete tokens and construct a codebook, and uses the discrete tokens as the reconstruction target of the model; The teacher model and the student model have the same structure, both of which are composed of a teacher encoder f θt , student encoder f θs And two decoders Composition, of which The reconstruction loss used to predict image patches, d φ Used for pixel reconstruction of covered patches. The subscripts t and s represent the teacher model and the student model. Both the encoder and the decoder are composed of the ViTbackbone network.

3. The remote sensing image self-supervised representation learning method according to claim 2, characterized in that: During the training process, the input remote sensing image I∈R H×W×C Divide into a series of non-overlapping (H, W) is the resolution of the remote sensing image, C is the number of channels, P is the number of original pixels contained in the patches, N = HW / P 2 , represents the number of patches; Then the remote sensing image is used as the teacher model encoder f θt The input of the encoder f θt The output intermediate latent feature vector h t As The input of is used to predict the reconstruction difficulty of each patch and guide the generation of the covering strategy; The student model uses the masked image patches generated by the teacher model as input to complete the image reconstruction. The teacher model and the student model work alternately, and the parameters θ of the student model are s The update is completed according to the back propagation of the loss function, and the parameters θ of the teacher model t The updated student model θ s Exponential Moving Average EMA obtains: i t <-mθ t +(1-m)θ s Where m represents the momentum coefficient.

4. The remote sensing image self-supervised representation learning method according to claim 3, characterized in that: The unsupervised clustering method KMeans is used as a tokenizer. The original image dataset is divided into non-overlapping patchesx as input. The capacity of the generated codebook is regulated by the hyperparameter k. The KMeans method is a function K, so the codebookZ is expressed as: Z=K(x),Z={z1,z2,z3……z q },q∈[1,k] where codebook Z ∈ R K×D , D represents the feature dimension, and each codebook vector {z1, z2, z3…zq} contained therein represents a cluster center, and these cluster centers represent the semantic feature categories in the remote sensing image; accordingly, the discrete tokent corresponding to each patch of each image is expressed as: t=argmin||x-z i ||,z i ∈Z argmin(.) represents the nearest neighbor search, that is, for each patch, the cluster center obtained by the nearest neighbor search is used as its corresponding discrete token, which is used as the prediction target of the subsequent model.

5. The remote sensing image self-supervised representation learning method according to claim 4, characterized in that: The N×N image patches are flattened to a length of N 2 The sequence is used as the input of the teacher model and the encoder f θt , and encoding Complete the loss prediction work; then calculate the importance of each cluster center based on the discrete token, and calculate the covering priority of each patch based on the loss value and cover it; Use N×N image patchesx as the teacher model encoder Input, get the potential feature vector output by the encoder Then use the decoder Perform loss prediction and predict the loss L pred Depend on The output value is calculated by taking the average value. The specific formula is as follows: Where Mean(.) represents the average operation; based on the predicted loss, the importance of each type of discrete token is calculated and used as an indicator of its importance in the current image. The specific formula for the indicator of patches that should be covered is: Represents the average value of the prediction loss corresponding to each type of token, which represents the importance of the current image.

6. The remote sensing image self-supervised representation learning method according to claim 5, characterized in that: The masking strategy is to perform argsort(.) operation on the loss index T in descending order to obtain the reconstruction difficulty of the patches inside the remote sensing image, intercept a certain proportion of the patches according to the sorting for masking; and introduce some randomness, that is, the overall masking is 75%, of which 25% is randomly masked, and the first 50% of the patches are intercepted according to the sorting for masking, which is specifically expressed as: Where λ represents the sequence of index T after argsort(.), RandomSubset represents random selection from the remaining patches, and the resulting matrix M∈{0,1} representing the coverage is an N×N two-dimensional matrix, where elements 0 and 1 represent uncovered and covered patches, respectively.

7. The remote sensing image self-supervised representation learning method according to claim 6, characterized in that: After masking guided by the teacher model, the unmasked remote sensing image patches are used as the student model encoder The input is passed through an image reconstruction encoder d φs , whose output Represents the original pixels obtained by image reconstruction; For each covered patch, a softmax classifier is used to predict the corresponding discrete where x M represents the masked image, W c ∈R k×D ,b c ∈R k Represents the weight of the linear layer in the sotmax classifier; The pre-training goal is to maximize the log-likelihood of correctly classifying discrete tokens given a masked image: Where ν represents the entire training data set, x M Represents the masked image that has been masked according to the masking matrix.

8. The remote sensing image self-supervised representation learning method according to claim 7, characterized in that: Loss prediction for the student model Set up an additional training objective for loss prediction: For each patch in the remote sensing image, predict the relative size of the prediction loss value between each patch, and implicitly learn the ranking relationship of the prediction loss between patches; Finally, the training objective is expressed as: in and is from the decoder in the student model The i-th and j-th output Represents the prediction loss value of the student model, where i, j = 1, 2, 3... N represents the different patches in the image, used to identify each patch; σ(z) = e z / (e z +1) represents the sigmoid activation function; and As two identifiers, they are used to identify the relative size between the prediction loss and the standard reconstruction loss. The specific formula is as follows: Among them, M i =M j =1 means that patches i and patches j are covered during training. is the loss of model image reconstruction; The final training goal of the model combined with the goal of loss prediction and the goal of image reconstruction is finally expressed as: The training objective of the model forces the pre-training tasks generated by the model to imply less visual concepts of objects; The training goal is to enable the model to learn abstract semantic information.

Images