A remote sensing image self-supervised representation learning method

Abstract

The application discloses a kind of remote sensing image self-supervised representation learning method, comprising the following steps: obtaining remote sensing image dataset;Whole remote sensing image dataset is used as input, and the token representing abstract semantic information in image is generated using unsupervised tokenizer, and it is used as the prediction target of model;In order to reduce the ground object information provided by pre-training task, the reconstruction loss is predicted by teacher model, the prediction loss is constructed by combining the importance of token, the pre-training task is generated according to the prediction loss ordering and the pre-training task is passed to student model, and the image reconstruction is completed by student model;Output classified remote sensing image.This application reduces the influence of noise in original pixel, and avoids that model is limited to learn low-level pixel features.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing, and particularly relates to a remote sensing image self-supervised representation learning method. BACKGROUND

[0002] Self-supervised representation learning (SSL) is a new pre-training paradigm derived from traditional supervised representation learning, aiming at the strong dependence of supervised representation learning on high-quality labeled data and the problem of easy overfitting. This paradigm inputs a large amount of unlabeled data set in the pre-training process, and then uses a small amount of labeled data set for fine-tuning according to different downstream tasks. The purpose is to use only a small amount of labeled data to make the pre-trained model migrate to the downstream task performance as much as possible, even exceeding the supervised learning. Obviously, self-supervised representation learning can make the model free from the dependence on large-scale labeled data set, and make the obtainable performance of the interpretable model become a reality, which is in line with the current situation of the field of remote sensing, i.e., the vigorous development of satellite technology and the explosive growth of unlabeled remote sensing image data set, so that self-supervised representation learning has achieved superior performance in the field of scene classification and target detection of remote sensing image data set.

[0003] Self-supervised representation learning methods can be divided into two main categories: contrastive learning and generative learning. Contrastive learning is a discriminative method, aiming to use a similarity measure to achieve the purpose of pulling away negative samples and pulling close positive samples in the training process. Generative learning usually adopts the form of image-encoder encoding-potential feature vector-decoder decoding to reconstruct the input image, among which the mask image modeling is the most widely used and superior representative method, which has attracted extensive attention. This kind of method allows the image to be partially covered, and the remaining image not covered is input into the model as known information, prompting the model to predict the specific content of the covered image, so as to achieve the purpose of image feature learning. Many existing examples, such as MAEBEiT, etc., are all following the paradigm of mask image modeling, and have achieved great success.

[0004] However, due to the higher redundancy of remote sensing images compared with natural images, and the diversity of scales and ground object categories, the existing mask image modeling method still has the following problems: 1) overemphasizing pixel details. The prediction target of the model is mostly focused on the original pixel value, and the training target excessively pursues the absolute similarity between the original pixel and the reconstructed pixel, resulting in that the model learns to focus on learning the low-level features of local pixels in the image, and is sensitive to noise in remote sensing images. 2) The pre-training task is mainly generated by a random masking strategy, which still exposes the instance features of ground objects in remote sensing images. Influenced by the pre-training task hints, the model representation learning cannot learn the deeper features of the images. SUMMARY

[0005] Based on the above problems, the present application proposes a MIM pre-training framework for loss-guided generation of pre-training tasks. The framework consists of three parts: an unsupervised tokenizer for integrating high-level abstract features of images, a teacher model for guiding the construction of a masking pre-training task, and a student model for image reconstruction. Existing models in the field of remote sensing often take pixels as the prediction target, which is easily affected by noise in the original image and limits the model to learning low-level pixel features. Therefore, the entire remote sensing image dataset is taken as input, and the unsupervised tokenizer is used to generate tokens representing abstract semantic information in the image, which are used as the prediction target of the model, thereby ensuring that the model learns high-level image features without losing low-level pixel details. Due to the multi-scale and multi-terrain features of remote sensing images, the redundant features of the image are greatly amplified, and even with a large proportion of masking, the existing random masking strategy is unlikely to accurately mask the terrain patches. The remaining unmasked views of the instance cause the exposure of instance features, i.e., a reconstruction hint to the model. For the model, this type of pre-training task still retains a lot of terrain visual information, and the features learned by the model are greatly inhibited. In order to reduce the terrain information provided by the pre-training task, it is assumed that a higher reconstruction loss value represents a rich amount of information. The teacher model predicts the reconstruction loss, constructs a prediction loss combining the importance of the token, generates pre-training tasks according to the prediction loss, and passes the pre-training tasks to the student model, which completes the image reconstruction task.

[0006] In particular, the remote sensing image self-supervised representation learning method disclosed in the present application comprises the following steps:

[0007] An image dataset is obtained.

[0008] The entire remote sensing image dataset is taken as input, and an unsupervised tokenizer is used to generate tokens representing abstract semantic information in the image, which are used as the prediction target of the model.

[0009] In order to reduce the terrain information provided by the pre-training task, the teacher model predicts the reconstruction loss, constructs a prediction loss combining the importance of the token, generates pre-training tasks according to the prediction loss, and passes the pre-training tasks to the student model, which completes the image reconstruction.

[0010] The classified remote sensing image is output.

[0011] Preferably, the unsupervised tokenizer uses the KMeans clustering method to generate discrete tokens and construct a codebook, and the discrete tokens are used as the reconstruction target of the model.

[0012] The teacher model and the student model have the same structure, and both are composed of a teacher encoder fθt , student encoder f θs and two decoders comprising a reconstruction loss for predicting image patches, d φ for pixel reconstruction of occluded patches, subscript t, s represents the teacher model and student model; both the encoder and the decoder are composed of a ViT backbone network.

[0013] Preferably, during the training process, the input remote sensing image I ∈ R H×W×C is first divided into a series of non-overlapping patches (H, W) is the resolution of the remote sensing image, C is the number of channels, P is the number of original pixels contained in the patch, N = HW / P 2 , representing the number of patches;

[0014] Then the remote sensing image is input into the teacher model encoder f θt , the intermediate latent feature vector h θt output by the encoder f t is input into the teacher model f to predict the reconstruction difficulty of each patch and guide the generation of the occlusion strategy;

[0015] The student model takes the occluded image patches generated by the teacher model as input to complete the image reconstruction work; the teacher model and the student model work alternately, and the parameters θ s of the student model are updated according to the loss function, while the parameters θ t of the teacher model are updated by the updated student model θ s Exponential moving average EMA is obtained:

[0016] θ t <-mθ t +(1-m)θ s

[0017] Where m represents the momentum coefficient.

[0018] Preferably, an unsupervised clustering method KMeans is used as tokenizer, and the original image dataset is segmented into non-overlapping patches x as input, and the capacity size of the generated codebook is regulated by the hyperparameter k; the KMeans method is a function K, then the codebook Z is represented as:

[0019] Z = K(x), Z = {z1, z2, z3…z q}, q ∈ [1, k]

[0020] where codebook Z ∈ R K×D , D represents the feature dimension, and each codebook vector {z1, z2, z3…zq} contained therein represents a cluster center, which represents a semantic feature category in the remote sensing image; correspondingly, the discrete token corresponding to each patch of the image is represented as:

[0021] t = argmin ||x - z i ||, z i ∈ Z

[0022] argmin(.) represents nearest neighbor search, i.e., the cluster center obtained by using nearest neighbor search for each patch is used as the discrete token corresponding thereto, which serves as the prediction target of the subsequent model.

[0023] Preferably, the N×N image patch is flattened into a sequence with a length of N 2 , which is used as the input of the teacher model, and the loss prediction is completed by using an encoder f θt and an encoder ; then the importance of each cluster center is calculated according to the discrete token, the covering priority of each patch is calculated in combination with the loss value, and the patch is covered;

[0024] The N×N image patch x is used as the input of the teacher model encoder , and the latent feature vector output by the encoder is obtained; then a decoder is used to predict the loss, and the prediction loss L pred is calculated by the output value of and taking the average value, and the specific formula is as follows:

[0025]

[0026] where Mean(.) represents the average value operation; on the basis of the prediction loss, the importance degree of each type of discrete token is calculated, which serves as an index of the importance degree in the current image, and the specific formula of the index of the patch to be covered is as follows:

[0027]

[0028] represents the average value of the prediction loss corresponding to each type of token, and represents the importance degree of the current image.

[0029] Preferably, the masking strategy is to argsort(.) operation in descending order of loss indicator T to obtain the reconstruction difficulty of patches inside the remote sensing image, and a certain proportion of patches are masked according to the order; and partial randomness is introduced, that is, the total masking is 75%, in which 25% is randomly masked, and the first 50% of the patches are masked according to the order, which is specifically represented as:

[0030]

[0031] λ = argsort(T)

[0032] Where λ represents the sequence of indicator T after the argsort(.) sorting, and RandomSubset represents random selection in the remaining patches, so that the matrix M ∈ {0, 1} representing the masking is obtained, which is an N × N two-dimensional matrix, where elements 0 and 1 represent unmasked and masked patches respectively.

[0033] Preferably, after being guided by the teacher model, the unmasked remote sensing image patches are used as the input of the student model encoder , and an image reconstruction encoder d φs is used to output representing the original pixels obtained by image reconstruction.

[0034] For each masked patch, a softmax classifier is used to predict the corresponding discrete token Where x M represents the masked image, W c ∈ R k×D ,b c ∈ R k represents the weight of the linear layer in the sotmax classifier.

[0035] The pre-training goal is to maximize the log-likelihood of correctly classified discrete tokens given the masked image:

[0036]

[0037] Where ν represents the entire training data set, and x M represents the masked image according to the masking matrix M.

[0038] Preferably, for the student model loss prediction An additional training goal for loss prediction is set:

[0039] For each patch in the remote sensing image, the relative size of the prediction loss value between each patch is predicted, and the ranking relationship of the prediction loss between the patches is learned implicitly;

[0040] Finally, the training target is represented as:

[0041]

[0042] Wherein and is the output of the decoder of the student model represent the prediction loss value of the student model, wherein i, j = 1, 2, 3 … N represent the different patch indexes in the image, used to identify each patch; σ(z) = e z / (e z +1) represents the sigmoid activation function; and as two identifiers, used to identify the relative size between the prediction loss and the standard reconstruction loss, the specific formula is as follows:

[0043]

[0044] Wherein M i =M j =1 indicates that the patches i and the patches j are both covered in the training process, is the loss of image reconstruction of the model;

[0045] The final training target of the model combines the loss prediction target and the image reconstruction target and is finally represented as:

[0046]

[0047] The training target of the model promotes the pre-training task generated by the model to be less suggestive of the visual concept of the ground object; The training target of the model is to make the model learn abstract semantic information.

[0048] Through the new pre-training framework, the basic pre-training model is obtained, and the representation learning ability of the model is verified in the downstream task of scene classification. The model representation learning performance is evaluated in three remote sensing image scene classification public data sets Aerial Image Dataset (AID), UC-Merced Land Use (UCM) and NWPU-RESISC45 (NWPU45).

[0049] The beneficial effects of the present application are as follows:

[0050] (1) The unsupervised method is used as a tokenizer, and the tokens of the high-level abstract features of the collection dataset are used as the prediction target, which is to reduce the influence of noise in the original pixels and avoid the limitation of the model to learn low-level pixel features.

[0051] (2) The prediction reconstruction loss guided masking strategy is proposed, and the self-supervised representation learning framework of the teacher model guided masking-student model reconstructed image is built, which is to reduce the problem that the random masking strategy is easy to retain the visual concept identity of the ground features in the remote sensing image, and gives too much hint to the model feature learning.

[0052] (3) The verification on the public dataset shows that the model of the present application has been improved by 1.04% compared with MAE under the same conditions, which verifies the effectiveness of the strategy. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The framework diagram of the present application, which contains a student network and a teacher network, the teacher network is updated by the student network in the form of exponential moving average (EMA);

[0054] Figure 2 The algorithm flow of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described below in conjunction with the drawings, but in no way limits the present application, any transformation or replacement based on the teaching of the present application belongs to the protection scope of the present application.

[0056] Artificial intelligence is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0057] Nature Language Processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, the research in this field will involve natural language, i.e. the language used in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc.

[0058] The technical scheme provided by the embodiments of the present application relates to machine learning and natural language processing of artificial intelligence, and is specifically introduced and explained by the following embodiments.

[0059] The method proposed in the present application is a typical method of remote sensing image representation learning through generative self-supervision. The framework of the model is as shown in Figure 1 , which consists of two key parts, i.e. prediction loss guided masking and unsupervised tokenizer component.

[0060] The model of the present application is composed of an unsupervised tokenizer, a teacher model and a student model, wherein the unsupervised tokenizer adopts a KMeans clustering method to generate discrete tokens and construct a codebook, and the discrete tokens are used as the reconstruction target after the model. The teacher model and the student model have the same structure, and are both composed of an encoder (f θt ,f θs ) and two decoders , wherein is used to predict the reconstruction loss of image patches, d φ is used to reconstruct the pixels of the masked patches, and the subscripts t and s represent the teacher model and the student model. The encoder and the decoder are both composed of a ViTbackbone network. In the training process, first, the input remote sensing image I∈R H×W×C is divided into a series of non-overlapping patches (H,W) is the resolution of the remote sensing image, C is the number of channels, P is the number of original pixels contained by the patch, and N=HW / P 2 represents the number of patches. Then the remote sensing image is input into the teacher model encoder f θt , and the intermediate latent feature vector h θt output by the encoder f θt is used as the input of the decoder f t . input from the teacher model, so as to predict the reconstruction difficulty of each patch and guide the generation of the masking strategy. The student model takes the teacher model guided generated masking image patches as input to complete the image reconstruction work. The teacher model and the student model work alternately, and the parameters θ s According to the loss function, the update is completed by back propagation, and the parameters θ t The updated student model θ s The exponential moving average (EMA) is obtained.

[0061] θ t <-mθ t +(1-m)θ s (1)

[0062] Where m represents the momentum coefficient.

[0063] In the current MIM method (Masked image modeling as a representative method in generative self-supervised model), the model focuses on the pixel consistency of the original image and the predicted image, and pays too much attention to the local pixel details of the image. However, in fact, remote sensing images contain rich semantic information of ground objects, and ignoring these information will affect the effect of model representation learning. In this application, an unsupervised clustering method is used to generate discrete tokens for each image patch, which represent some more abstract features of the image than pixels, and are used as the prediction target of the model, so as to promote the model to learn high-level image representation.

[0064] Specifically, in this application, the unsupervised clustering method KMeans is used as tokenizer, and the original image dataset is divided into non-overlapping patches x as input, and the capacity size of the generated codebook is regulated by the hyperparameter k. The KMeans method is regarded as a function K, and then the codebook Z can be represented as:

[0065] Z=K(x),Z={z1,z2,z3……z q},q∈[1,k] (2)

[0066] Where codebook Z∈R K×D , D represents the feature dimension (for example, 768). The codebook vectors {z1, z2, z3……zq} contained therein all represent a clustering center, and in remote sensing images, more deeply, these clustering centers can represent semantic feature categories in remote sensing images. Accordingly, the discrete token t corresponding to each patch of each image can be represented as:

[0067] t=argmin||x-zi ||,z i ∈Z (3)

[0068] argmin(.) represents the nearest neighbor search, that is, the cluster center obtained by using the nearest neighbor search for each patch can be used as its corresponding discrete token, which can be used as the prediction target of the subsequent model.

[0069] In the traditional MIM method, the mask strategy of the remote sensing image mostly uses a certain proportion of random mask, however, the pre-training task generated by this method still shows information redundancy, and for the multi-scale multi-terrain remote sensing image, even if a large proportion of covering is performed, the remaining part can still reveal the basic appearance of the terrain (that is, the unity of the visual concept), so that the model cannot learn more complete terrain representation in the process of feature learning. In the present application, the N*N image patch is flattened into a sequence with a length of N 2 , which is used as the input of the teacher model, and the loss prediction is completed by using the encoder f θt and the encoding . Then the importance of each cluster center is calculated according to the discrete token, and the covering priority of each patch is calculated combined with the loss value and the covering is performed.

[0070] The N*N image patch x is used as the input of the teacher model encoder , and the latent feature vector output by the encoder is obtained. Then we use the decoder to predict the loss L pred , which is calculated by the output value of and the average value, and the specific formula is as follows:

[0071]

[0072] Where Mean(.) represents the average value operation. However, since the remote sensing image has the characteristics of multiple and chaotic terrain, the importance of different terrains in the image is not the same, and the importance of each class of discrete token can be calculated on the basis of the predicted loss, which is used as an indicator of the importance in the current image, so the specific formula of the indicator of the patch that should be covered is:

[0073]

[0074] represents the average value of the predicted loss of each class of token, and represents the importance of the current image, which is to make the patches with a higher overall predicted loss more prominent, rather than just highlighting a single patch.

[0075] The masking strategy adopted in the model is to perform argsort(.) operation on the loss indicator T in descending order to obtain the reconstruction difficulty of the patches inside the remote sensing image, and to mask a certain proportion of patches according to the order. However, considering that it is unreasonable to reconstruct the target ground object only according to the background of the remote sensing image if the target ground object is completely masked, partial randomness is introduced. Finally, the implementation adopted is to mask 75% in total, of which 25% is randomly masked, and the first 50% of the patches are masked according to the order, which can be specifically represented as:

[0076]

[0077] λ = argsort(T) (6)

[0078] Where λ represents the sequence after the argsort(.) sorting of the indicator T, and RandomSubset represents random selection in the remaining patches. The matrix M obtained by representing the masking is ∈ {0, 1}, which is an N x N two-dimensional matrix, where elements 0 and 1 represent unmasked and masked patches, respectively.

[0079] In the current mainstream generative SSL method, the content to be masked is often the same as the reconstructed content, which is the original pixel value. In this way, consistency is achieved at the pixel level as much as possible, but at the same time, learning at a higher level of features, such as semantic information, is ignored, and at the same time, the model focuses on learning detailed pixels. In the present application, the token obtained using an unsupervised clustering method aggregates high-level abstract features, and converts the loss function measuring the accuracy of pixel prediction into the accuracy of predicting tokens.

[0080] Specifically, after being guided by the teacher model, the unmasked remote sensing image patches are input into the student model encoder , and after passing through an image reconstruction encoder d φs , the output represents the original pixels obtained by image reconstruction. For each masked patch, the present application uses a softmax classifier to predict the corresponding discrete token Where x M represents the masked image, W c ∈ R k×D ,b c ∈ R k represents the weight of the linear layer in the softmax classifier. The pre-training target is to maximize the log-likelihood of correctly classifying discrete tokens given the masked image.

[0081]

[0082] where v represents the whole training dataset, x M represents the occluded image after occlusion according to the occlusion matrix M.

[0083] Prediction of student model loss An additional training objective for loss prediction is set in this application. The loss of model image reconstruction However, when we measure whether the predicted loss is predicted accurately, we cannot measure the actual distance between the predicted loss and the reconstruction loss , and the argsort(.) operation is not differentiable, so we convert the problem into a relative loss problem. Specifically, for each patch in the remote sensing image, we can predict the relative size of the predicted loss value between each patch, and implicitly learn the ordering relationship of the predicted loss between the patches. Finally, the training objective can be represented as:

[0084]

[0085] where is the output from the decoder of the student model represents the predicted loss value of the student model, where i, j = 1, 2, 3 … N represent the different patch indices in the image, used to identify each patch. σ(z) = e z / (e z +1) represents the sigmoid activation function. and as two identifiers, used to identify the relative size between the predicted loss and the standard reconstruction loss, the specific formula is as follows:

[0086]

[0087] where M i = M j = 1 indicates that patches i and patches j are both occluded during training.

[0088] In summary, the final training objective of the model combines the loss prediction objective and the image reconstruction objective, which can finally be represented as:

[0089]

[0090] The first training target prompts the model to lose the prediction more accurately, and the generated pre-training task less implies the visual concept of the object. The second training target is to make the model accurately learn the abstract semantic information, and the two together constitute the training target of the model.

[0091] The present application uses the scene classification task of remote sensing images to evaluate the representation learning results of the model. The performance of the proposed model is evaluated on the following three public remote sensing image classification datasets: Aerial Image Dataset (AID), UC-Merced Land Use (UCM) and NWPU-RESISC45 (NWPU45).

[0092] The AID dataset is a sample dataset collected from Google Earth, which includes 30 categories such as airports and bare land, each sample category contains 220 to 440 images, the dataset covers the global range, and the spatial resolution is about 0.5-8m. The UCM dataset is derived from the United States Geological Survey (USGS), which includes 21 categories such as agriculture and aircraft, the dataset covers the United States region, and the spatial resolution is about 1 foot. The NWPU45 dataset is extracted from Google Earth by remote sensing image experts, which includes 45 categories such as aircraft and airports, and the resolution of most images is between 30-0.2m. The following table shows more details of the AID, UCM and NWPU45 datasets.

[0093] Table 1. Detailed description of three public datasets

[0094] Dataset AID UCM NWPU45 Year 2017 2010 2017 Resolution 0.5～8m 1 foot 0.2～30m Class Number 30 21 45 Size 600*600 256*256 256*256 Amount of Data for SSL Pretraining 8500 1980 25200

[0095] 4.2Baseline and Evaluation

[0096] In order to verify the performance of the proposed model, the present application uses 7 mask modeling methods with different masking strategies as baseline methods:

[0097] MAE structure is simple, which uses an asymmetric encoder structure to reconstruct the masked image, and adopts a certain proportion of random masking strategy.

[0098] The SimMIM method is similar to the MAE structure, both of which use an asymmetric encoder structure, but its masking strategy is based on MAE, and the size of the masked image patches is an integer multiple of the size of the segmented image blocks.

[0099] The CAE method uses tokens output from a discrete tokenizer (such as d-VAE) as the prediction target of the model by randomly masking part of the patches.

[0100] ConvMAE uses a blockwise masking strategy to randomly mask a certain proportion of image patches in the smallest scale view and upsamples accordingly, applying the masking strategy to larger scale convolutional views.

[0101] RingMo redesigns the pre-training task for the problem of dense ground objects in complex remote sensing scenes. It adopts an incomplete masking strategy that randomly masks image patches while retaining some random pixels in the masked image patches, effectively retaining the feature information of small targets.

[0102] SelectiveMAE proposes a selective masking strategy based on MAE, which dynamically and selectively masks image patches to enable the model to focus more on areas with high information content in the image. Unlike traditional MAE methods, SelectiveMAE does not uniformly mask all image patches, but rather selects them based on their importance, thereby avoiding ineffective learning of redundant or low-information areas.

[0103] In addition, we use top-1 accuracy as the evaluation indicator of model performance, which represents the proportion of samples whose predicted class completely matches the true class on the test dataset, only considering the class with the highest probability predicted by the model.

[0104] In the data augmentation phase, the original image is cropped to 224x224 size (RandomResizedCrop) and randomly flipped (RandomHorizontalFlip) operation is performed.

[0105] In the model of the present application, the teacher model and the student model use the same model architecture, the encoder f θt / f θs and the decoder all adopt the conventional ViT architecture, which is ViT-base by default.

[0106] In the pre-training phase, 600 epochs are uniformly trained with a batchsize of 32, and the Adam W optimizer

[45] is used with a weight decay of 0.05. The learning rate is preheated for 40 epochs, with an initial learning rate of 1.5e-4, and then decays according to the cosine law. In the scene classification fine-tuning phase, the batchsize is 32, and 300 epochs are uniformly trained. The quantitative results of performance analysis are shown in Table 2 below.

[0107] Table 2. Comparison with sota models on three scene classification datasets

[0108]

[0109]

[0110] In the performance analysis table, MAE performs best on the UC-Merced Land Use dataset, and the classification accuracy of the model of the present application is suboptimal. On the other two datasets, the model framework of the present application achieves the best scene classification result, which also verifies the effectiveness of the present application. It can be observed from the table that several other methods perform poorly on the UCM dataset, but the accuracy significantly increases on the other two datasets, which also represents that the size of the dataset will significantly affect the ability of model representation learning, and the backbone network transformer used in these methods is more suitable for large datasets. MAE reduces the influence of the pre-training dataset on the model representation learning ability due to its simpler model structure, simple image processing steps, and smaller amount of data. And the present application makes up for the negative effects of the size of the dataset due to the use of a more accurate masking strategy and a more abstract prediction target, which is also a research direction that can be further improved in the future.

[0111] The present application proposes a model framework for feature learning of remote sensing image image mask modeling. First, starting from the construction of the remote sensing image pre-training task, aiming at the instance information leakage problem caused by random masking, combining the teacher-student model framework, the teacher model calculates the prediction loss according to the image patches, and the student model completes the masking strategy. Secondly, by changing the prediction target into discrete tokens obtained by clustering, the prediction target of the model is changed from low-level pixel information to higher-level abstract information, avoiding the representation learning deviation caused by excessive emphasis on detailed pixel information. The present application uses the trained basic model to fine-tune the downstream scene classification task, compares several state-of-the-art image mask modeling models, and verifies the superiority of the proposed framework.

[0112] The word "preferred" is used herein as a term of art to denote features that, while not necessarily advantageous, are nevertheless desirable and / or beneficial. Any aspect or design described herein as "preferred" is not necessarily to be construed as being advantageous over other aspects or designs. Rather, the use of the word "preferred" is intended to present a concept in a particular manner. The term "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise, or as is clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. However, in the alternative, the phrase "X employs A or B" can be construed to mean an exclusive "or" if the specification states such intention, or if such intention is otherwise clear from the context.

[0113] Moreover, although the present disclosure has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and accompanying drawings. The present disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, blocks, etc.) the terms (including a reference to a "means") used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the present disclosure. In addition, while a particular feature of the disclosure can have been disclosed with respect to only one of several implementations, such feature can be combined with one or other features of the other implementations as can be desired and advantageous for any given or particular application. Furthermore, structures and functionality presented as discrete components in the exemplary configurations can be implemented as a combined structure or component. Such modifications, changes, variations, alternatives, combinations, sub-combinations, and equivalents are therefore contemplated as being within the scope of the present disclosure. It will be appreciated that the specified systems or devices referred to herein can be considered to be generally representative of a corresponding type of system or device, and therefore the specific disclosure of such systems or devices is not intended to limit the scope of the present disclosure to the specific implementations described herein. Moreover, although the terms "include", "have", "with", or variations such terms are used in the detailed description and / or in the claims, such terms are intended to be inclusive in a manner similar to the term "comprise" as an open transition such that figures are stated to "include", "have", "with", or "comprise" something have the same open meaning that the term "comprise" has in this context.

[0114] The various functional units in the embodiments of the present application can be integrated in one processing module, or each unit can be physically present separately, or a plurality of or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. The above-mentioned devices or systems can execute the storage method in the corresponding method embodiments.

[0115] In summary, the above embodiments are one embodiment of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. A method for remote sensing image self-supervised representation learning, characterized in that, The method comprises the following steps: Obtain a remote sensing image dataset; Use an unsupervised tokenizer to generate tokens representing abstract semantic information in the image as a prediction target of the model by taking the entire remote sensing image dataset as input; To reduce the ground object information provided by the pre-training task, a teacher model is used to predict reconstruction loss, a prediction loss is constructed by combining the importance of the tokens, the pre-training task is generated in order of the prediction loss, and the pre-training task is passed to a student model to complete image reconstruction; Output the classified remote sensing image; The unsupervised tokenizer adopts a KMeans clustering method to generate discrete tokens and construct a codebook, and the discrete tokens are used as the reconstruction target of the model; The teacher model and the student model have the same structure and are composed of a teacher encoder , a student encoder , and two decoders , wherein a reconstruction loss for predicting image patches, pixel reconstruction for occluded patches, the subscript represents the teacher model and the student model; the encoder and the decoder are composed of a ViT backbone network; During the training process, the input remote sensing image is first divided into a series of non-overlapping patches , is the resolution of the remote sensing image, is the number of channels, is the number of original pixels contained in the patch, , represents the number of patches;​ Then, remote sensing imagery is used as the teacher model encoder. Input, encoder Output intermediate latent feature vector As The input is used to predict the reconstruction difficulty of each patch and guide the generation of a masking strategy; The student model takes the generated occlusion image patches guided by the teacher model as input to complete image reconstruction; the teacher model and the student model work alternately, and the parameters of the student model are updated According to the loss function, the parameters of the teacher model are updated The updated student model Exponential moving average EMA is obtained: wherein represents the momentum coefficient. 2.The method of claim 1, wherein, Using the unsupervised clustering method KMeans as tokenizer, the original image dataset is split into non-overlapping patches As input, the hyperparameters The capacity size of the generated codebook is regulated; the KMeans method is a function , then the codebook is represented as: wherein codebook , represents a feature dimension, wherein each codebook vector contained in the codebook represents a cluster center, which represents a semantic feature category in the remote sensing image; correspondingly, the discrete token corresponding to each patch of the image is represented as: Representing the nearest neighbor search, i.e. using the cluster center obtained by the nearest neighbor search for each patch as its corresponding discrete token, which serves as the prediction target for the model later on. 3.The method of claim 2, wherein, The image patches of are flattened into sequences with length , which are input into the teacher model, and loss prediction is completed by using the encoder and the encoder ; then the importance of each cluster center is calculated according to the discrete token, and the covering priority of each patch is calculated combined with the loss value and covering is performed; Will Image patches As a teacher model encoder The input is used to obtain the latent feature vector of the encoder output. Then use the decoder Perform loss prediction, predict loss Depend on The output value is calculated by averaging the values, and the specific formula is as follows: wherein represents the average operation; on the basis of the prediction loss, the importance of each type of discrete token is calculated, which is used as an indicator of the importance in the current image, and the specific formula of the indicator of the patches that should be covered is: The prediction loss corresponding to each token is averaged, representing the importance of the current image. 4.The method of claim 3, wherein, The covering strategy is for a loss index In descending order Operation to obtain the reconstruction difficulty of the patches inside the remote sensing image, a certain proportion of patches are intercepted according to the order to cover; and introduce partial randomness, that is, the total covering is 75%, in which 25% is randomly covered, and the first 50% of the patches are covered according to the order, which is specifically represented as: ； ； wherein representative index experienced The sequence after sorting, RandomSubset represents randomly selecting in the remaining patches, so the representative covering matrix M e {0,1} is obtained, which is a two-dimensional matrix, where elements 0 and 1 represent uncovered and covered patches, respectively. 5.The method of claim 4, wherein, After the teacher model guided masking, the unmasked remote sensing image patches are input to a student model encoder , which is followed by an image reconstruction encoder , whose output represents the original pixels resulting from image reconstruction. For each occluded patch, a softmax classifier is used to predict the corresponding discrete token , wherein represents the occluded image, represents the weights of the linear layer in the softmax classifier; The pre-training target is to maximize the log-likelihood of correctly classifying the discrete tokens given the covered image: wherein represent the entire training data set, represent the entire training data set, the occluded image that is completed by occlusion.

6. The remote sensing image self-supervised representation learning method according to claim 5, characterized in that, For student model loss prediction , an additional training objective for loss prediction is set: For each patch in the remote sensing image, the relative size of the prediction loss value between each patch is predicted, and the ordering relationship of the prediction loss between the patches is implicitly learned; Finally, the training target is represented as: wherein and are the decoder outputs from the student model the first and the second represent the predicted loss value of the student model, wherein represents the different patches index in the image, which is used to identify each patch; represents the sigmoid activation function; and are two identifiers, which are used to identify the relative size between the predicted loss and the standard reconstruction loss, and the specific formula is as follows:​ wherein represent patches during the training process are covered, is the loss of model image reconstruction; The final training target of the model combines the loss prediction target and the image reconstruction target and is finally represented as: the training objective of the model to generate the pre-training task less implicitly suggests the visual concept of the object; the training objective of the model to make the model learn the abstract semantic information.

Images