Remote sensing image multi-modal unified representation learning method based on mask image modeling
Through the multimodal unified representation learning method of remote sensing images based on mask image modeling, the problems of heterogeneous data feature differences and insufficient label data are solved, and the effective utilization of massive label-free multimodal data is achieved, and the model is improved. Generalization ability and classification accuracy provide new directions for the development of remote sensing technology.
Patent Information
- Application Number
- CN202411113632.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-14
AI Technical Summary
Due to the large differences in heterogeneous data characteristics and insufficient amount of labeled data, existing remote sensing technologies are difficult to make full use of massive labelless multimodal remote sensing data, resulting in overfitting of the model and low classification accuracy.
Using the multimodal unified representation learning method of remote sensing images based on mask image modeling, by constructing pre-trained models and fine-tuning models, using label-free multimodal remote sensing to train images, extract general and generalized features, and adapt to multimodal state data input enhances the generalization ability and robustness of the model.
It significantly improves the performance and generalization capabilities of the model in a variety of land use analysis tasks, improves the performance and robustness of the model under different mode data, provides more accurate and reliable decision support, and enhances remote sensing data analysis adaptability.
Smart Images

Figure CN119007031A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a remote sensing image multi-modal unified representation learning method, and belongs to the technical field of multi-modal satellite remote sensing. Background Art
[0002] With the launch of various series of satellites, it is now possible to obtain data from various sensors such as panchromatic, multispectral, infrared, and SAR. These sensors describe the observation scene from different angles, and the physical meanings contained in their characteristics are different and highly complementary. Optical sensors use the reflection characteristics of the target to form images, and the images have high spatial resolution and clear texture details, but they are easily affected by weather and have difficulty detecting surface features covered by clouds and fog. SAR sensors can penetrate clouds and fog by actively emitting electromagnetic waves, and are not affected by factors such as weather. They can obtain the intensity and phase information of the target. Infrared sensors use the radiation characteristics of the target to form images, have strong anti-interference capabilities, and can obtain temperature distribution images of the surface, but the color and texture information of the target they obtain is relatively rough. At present, it is difficult to fully reflect the characteristic information of ground objects by relying solely on data from a single sensor, which limits the further improvement of the performance of various downstream tasks.
[0003] In addition, with the rapid growth of data volume and the significant improvement of computing power, the scale of artificial intelligence models is expanding. Existing large models have proven their effectiveness in natural language processing and computer vision. By pre-training on large-scale data, model parameters can be better initialized, thereby improving the performance and generalization ability of the model in downstream tasks. In the field of remote sensing, with the explosive growth of the number of satellites, the amount of new data added every day is about 10PB. The large amount of unlabeled data has prompted people to combine pre-training frameworks with self-supervision paradigms to make better use of massive unlabeled data.
[0004] Due to the large differences in the characteristics of heterogeneous remote sensing data and the insufficient amount of labeled remote sensing data, existing methods are unable to fully utilize massive unlabeled multimodal remote sensing data, and the models are prone to overfitting and have low classification accuracy. Summary of the invention
[0005] The purpose of the present invention is to solve the problems that the existing methods are difficult to fully utilize massive unlabeled multimodal remote sensing data, the model is overfitted and the classification accuracy is low due to the large differences in the characteristics of remote sensing heterogeneous data and the insufficient amount of labeled remote sensing data. A multimodal unified representation learning method for remote sensing images based on mask image modeling is proposed.
[0006] The specific process of the multimodal unified representation learning method for remote sensing images based on mask image modeling is as follows:
[0007] Step 1: Obtain unlabeled multimodal remote sensing paired images;
[0008] Obtain multimodal remote sensing paired images with scene classification labels;
[0009] Obtain multimodal remote sensing paired images with semantic segmentation labels;
[0010] Step 2: Construct a pre-training model for multimodal unified representation learning of remote sensing images based on mask image modeling;
[0011] Step 3: training the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 2 based on unlabeled multimodal remote sensing paired images until convergence, thereby obtaining a trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling;
[0012] Step 4: Based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed;
[0013] Step 5: Based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed;
[0014] Step 6: Based on the multimodal remote sensing paired images with scene classification labels, the classification downstream task fine-tuning model of the multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 4 is trained until convergence, and a trained classification downstream task fine-tuning model of the multimodal unified representation learning of remote sensing images based on mask image modeling is obtained;
[0015] Step 7: Based on the multimodal remote sensing paired images with semantic segmentation labels, the semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 5 is trained until convergence, and a trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is obtained;
[0016] Step 8: Input the multimodal remote sensing paired image to be tested into the trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling outputs the classification result;
[0017] The multimodal remote sensing paired images to be tested are input into the trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling outputs the semantic segmentation result.
[0018] The beneficial effects of the present invention are:
[0019] This paper proposes a multi-modal unified representation learning framework (MURLF) for remote sensing images based on mask image modeling, which aims to effectively integrate and utilize massive unlabeled multi-modal remote sensing data, and improve various land use analysis tasks by learning more general and generalized features. The innovation of this technology is that it significantly improves the performance and generalization ability of the model, and provides a new solution for comprehensively characterizing the characteristics of ground objects.
[0020] Specifically, the present invention solves the limitation that the current remote sensing basic model is only applicable to single-modal data input. By expanding the existing remote sensing basic model to adapt it to multi-modal data input, the present invention enables more types of unlabeled data to be fully utilized, thereby enhancing the generalization ability and adaptability of the model. This method not only improves the performance of the model under different modal data, but also significantly increases the diversity of model training data, thereby improving the robustness and prediction accuracy of the model. In practical applications, MURLF can provide more accurate and reliable decision support in multiple fields such as resource exploration, agricultural production, urban planning, and environmental monitoring.
[0021] In addition, by introducing mask image modeling technology, the present invention successfully captures the inherent correlation and differences between different modal data, so that the model can better maintain the integrity and consistency of the data when processing complex multimodal data. This method not only improves the performance of the model in a single task, but also enhances its adaptability in a multi-task environment, thus bringing new opportunities and challenges to the field of remote sensing data analysis. Through this innovative framework, the present invention has made an important contribution to promoting the development of remote sensing image processing technology, and also explored new directions for the in-depth mining and application of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 Pre-trained model structure diagram for multimodal unified representation learning;
[0023] Figure 2 is a multi-level mask block flow chart;
[0024] Figure 3 Figure 1 is a diagram of the sensor-dependent feature extractor structure. (a) The sensor-dependent feature extractor structure for high channel correlation, (b) The sensor-dependent feature extractor structure for low channel correlation.
[0025] Figure 4Diagram of the model architecture for fine-tuning the classification downstream task for multimodal unified representation learning;
[0026] Figure 5 Diagram of the model architecture for fine-tuning the downstream task of segmentation for multimodal unified representation learning. DETAILED DESCRIPTION
[0027] Specific implementation method 1: The specific process of the multi-modal unified representation learning method of remote sensing images based on mask image modeling in this implementation method is as follows:
[0028] Step 1: Obtain a large amount (more than 1 million) of unlabeled multimodal remote sensing paired images;
[0029] Obtain a small number (less than 10,000) of multimodal remote sensing paired images with scene classification labels;
[0030] Obtain a small number (less than 10,000) of multimodal remote sensing paired images with semantic segmentation labels;
[0031] The multimodality refers to multiple types of data from different sensors or data sources;
[0032] The multimodal remote sensing paired images refer to images of the same area from different sensors;
[0033] Step 2: Construct a pre-training model for multimodal unified representation learning of remote sensing images based on mask image modeling;
[0034] Step 3: training the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 2 based on massive unlabeled multimodal remote sensing paired images until convergence, thereby obtaining a trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling;
[0035] Step 4: Based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed;
[0036] Step 5: Based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed;
[0037] Step 6: Based on a small number of multimodal remote sensing paired images with scene classification labels, the classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 4 is trained until convergence, and a trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is obtained;
[0038] Step 7: Based on a small number of multimodal remote sensing paired images with semantic segmentation labels, the semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 5 is trained until convergence, and a trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is obtained;
[0039] Step 8: Input the multimodal remote sensing paired image to be tested into the trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling outputs the classification result;
[0040] The multimodal remote sensing paired images to be tested are input into the trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling outputs the semantic segmentation result.
[0041] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that in step 2, a pre-training model for multi-modal unified representation learning of remote sensing images based on mask image modeling is constructed;
[0042] The specific process is:
[0043] The pre-trained model for multimodal unified representation learning of remote sensing images based on mask image modeling includes a sensor-dependent feature extractor, an encoder, a decoder, and a multi-level mask block;
[0044] The sensor-related feature extractor is divided into two structures:
[0045] For images with high correlation between channels, sensor-dependent feature extractors include downsampling convolution, depthwise convolution, one-dimensional convolution, multi-layer perceptron, and average pooling layer;
[0046] The image with high inter-channel correlation is an image acquired by a visible light sensor, a multispectral sensor, or a hyperspectral sensor;
[0047] For images with low inter-channel correlation, the sensor-dependent feature extractor includes downsampling convolution, depthwise convolution, 1×1 convolution, multi-layer perceptron, and average pooling layer;
[0048] The images with low inter-channel correlation are images acquired by SAR sensors, thermal infrared sensors, and depth sensors;
[0049] The encoder includes a shared feature encoder, a first independent feature encoder, and a second independent feature encoder;
[0050] The shared feature encoder consists of 12 cascaded Transformer blocks (connected sequentially);
[0051] The first independent feature encoder consists of 2 cascaded Transformer blocks (connected sequentially);
[0052] The second independent feature encoder consists of 2 cascaded Transformer blocks (connected sequentially);
[0053] The shared feature encoder learns the unified identification of data between different modalities by sharing weight parameters for all input modalities; each modality corresponds to an independent feature encoder, and its weight is only related to the current modality;
[0054] The decoder consists of 8 cascaded Transformer blocks (connected sequentially);
[0055] The multi-level mask block includes mask block 1, mask block 2, and mask block 3.
[0056] A pre-trained model for multimodal unified representation learning of remote sensing images based on mask image modeling; using the consistency and complementary relationship between different modal data, common features and unique features are extracted to alleviate the burden on the encoder caused by the large difference in data characteristics between different modalities. For each modality, it independently and randomly masks a large number of ground features and predicts the masked pixels using the same-modality and cross-modal slices.
[0057] The other steps and parameters are the same as those in the first embodiment.
[0058] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the multi-level mask block acquisition process is:
[0059] Generate mask block 3 using random numbers ( H is the height of the input image, W is the width of the input image), mask block 3 is the mask vector;
[0060] Reshape the mask block 3 into a two-dimensional tensor ( Box shape), upsample the two-dimensional tensor to obtain mask block 2 ( Box shape);
[0061] Perform upsampling on mask block 2 to obtain mask block 1 ( box shape).
[0062] The final multi-level mask block generates three mask images of different scales.
[0063] The other steps and parameters are the same as those in the first or second embodiment.
[0064] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that, in step 3, the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 2 is trained based on massive unlabeled multimodal remote sensing paired images until convergence, thereby obtaining a trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling;
[0065] The specific process is:
[0066] Step 31: remote sensing image 1 in the remote sensing paired image is input into the first sensor related feature extractor, and the first sensor related feature extractor outputs features; the specific process is:
[0067] 1) For images with high correlation between channels, remote sensing image 1 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain feature G;
[0068] Feature G is input into the deep convolution layer and the one-dimensional convolution layer in sequence. The output feature of the one-dimensional convolution layer is added to feature G to obtain feature H.
[0069] Feature H is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature I;
[0070] Feature I is input into the average pooling layer, and the output feature of the average pooling layer is added to feature I to obtain feature J;
[0071] Feature J is added to feature H to obtain feature K; feature K is the output feature of the first sensor-related feature extractor;
[0072] 2) For images with low inter-channel correlation, remote sensing image 2 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain feature g;
[0073] Feature g is input into the deep convolution layer and the 1×1 convolution layer in sequence. The output feature of the 1×1 convolution layer is added to feature g to obtain feature h.
[0074] Feature h is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature i;
[0075] Feature i is input into the average pooling layer, and the output feature of the average pooling layer is added to feature i to obtain feature j;
[0076] Feature j is added to feature h to obtain feature k; feature k is the output feature of the first sensor-related feature extractor;
[0077] Step 32: The output feature of the first sensor-related feature extractor is input into the second sensor-related feature extractor, and the second sensor-related feature extractor outputs the feature;
[0078] 1) For images with high correlation between channels, feature K is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 2 to obtain feature G′;
[0079] Feature G′ is input into the deep convolution layer and the one-dimensional convolution layer in sequence. The output feature of the one-dimensional convolution layer is added to feature G′ to obtain feature H′.
[0080] Feature H′ is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature I′;
[0081] Feature I′ is input into the average pooling layer, and the output feature of the average pooling layer is added to feature I′ to obtain feature J′;
[0082] Feature J′ is added to feature H′ to obtain feature K′, which is the output feature of the second sensor-related feature extractor;
[0083] 2) For images with low inter-channel correlation, feature k is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by the mask block 2 to obtain feature g′;
[0084] Feature g′ is input into the deep convolution layer and the 1×1 convolution layer in sequence. The output feature of the 1×1 convolution layer is added to the feature g′ to obtain feature h′.
[0085] Feature h′ is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature i′;
[0086] Feature i′ is input into the average pooling layer, and the output feature of the average pooling layer is added to feature i′ to obtain feature j′;
[0087] Feature j′ is added to feature h′ to obtain feature k′, which is the output feature of the second sensor-related feature extractor;
[0088] Step 33: Perform feature graph linearization processing on the output features of the second sensor-related feature extractor to obtain linearized features, and multiply the linearized features by mask block 3 to obtain masked features;
[0089] Step 3 and 4: input the masked features into the shared feature encoder and the first independent feature encoder respectively, the first independent feature encoder outputs feature A, and the shared feature encoder outputs feature B;
[0090] The shared feature encoder consists of 12 cascaded Transformer blocks (connected sequentially);
[0091] The first independent feature encoder consists of 2 cascaded Transformer blocks (connected sequentially);
[0092] The output feature A of the first independent feature encoder and the output feature B of the shared feature encoder are added to obtain the added feature E; the specific process is:
[0093] The masked features are input into the shared feature encoder, and the shared feature encoder outputs feature B;
[0094] Input the masked features into a first independent feature encoder, and the first independent feature encoder outputs feature A;
[0095] Add the feature A output by the first independent feature encoder and the feature B output by the shared feature encoder to obtain a summed feature E;
[0096] Remote sensing image 1 and remote sensing image 2 in the remote sensing paired images are simultaneously input into the sensor-related feature extractor;
[0097] Step 35: remote sensing image 2 in the remote sensing paired image is input into the third sensor related feature extractor, and the third sensor related feature extractor outputs features; the specific process is:
[0098] 1) For images with high correlation between channels, remote sensing image 2 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain the feature
[0099] feature Input the deep convolution layer and the one-dimensional convolution layer in sequence, and the one-dimensional convolution layer outputs features and features Add and get the feature
[0100] feature Input multi-layer perceptron, multi-layer perceptron output features
[0101] feature Input average pooling layer, average pooling layer output features and features Add and get the feature
[0102] feature With features Add and get the feature That is, the third sensor-related feature extractor outputs features;
[0103] 2) For images with low inter-channel correlation, remote sensing image 2 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain the feature
[0104] feature Input the depth convolution layer and 1×1 convolution layer in sequence, and the 1×1 convolution layer outputs features and features Add and get the feature
[0105] feature Input multi-layer perceptron, multi-layer perceptron output features
[0106] feature Input average pooling layer, average pooling layer output features and features Add and get the feature
[0107] feature With features Add and get the feature That is, the third sensor-related feature extractor outputs features;
[0108] Step 36: The output feature of the third sensor-related feature extractor is input into the fourth sensor-related feature extractor, and the fourth sensor-related feature extractor outputs the feature; the specific process is:
[0109] 1) For images with high correlation between channels, features Input the downsampling convolution layer, the output feature of the downsampling convolution layer is multiplied by the mask block 2 to obtain the feature
[0110] feature Input the deep convolution layer and the one-dimensional convolution layer in sequence, and the one-dimensional convolution layer outputs features and features Add and get the feature
[0111] feature Input multi-layer perceptron, multi-layer perceptron output features
[0112] feature Input average pooling layer, average pooling layer output features and features Add and get the feature
[0113] feature With features Add and get the feature That is, the fourth sensor-related feature extractor outputs features;
[0114] 2) For images with low inter-channel correlation, features Input the downsampling convolution layer, the output feature of the downsampling convolution layer is multiplied by the mask block 2 to obtain the feature
[0115] feature Input the depth convolution layer and 1×1 convolution layer in sequence, and the 1×1 convolution layer outputs features and features Add and get the feature
[0116] feature Input multi-layer perceptron, multi-layer perceptron output features
[0117] feature Input average pooling layer, average pooling layer output features and features Add and get the feature
[0118] feature With features Add and get the feature That is, the fourth sensor-related feature extractor outputs features;
[0119] Step 37: Perform feature graph linearization processing on the output features of the fourth sensor-related feature extractor to obtain linearized features, and multiply the linearized features by the mask block 3 to obtain masked features;
[0120] Step 38: Input the masked features into the shared feature encoder and the second independent feature encoder respectively, the second independent feature encoder outputs feature C, and the shared feature encoder outputs feature D;
[0121] The shared feature encoder consists of 12 cascaded Transformer blocks (connected sequentially);
[0122] The second independent feature encoder consists of 2 cascaded Transformer blocks (connected sequentially);
[0123] Add the feature C output by the second independent feature encoder and the feature D output by the shared feature encoder to obtain the added feature F;
[0124] The specific process is:
[0125] The masked features are input into the shared feature encoder, and the shared feature encoder outputs feature D;
[0126] Input the masked features into a second independent feature encoder, and the second independent feature encoder outputs feature C;
[0127] Add the feature C output by the second independent feature encoder and the feature D output by the shared feature encoder to obtain the added feature F;
[0128] Step 39: input the feature E after adding the features in steps 3 and 4 into the first decoder, and the first decoder outputs a reconstructed image of the remote sensing image 1;
[0129] The first decoder consists of 8 cascaded Transformer blocks (connected sequentially);
[0130] Step 30: input the feature F after adding in step 38 into the second decoder, and the second decoder outputs the reconstructed image of the remote sensing image 2;
[0131] The second decoder consists of 8 cascaded Transformer blocks (connected sequentially);
[0132] Step 31: Repeat steps 31 to 30 until convergence or all remote sensing paired images are input, to obtain a pre-trained model for multimodal unified representation learning of remote sensing images based on mask image modeling.
[0133] The other steps and parameters are the same as those in the first to third embodiments.
[0134] Specific implementation method five: This implementation method is different from any one of specific implementation methods one to four in that the feature F1 is obtained by splicing the output feature of the first sensor-related feature extractor and the output feature of the third sensor-related feature extractor.
[0135] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0136] Specific implementation example 6: This implementation example is different from any one of specific implementation examples 1 to 5 in that the feature F2 is obtained by splicing the output feature of the second sensor-related feature extractor and the output feature of the fourth sensor-related feature extractor.
[0137] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.
[0138] Specific implementation method seven: This implementation method is different from any one of specific implementation methods one to six in that, in step four, based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed;
[0139] The specific process is:
[0140] The first decoder and the first decoder in the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling are replaced with a linear classifier.
[0141] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0142] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that, in step five, based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed;
[0143] The specific process is:
[0144] The first decoder and the first decoder in the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling are replaced by the first downsampling, the second downsampling, and the UperNet head.
[0145] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.
[0146] Specific implementation nine: This implementation is different from any one of specific implementations one to eight in that, in step six, the classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step four is trained based on a small number of multimodal remote sensing paired images with scene classification labels until convergence, thereby obtaining a trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling;
[0147] The specific process is:
[0148] A small number of multimodal remote sensing paired images with scene classification labels are input into the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the summed features E and F obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling are spliced to obtain the spliced features, which are input into the linear classifier, and the linear classifier outputs the classification results (the classification results of each image as a whole, the category corresponding to each image as a whole); until convergence or all multimodal remote sensing paired images with scene classification labels are input, a trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is obtained.
[0149] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0150] Specific implementation ten: This implementation differs from one of specific implementations one to nine in that, in step seven, the semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step five is trained based on a small number of multimodal remote sensing paired images with semantic segmentation labels until convergence, thereby obtaining a trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling;
[0151] The specific process is:
[0152] Input a small amount of multimodal remote sensing paired images with semantic segmentation labels into a pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, and concatenate the summed feature E and the summed feature F obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling to obtain a concatenated feature F3;
[0153] Down-sampling operation is performed on the summed feature E obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling to obtain feature f4;
[0154] The summed feature F obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling is downsampled to obtain feature f4′;
[0155] Concatenate feature f4 and feature f4′ to obtain feature F4;
[0156] Input features F1, F2, F3, and F4 into the UperNet head (the head refers to the segmentation head of UperNet), and the UperNet head outputs the segmentation result (pixel-level classification result, each specific classification in each image, pixel level);
[0157] Until convergence or all multimodal remote sensing paired images with semantic segmentation labels are input, a trained semantic segmentation downstream task fine-tuning model for multimodal unified representation learning of remote sensing images based on mask image modeling is obtained.
[0158] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.
[0159] The encoder and decoder structures used in pre-training are as follows Figure 1 As shown in the figure, the encoder consists of a shared feature encoder for all modalities and an independent feature encoder for each modality. This architecture helps to learn common representations across modalities and unique representations for each mode. Each encoder consists of multiple cascaded Transformer layers, each of which contains a multi-head self-attention block and a multi-layer perceptron. The use of multi-head self-attention blocks can help the model understand the complex and changing relationships in multimodal data and improve the model's ability to capture the correlation between features.
[0160] A separate decoder is used for each modality to enhance feature learning. The structure of the decoder is similar to that of the encoder. In the first layer of the decoder, cross-attention blocks are used instead of self-attention blocks to achieve effective information interaction. After that, the visible markers and mask markers are input into the decoder in sequence, and the pixels corresponding to the mask area are reconstructed by jointly utilizing the unimodal and cross-modal information.
[0161] The proposed multi-level mask block structure is as follows Figure 2 As shown in the figure, firstly, a vector mask block 3 of length L is generated using a random number seed, and then the generated mask block 3 vector is transformed into a matrix form using a deformation operation, and then the generated mask feature map is upsampled into mask block 2 and mask block 1 respectively through interpolation.
[0162] The specific structure of the sensor-related feature extractor is as follows: Figure 3 As shown in the figure, its structure is similar to the Transformer block, but it uses different convolution blocks instead of self-attention mechanisms to more clearly extract the neighbor information of the feature map. It first adjusts the size of the feature map through stride convolution, and then fully extracts the effective features related to the sensor through the sensor-related attention mechanism. Finally, after the multi-layer perceptron and mean pooling operations, the output can clearly reflect the characteristics of the sensor-related characteristics, which is expressed as:
[0163] x mask =x in Mask
[0164] x attn =x in Mask+Attention(x in ·Mask)
[0165] x mlp =MLP(x attn )
[0166] x out =x attn +x mlp +Meanpool(x mlp )
[0167] where x in represents the input feature after stride convolution, Mask represents the mask map, x mask represents the masked feature map, Attention(·) represents the sensor-related attention mechanism, and x attn represents the feature map output after the attention mechanism, MLP(·) represents the multi-layer perceptron operation, x mlp Represents the features output by the multi-layer perceptron, x out Represents the characteristics of the output. The sensor-related attention mechanism has different structures for the data of different input sensors. When the input image has high channel correlation, it uses a combination of deep convolution and one-dimensional convolution to explicitly extract the neighboring spatial and spectral features. It can be expressed as:
[0168] Attention(·)=Conv 1D (ConvDW (·)
[0169] Among them, Conv DW (·) represents the depth convolution operation, Conv 1D (·) represents a one-dimensional convolution operation. When the input image has low channel correlation, a combination of deep convolution and 1×1 convolution is used to extract neighboring spatial features. It can be expressed as:
[0170] Attention(·)=Conv 1×1 (Conv DW (·)
[0171] Figure 1 The training process of the pre-trained model of multimodal unified representation learning shown in the figure includes the following detailed steps: using the collected massive unlabeled multimodal remote sensing paired images to construct pre-trained input sample pairs. The network parameters of the pre-trained model of multimodal unified representation learning are optimized through the training data set to obtain the pre-trained pre-trained model of multimodal unified representation learning. The network parameters are optimized by minimizing the loss function shown in the figure:
[0172]
[0173] Where m and n represent the number of mask blocks from modalities M1 and M2, respectively, and x represents the reconstructed image block output by the network structure. Represents the real image patch.
[0174] The structure of the fine-tuning model for the classification downstream task of multimodal unified representation learning is as follows: Figure 4 As shown in Figure 1, MURLF removes the decoder structure, concatenates the feature outputs of the encoders of the two modes, and inputs them into the linear classifier for fine-tuning, which can be expressed as:
[0175] Output=Linear(Concat(F M1 ,F M2 ))
[0176] Among them, F M1 With F M2 They represent the output characteristics of different modal encoders respectively.
[0177] The detailed steps of the fine-tuning training process of the multimodal unified representation learning classification downstream task include: using a small number of multimodal remote sensing paired images with scene classification labels collected to construct fine-tuning sample pairs for downstream scene classification tasks. The network parameters of the multimodal unified representation learning classification downstream task fine-tuning model are optimized through the training data set to obtain the fine-tuned multimodal unified representation learning classification downstream task fine-tuning model. In this process, the cross entropy loss function is used to optimize the network parameters.
[0178] The structure of the fine-tuning model for the segmentation downstream task of the constructed multimodal unified representation learning is as follows Figure 5 As shown in Figure 1, MURLF removes the decoder structure and concatenates the output features of different modalities at the same scale, including the output features F1 concatenated from the first sensor-related feature extractor and the third sensor feature extractor, the output features F2 concatenated from the second sensor feature extractor and the fourth sensor feature extractor, the output concatenation features F3 of different encoders, and the output features F4 concatenated from the sampled features of different modalities. The multi-scale features are input to the UperNet head for fine-tuning, which can be expressed as
[0179] Output = UperNet(F1,F2,F3,F4)
[0180] The detailed steps of the fine-tuning training process of the multimodal unified representation learning segmentation downstream task include: using the multimodal remote sensing paired images with a small number of semantic segmentation labels collected to construct the downstream semantic segmentation task fine-tuning sample pairs. The network parameters of the multimodal unified representation learning segmentation downstream task fine-tuning model are optimized through the training data set to obtain the fine-tuned multimodal unified representation learning segmentation downstream task fine-tuning model. In this process, the cross entropy loss function is used to optimize the network parameters.
[0181] The following examples are used to verify the beneficial effects of the present invention:
[0182] The MURLF method is pre-trained on the SSL4EO-S12 dataset, which is a large multimodal benchmark dataset consisting of 1 million pairs of unlabeled optical-SAR images. The optical data consists of 13 multispectral bands and comes from the Sentinel-2 satellite, while the SAR data comes from the Sentinel-1 satellite with VH and VV polarizations. All images in the SSL4EO-S12 dataset are used for pre-training and resized to 224×224 pixels to meet the input specification of the network.
[0183] Pre-training was performed using MURLF for 100 iterations. The model was optimized using the AdamW optimizer with a cosine learning rate schedule. The learning rate was set to 5e-4 and the batch size was set to 256. In addition, a weight decay of 0.02 was used, and RandomResizedCrop and RandomHorizontalFlip were used to augment the data.
[0184] The effectiveness of the present invention in scene classification tasks is verified on two datasets: So2Sat-LCZ42 and BigEarthNet-MM.
[0185] Table 1 Scene classification results of So2Sat-LCZ42 dataset
[0186]
[0187] C1: Compact high-rise buildings. C2: Compact medium-rise buildings. C3: Compact low-rise buildings. C4: Open high-rise buildings. C5: Open medium-rise buildings. C6: Open low-rise buildings. C7: Sparse low-rise buildings. C8: Large low-rise buildings. C9: Sparse buildings. C10: Heavy industrial buildings. C11: Dense trees. C12: Scattered trees. C13: Shrubs. C14: Low plants. C15: Exposed rocks. C16: Exposed soil or sand. C17: Water.
[0188] Table 2 Scene classification results of BigEarthNet-MM dataset
[0189]
[0190] C1: Urban structures. C2: Industrial or commercial units. C3: Cultivated land. C4: Permanent crops. C5: Pastures. C6: Complex farming patterns. C7: Natural vegetation. C8: Agroforestry areas. C9: Broadleaf forests. C10: Coniferous forests. C11: Mixed forests. C12: Natural grasslandsC13: Wasteland, heath and sclerophyllous vegetation. C14: Transitional woodland / shrubland. C15: Beaches, dunes, sandy areas. C16: Inland wetlands. C17: Coastal wetlands. C18: Inland waters. C19: Marine waters.
[0191] As shown in Tables 1 and 2, compared with the random initialization method without pre-training, the proposed multimodal pre-training method can effectively improve its performance in classification downstream tasks.
[0192] The effectiveness of the present invention in semantic segmentation tasks was verified on two datasets: DFC2020 and WHU-OPT-SAR.
[0193] Table 3 DFC2020 semantic segmentation results
[0194]
[0195] C1: Forest. C2: Shrubbery. C3: Grassland. C4: Wetland. C5: FarmlandC6: Urban Buildings. C7: Barren. C8: Water.
[0196] Table 4 WHU-OPT-SAR semantic segmentation results
[0197]
[0198] C1: Farmland. C2: City. C3: Countryside. C4: Water. C5: Forest. C6: Road. C7: Others.
[0199] As shown in Tables 3 and 4, compared with the random initialization method without pre-training, the proposed multimodal pre-training method can effectively improve its performance in downstream segmentation tasks.
[0200] In summary, in order to make more effective use of massive multimodal unlabeled paired data, this embodiment provides a remote sensing image multimodal unified representation learning method and system based on mask image modeling, which can effectively utilize massive unlabeled remote sensing data, so that it can be more efficiently performed in various downstream tasks.
[0201] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. A multimodal unified representation learning method for remote sensing images based on mask image modeling, characterized by: The specific process of the method is: Step 1: Obtain unlabeled multimodal remote sensing paired images; Obtain multimodal remote sensing paired images with scene classification labels; Obtain multimodal remote sensing paired images with semantic segmentation labels; Step 2: Construct a pre-training model for multimodal unified representation learning of remote sensing images based on mask image modeling; Step 3: training the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 2 based on unlabeled multimodal remote sensing paired images until convergence, thereby obtaining a trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling; Step 4: Based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed; Step 5: Based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed; Step 6: Based on the multimodal remote sensing paired images with scene classification labels, the classification downstream task fine-tuning model of the multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 4 is trained until convergence, and a trained classification downstream task fine-tuning model of the multimodal unified representation learning of remote sensing images based on mask image modeling is obtained; Step 7: Based on the multimodal remote sensing paired images with semantic segmentation labels, the semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 5 is trained until convergence, and a trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is obtained; Step 8: Input the multimodal remote sensing paired image to be tested into the trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling outputs the classification result; The multimodal remote sensing paired images to be tested are input into the trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the trained semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling outputs the semantic segmentation result.
2. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 1, characterized in that: In the step 2, a pre-training model for multi-modal unified representation learning of remote sensing images based on mask image modeling is constructed; the specific process is as follows: The pre-trained model for multimodal unified representation learning of remote sensing images based on mask image modeling includes a sensor-dependent feature extractor, an encoder, a decoder, and a multi-level mask block; The sensor-related feature extractor is divided into two structures: For images with high correlation between channels, sensor-dependent feature extractors include downsampling convolution, depthwise convolution, one-dimensional convolution, multi-layer perceptron, and average pooling layer; The image with high inter-channel correlation is an image acquired by a visible light sensor, a multispectral sensor, or a hyperspectral sensor; For images with low inter-channel correlation, the sensor-dependent feature extractor includes downsampling convolution, depthwise convolution, 1×1 convolution, multi-layer perceptron, and average pooling layer; The images with low inter-channel correlation are images acquired by SAR sensors, thermal infrared sensors, and depth sensors; The encoder includes a shared feature encoder, a first independent feature encoder, and a second independent feature encoder; The shared feature encoder consists of 12 cascaded Transformer blocks; The first independent feature encoder consists of two cascaded Transformer blocks; The second independent feature encoder consists of 2 cascaded Transformer blocks; The decoder consists of 8 cascaded Transformer blocks; The multi-level mask block includes mask block 1, mask block 2, and mask block 3.
3. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 2, characterized in that: The multi-level mask block acquisition process is: Generate mask block 3 using random numbers, where mask block 3 is a mask vector; Reshape the mask block 3 into a two-dimensional tensor, and upsample the two-dimensional tensor to obtain the mask block 2; An upsampling operation is performed on mask block 2 to obtain mask block 1.
4. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 3 is characterized in that: In the step 3, the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 2 is trained based on the unlabeled multimodal remote sensing paired images until convergence, thereby obtaining a trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling; The specific process is: Step 31: remote sensing image 1 in the remote sensing paired image is input into a first sensor-related feature extractor, and the first sensor-related feature extractor outputs features; The specific process is: 1) For images with high correlation between channels, remote sensing image 1 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain feature G; Feature G is input into the deep convolution layer and the one-dimensional convolution layer in sequence. The output feature of the one-dimensional convolution layer is added to feature G to obtain feature H. Feature H is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature I; Feature I is input into the average pooling layer, and the output feature of the average pooling layer is added to feature I to obtain feature J; Feature J is added to feature H to obtain feature K; feature K is the output feature of the first sensor-related feature extractor; 2) For images with low inter-channel correlation, remote sensing image 2 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain feature g; Feature g is input into the deep convolution layer and the 1×1 convolution layer in sequence. The output feature of the 1×1 convolution layer is added to feature g to obtain feature h. Feature h is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature i; Feature i is input into the average pooling layer, and the output feature of the average pooling layer is added to feature i to obtain feature j; Feature j is added to feature h to obtain feature k; feature k is the output feature of the first sensor-related feature extractor; Step 32: The output feature of the first sensor-related feature extractor is input into the second sensor-related feature extractor, and the second sensor-related feature extractor outputs the feature; The specific process is: 1) For images with high correlation between channels, feature K is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 2 to obtain feature G′; Feature G′ is input into the deep convolution layer and the one-dimensional convolution layer in sequence. The output feature of the one-dimensional convolution layer is added to feature G′ to obtain feature H′. Feature H′ is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature I′; Feature I′ is input into the average pooling layer, and the output feature of the average pooling layer is added to feature I′ to obtain feature J′; Feature J′ is added to feature H′ to obtain feature K′, which is the output feature of the second sensor-related feature extractor; 2) For images with low inter-channel correlation, feature k is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by the mask block 2 to obtain feature g′; Feature g′ is input into the deep convolution layer and the 1×1 convolution layer in sequence. The output feature of the 1×1 convolution layer is added to the feature g′ to obtain feature h′. Feature h′ is input into the multi-layer perceptron, and the multi-layer perceptron outputs feature i′; Feature i′ is input into the average pooling layer, and the output feature of the average pooling layer is added to feature i′ to obtain feature j′; Feature j′ is added to feature h′ to obtain feature k′, which is the output feature of the second sensor-related feature extractor; Step 33: Perform feature graph linearization processing on the output features of the second sensor-related feature extractor to obtain linearized features, and multiply the linearized features by mask block 3 to obtain masked features; Step 3 and 4: input the masked features into the shared feature encoder and the first independent feature encoder respectively, the first independent feature encoder outputs feature A, and the shared feature encoder outputs feature B; The shared feature encoder consists of 12 cascaded Transformer blocks; The first independent feature encoder consists of two cascaded Transformer blocks; Add the feature A output by the first independent feature encoder and the feature B output by the shared feature encoder to obtain a summed feature E; The specific process is: The masked features are input into the shared feature encoder, and the shared feature encoder outputs feature B; Input the masked features into a first independent feature encoder, and the first independent feature encoder outputs feature A; Add the feature A output by the first independent feature encoder and the feature B output by the shared feature encoder to obtain a summed feature E; Step 35: remote sensing image 2 in the remote sensing paired image is input into a third sensor related feature extractor, and the third sensor related feature extractor outputs features; The specific process is: 1) For images with high correlation between channels, remote sensing image 2 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain the feature feature Input the deep convolution layer and the one-dimensional convolution layer in sequence, and the one-dimensional convolution layer outputs features and features Add and get the feature feature Input multi-layer perceptron, multi-layer perceptron output features feature Input average pooling layer, average pooling layer output features and features Add and get the feature feature With features Add and get the feature That is, the third sensor-related feature extractor outputs features; 2) For images with low inter-channel correlation, remote sensing image 2 in the remote sensing paired image is input into the downsampling convolution layer, and the output feature of the downsampling convolution layer is multiplied by mask block 1 to obtain the feature feature Input the depth convolution layer and 1×1 convolution layer in sequence, and the 1×1 convolution layer outputs features and features Add and get the feature feature Input multi-layer perceptron, multi-layer perceptron output features feature Input average pooling layer, average pooling layer output features and features Add and get the feature feature With features Add and get the feature That is, the third sensor-related feature extractor outputs features; Step 36: The output feature of the third sensor-related feature extractor is input into the fourth sensor-related feature extractor, and the fourth sensor-related feature extractor outputs the feature; The specific process is: 1) For images with high correlation between channels, features Input the downsampling convolution layer, the output feature of the downsampling convolution layer is multiplied by the mask block 2 to obtain the feature feature Input the deep convolution layer and the one-dimensional convolution layer in sequence, and the one-dimensional convolution layer outputs features and features Add and get the feature feature Input multi-layer perceptron, multi-layer perceptron output features feature Input average pooling layer, average pooling layer output features and features Add and get the feature feature With features Add and get the feature That is, the fourth sensor-related feature extractor outputs features; 2) For images with low inter-channel correlation, features Input the downsampling convolution layer, the output feature of the downsampling convolution layer is multiplied by the mask block 2 to obtain the feature feature Input the depth convolution layer and 1×1 convolution layer in sequence, and the 1×1 convolution layer outputs features and features Add and get the feature feature Input multi-layer perceptron, multi-layer perceptron output features feature Input average pooling layer, average pooling layer output features and features Add and get the feature feature With features Add and get the feature That is, the fourth sensor-related feature extractor outputs features; Step 37: perform feature graph linearization processing on the output features of the fourth sensor-related feature extractor to obtain linearized features, and multiply the linearized features by the mask block 3 to obtain masked features; Step 38: Input the masked features into the shared feature encoder and the second independent feature encoder respectively, the second independent feature encoder outputs feature C, and the shared feature encoder outputs feature D; The shared feature encoder consists of 12 cascaded Transformer blocks; The second independent feature encoder consists of 2 cascaded Transformer blocks; Add the feature C output by the second independent feature encoder and the feature D output by the shared feature encoder to obtain the added feature F; The specific process is: The masked features are input into the shared feature encoder, and the shared feature encoder outputs feature D; Input the masked features into a second independent feature encoder, and the second independent feature encoder outputs feature C; Add the feature C output by the second independent feature encoder and the feature D output by the shared feature encoder to obtain the added feature F; Step 39: input the feature E after adding the features in steps 3 and 4 into the first decoder, and the first decoder outputs a reconstructed image of the remote sensing image 1; The first decoder consists of 8 cascaded Transformer blocks; Step 30: input the feature F after adding in step 38 into the second decoder, and the second decoder outputs the reconstructed image of the remote sensing image 2; The second decoder consists of 8 cascaded Transformer blocks; Step 31: Repeat steps 31 to 30 until convergence or all remote sensing paired images are input, to obtain a pre-trained model for multimodal unified representation learning of remote sensing images based on mask image modeling.
5. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 4 is characterized in that: The feature F1 is obtained by concatenating the output feature of the first sensor-related feature extractor and the output feature of the third sensor-related feature extractor.
6. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 5, characterized in that: The feature F2 is obtained by concatenating the output feature of the second sensor-related feature extractor and the output feature of the fourth sensor-related feature extractor.
7. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 6, characterized in that: In the step 4, based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed; The specific process is: The first decoder and the first decoder in the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling are replaced with a linear classifier.
8. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 7, characterized in that: In the step 5, based on the trained pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, a semantic segmentation downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is constructed; The specific process is: The first decoder and the first decoder in the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling are replaced by the first downsampling, the second downsampling, and the UperNet head.
9. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 8, characterized in that: In the step 6, the classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling constructed in step 4 is trained based on the multimodal remote sensing paired images with scene classification labels until convergence, thereby obtaining a trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling; The specific process is: The multimodal remote sensing paired images with scene classification labels are input into the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, and the summed features E and F obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling are spliced to obtain the spliced features, which are input into the linear classifier, and the linear classifier outputs the classification results; until convergence or all the multimodal remote sensing paired images with scene classification labels are input, the trained classification downstream task fine-tuning model of multimodal unified representation learning of remote sensing images based on mask image modeling is obtained.
10. The multimodal unified representation learning method for remote sensing images based on mask image modeling according to claim 9, characterized in that: In the step 7, the semantic segmentation downstream task fine-tuning model of remote sensing image multimodal unified representation learning based on mask image modeling constructed in step 5 is trained based on the multimodal remote sensing paired images with semantic segmentation labels until convergence, thereby obtaining a trained semantic segmentation downstream task fine-tuning model of remote sensing image multimodal unified representation learning based on mask image modeling; The specific process is: Input the multimodal remote sensing paired images with semantic segmentation labels into the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling, and concatenate the summed feature E and the summed feature F obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling to obtain a concatenated feature F3; Down-sampling operation is performed on the summed feature E obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling to obtain feature f4; Down-sampling operation is performed on the summed feature F obtained from the pre-trained model of multimodal unified representation learning of remote sensing images based on mask image modeling to obtain feature f4′; Concatenate feature f4 and feature f4′ to obtain feature F4; The features F1, F2, F3, and F4 are input into the UperNet head, and the UperNet head outputs the segmentation result; Until convergence or all multimodal remote sensing paired images with semantic segmentation labels are input, a trained semantic segmentation downstream task fine-tuning model for multimodal unified representation learning of remote sensing images based on mask image modeling is obtained.
Citation Information
Patent Citations
Multi-modal brain tumor image segmentation method based on self-supervised learning
CN115841464A
Multi-modal MRI brain tumor image segmentation method based on modal cross attention
CN117635563A
Pre-training model determination method and device, equipment and storage medium
CN117745944A
Cited By
Cross-modal remote sensing image target segmentation method and device based on open domain
CN119478423A
Multi-mode-based remote sensing multi-task target detection method and device
CN119904746A
Remote sensing multi-task decision conversion method based on anaphora segmentation
CN120451552A
A remote sensing multi-task decision transformation method based on reference segmentation
CN120451552B
