Large-model-driven high-resolution remote sensing image interpretation method

Abstract

The invention discloses a large-model-driven high-resolution remote sensing image interpretation method, and belongs to the field of image processing and analysis. The method comprises the following steps: constructing a large-scale high-resolution remote sensing image data set; cleaning and labeling the data, and dividing a training set and a verification set; constructing a multi-modal remote sensing information intelligent extraction large model, and realizing extraction and collaboration of multi-modal remote sensing features; constructing a multi-modal remote sensing information fusion module to realize fusion of visible light coding features and thermal infrared coding features; and processing the fused multi-modal features based on an encoder of the current large language model to realize the interpretation of the high-resolution remote sensing image. By implementing the method provided by the invention, the interpretation accuracy of the high-resolution remote sensing image can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the fields of large models and deep learning, and in particular to a large model-driven high-resolution remote sensing image interpretation method. Background Art

[0002] With the rapid development of remote sensing technology, the ability to obtain large-scale remote sensing image data has been significantly improved. These data have shown wide application potential in many fields such as environmental monitoring, urban planning, agricultural management, and disaster warning. As a means of extracting valuable geospatial information from these data, high-resolution remote sensing image interpretation converts complex remote sensing images into human-readable and analyzable text information. It requires the interpretation method to understand the spatial relationship and semantic connection between various types of ground objects in high-resolution remote sensing images, so as to generate accurate and comprehensive geospatial information descriptions. High-resolution remote sensing image interpretation not only builds a bridge between remote sensing data and practical applications, converts raw pixel data into understandable and analyzable information resources, and provides a solid foundation for scientific research, policy formulation, and decision support; in addition, high-resolution remote sensing image interpretation plays a vital role in promoting global sustainable development, optimizing resource allocation, strengthening ecological and environmental protection, and improving disaster warning capabilities. Therefore, exploring efficient and accurate high-resolution remote sensing image interpretation methods has far-reaching significance for promoting the in-depth application of remote sensing technology and the development of disciplines.

[0003] Traditional remote sensing image interpretation methods mainly rely on manual experience and traditional image processing techniques. These methods are competent when dealing with small-scale, simple scenes, but they often perform poorly when faced with high-resolution remote sensing images with complex scenes and diverse ground objects. In addition, manual interpretation is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in poor reliability and consistency of interpretation results. In recent years, with the rapid development of artificial intelligence and deep learning technologies, image interpretation technology based on deep learning has made significant progress. These methods can automatically extract feature information from images by learning from massive data, and then accurately interpret complex scenes through large language models. However, when dealing with large-scale remote sensing image data, these methods still face problems such as large data volume, high computational complexity, and long training time. At present, whether in the field of image processing or natural language processing, large models have shown great advantages in feature extraction and processing due to their complex structure and large-scale parameters. Summary of the invention

[0004] In view of the current technical problem of poor interpretation effect of high-resolution remote sensing images, the present invention proposes a large model-driven high-resolution remote sensing image interpretation method to achieve the interpretation of high-resolution remote sensing images, thereby effectively improving the accuracy of high-resolution remote sensing image interpretation.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] A large model driven high-resolution remote sensing image interpretation method comprises the following steps:

[0007] Step 1: construct a large-scale multimodal high-resolution remote sensing image dataset, which includes visible light images and thermal infrared images; and perform image preprocessing on the image data in the multimodal high-resolution remote sensing image dataset to obtain a training set for model training;

[0008] The image preprocessing includes unifying the image data formats of each modality, normalizing the image size, and cropping the image; the size and number of the cropped sub-images of each image are consistent, and then the cropped image is scaled to the size of the cropped sub-image; for each modality, all the cropped sub-images and the cropped image scaled to the size of the cropped sub-image are regarded as one sample;

[0009] Step 2, constructing a multimodal remote sensing information extraction intelligent large model for multimodal remote sensing information encoding;

[0010] The multimodal remote sensing information extraction intelligent large model includes a visible light information encoder and a thermal infrared information encoder. The visible light information encoder is used to extract visible light coding features of visible light images, and the thermal infrared information encoder is used to extract thermal infrared coding features of thermal infrared images.

[0011] Step 3, construct a multimodal remote sensing information fusion module to perform multimodal feature fusion on visible light coding features and thermal infrared coding features, and output the fused multimodal features;

[0012] Step 4: construct a multimodal decoder for high-resolution remote sensing image interpretation based on the network structure of the encoder of the large language model, use the fused multimodal features as the input of the multimodal decoder, and obtain the interpretation result of the high-resolution remote sensing image based on its output;

[0013] Step 5: Based on the training set composed of multiple samples, the remote sensing image interpretation model composed of the multimodal remote sensing information extraction intelligent large model, the multimodal remote sensing information fusion module and the multimodal decoder is trained to learn the model parameters. When the preset training convergence conditions are met (the number of times reaches the upper limit or the set loss function value converges), the interpretation result of the input target image is obtained based on the trained remote sensing image interpretation model. The input target image includes the visible light image and the thermal infrared image of the same target object.

[0014] Furthermore, in step 1, image cropping includes: based on the specified overlap rate, the image size is normalized and cropped in a specified cropping order (such as from top to bottom, from left to right), and the image resolution of the cropped sub-images is consistent, and the image resolution of the cropped sub-images of each image is consistent, and the number of cropped sub-images is consistent. Then, all the cropped sub-images and the cropped images scaled to the size of the cropped sub-images are regarded as a sample of each modality.

[0015] For example, when the image size is normalized, the visible light image and the thermal infrared image can be cropped into images with a resolution of 2000×2000. For another example, when the image is cropped into multiple cropped sub-images, each image after size normalization can be cropped into 16-25 sub-images with a resolution of 512×512 according to an overlap rate of 10%-30%.

[0016] Furthermore, in step 2, the structures of the visible light information encoder and the thermal infrared information encoder in the multimodal remote sensing information extraction intelligent large model are consistent, each including n parallel global high-order information embedding Transformer branches, where the value of n is the number of cropped sub-images of each image plus 1; and each global high-order information embedding Transformer branch includes several consecutive feature extraction stages, that is, each global high-order information embedding Transformer branch includes m cascaded global high-order information embedding Transformer modules, which can usually be set to 4, and each global high-order information embedding Transformer module corresponds to one stage; and, in each stage of the visible light information encoder and the thermal infrared information encoder, information joint perception of different global high-order information embedding Transformer branches is performed through the cross-regional information joint intelligent perception module. That is, the output feature map of the global high-order information embedding Transformer module of the same order in n branches is subjected to information joint perception through the cross-regional information joint intelligent perception module, and the information joint perception features of each branch are output (that is, n information joint perception features are output), and then output to the next-order global high-order information embedding Transformer module of the corresponding branch.

[0017] Furthermore, the global high-order information embedding Transformer module of the present invention can improve the model's perception of similar objects and constantly changing objects, thereby laying a good foundation for accurate interpretation. The global high-order information embedding Transformer module in the present invention is specifically set as:

[0018] The input vector of the global high-order information embedding Transformer module is respectively obtained by querying the weight matrix W Q , key weight matrix W K, value weight matrix W V The linear mapping is query vector Q, key vector K and value vector V;

[0019] Perform covariance pooling operations on the query vector Q, key vector K, and value vector V respectively, and then perform a dot multiplication operation on the obtained pooled vector and the corresponding vector before pooling to obtain a new vector Q n , K n and V n ;

[0020] The vector Q n , K n and V n Send it to the multi-head self-attention module for self-attention mechanism calculation to get the output of the multi-head self-attention module Among them, softmax() represents the softmax activation function, d k Representative vector K n Dimensions;

[0021] The output of the multi-head self-attention module and the input vector of the global high-order information embedding Transformer module are superimposed and normalized through the first superposition & normalization layer to obtain the first self-attention fusion feature;

[0022] The first self-attention fusion feature is sent to the fully connected layer, and then the output of the fully connected layer is normalized after feature superposition of the output of the fully connected layer and the first self-attention fusion feature through the second superposition & normalization layer to obtain the second self-attention fusion feature, that is, the output feature of the global high-order information embedded in the Transformer module.

[0023] Furthermore, the covariance pooling operation is specifically as follows:

[0024] The vectors Q, K, and V are respectively subjected to channel reduction operations through a convolution layer with a convolution kernel of 1×1 to obtain new feature vectors Q′, K′, and V′ with the same number of channels. Define C M Indicates the number of channels after reduction;

[0025] Calculate the correlation between channels in each feature vector Q′, K′ and V′ respectively, and get the dimension C M ×C M The correlation matrix M Q 、M K 、M V ; Among them, the calculation of the correlation of the feature graphs can be carried out by Pearson correlation coefficient, cosine similarity, element-by-element multiplication, etc. In the present invention, cosine similarity is preferred, that is, each correlation matrix is ​​obtained by calculating the cosine similarity between the feature graphs;

[0026] The new eigenvectors Q′, K′ and V′ are respectively correlated with the corresponding correlation matrix M Q 、M K 、M V Perform matrix multiplication to obtain the eigenvector S Q , S K , S V ;

[0027] The feature vector S Q , S K , S V Three new feature vectors S′ are obtained through a global average pooling layer and a convolution layer with a convolution kernel of 1×1 respectively. Q , S′ K , S′ V ;

[0028] Then the eigenvector S′ Q , S′ K , S′ V Perform dot multiplication with the corresponding vectors Q, K and V respectively to obtain the new vector Q n , K n and V n .

[0029] Furthermore, the specific execution process of the cross-region information joint intelligent perception module includes:

[0030] (1) Each feature map of the input global high-order information embedded in the Transformer branch (i.e., the output feature maps of the n global high-order information embedded in the Transformer module of the same order) is subjected to global average pooling and global maximum pooling respectively, and then the two pooling results are added element by element to obtain the first feature vector of each branch;

[0031] (2) The first eigenvectors of all branches are added element by element and then passed through several layers (preferably 3 layers) of fully connected layers and a layer of ReLU activation function to obtain a global eigenvector;

[0032] (3) Each feature map of the global high-order information embedded in the Transformer branch is point-multiplied with the global feature vector to obtain the information joint perception feature of each branch, which is used as the input feature of the global high-order information embedded in the next stage of the corresponding branch Transformer module.

[0033] Furthermore, in the multimodal remote sensing information fusion module, visible light coding features and thermal infrared features are fused through a cross-headed self-attention mechanism to achieve multimodal remote sensing feature fusion.

[0034] Furthermore, the multimodal remote sensing information fusion module includes multiple layers of multi-head self-attention mechanism layers. Two groups of parallel self-attention calculations are performed in each layer of the multi-head self-attention mechanism layer, one group uses visible light coding features as value vectors and thermal infrared coding features as query vectors and key vectors; the other group uses thermal infrared coding features as value vectors and visible light coding features as query vectors and key vectors; and after each group of self-attention calculations is completed, the output feature maps of the two groups are added element by element as the input features of the multi-head self-attention mechanism layer of the next layer.

[0035] Furthermore, in step 5, the multimodal remote sensing information extraction intelligent large model in the remote sensing image interpretation model is first trained through the random size mask self-supervised learning strategy, specifically including:

[0036] A training object is selected from the visible light information encoder and the thermal infrared information encoder, and a universal decoder (such as a universal decoder based on a convolutional network) is connected after the training object, and the decoder is used to reconstruct the input image of the training object;

[0037] For the input image of the training object (visible light or thermal infrared image), randomly select one of the specified multiple image scales (for example, 224×224, 512×512 and 1024×1024, which can be set based on the actual application scenario) as the size of the current processing image;

[0038] On the image of this size, a certain proportion of image blocks are randomly selected for mask processing, and the scale of the mask matrix M used in the mask processing cannot exceed the size of the currently selected processing image (for example, one of the four sizes of 30×30, 90×90, 120×120 and 256×256); preferably, the specific operation of the mask can be to replace the selected image block with all black (pixel value is 0);

[0039] The loss function of the random size mask self-supervised learning strategy for the training object is set to:

[0040]

[0041] Where H, W, C represent the height, width and number of channels of the input image respectively; ∑M represents the total number of unmasked pixels in the mask matrix M. For example, corresponding to the mask operation of replacing the selected image block with all black (pixel value is 0), ∑M is the total number of elements with value 1 in the mask matrix M. hwc and Represent the pixel values ​​of the original image and the reconstructed image at the image position (h, w, c) respectively, where the original image is the input image of the training object.

[0042] When the preset training convergence conditions are met, the trained visible light information encoder and thermal infrared information encoder are obtained in the form of parameter sharing based on the trained training objects.

[0043] Furthermore, in step 5, the multimodal remote sensing information fusion module and the multimodal decoder in the remote sensing image interpretation model are trained on weight parameters based on the pre-trained multimodal remote sensing information extraction intelligent large model. Preferably, during training, a cross entropy loss function can be used to train it.

[0044] The technical solution provided by the present invention brings at least the following beneficial effects:

[0045] (1) By constructing a large model for intelligent extraction of multimodal remote sensing information, the present invention realizes the extraction and coordination of multimodal remote sensing features, effectively utilizes multiple information sources in remote sensing images, and improves the integrity and accuracy of feature extraction. This technical improvement makes the interpretation of high-resolution remote sensing images more accurate and reduces the possibility of misjudgment and missed judgment.

[0046] (2) The design of the multimodal remote sensing information fusion module achieves the deep fusion of visible light coding features and thermal infrared coding features, further improving the model’s interpretation ability. This cross-modal feature fusion method is technically innovative and provides new ideas for the interpretation of high-resolution remote sensing images.

[0047] (3) The present invention reduces the cost and time of manual intervention by improving the accuracy and efficiency of high-resolution remote sensing image interpretation. In the field of remote sensing image analysis, this technical improvement can significantly reduce the consumption of human resources and reduce the operating costs of enterprises.

[0048] (4) In terms of urban planning, the present invention can provide more accurate and timely remote sensing image interpretation results, providing a scientific basis for urban planners to optimize urban layout and resource allocation.

[0049] (5) In terms of environmental monitoring and disaster assessment, the application of the present invention can timely discover environmental problems and potential disasters, provide early warning and decision-making support for relevant departments, and protect people’s lives and property. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0052] Figure 2 A structural diagram of a large model for intelligent extraction of remote sensing information according to an embodiment of the present invention;

[0053] Figure 3 This is a multi-head attention structure diagram of global high-order information embedded in Transformer according to an embodiment of the present invention;

[0054] Figure 4 This is a structural diagram of a cross-region information joint intelligent perception module according to an embodiment of the present invention;

[0055] Figure 5 This is a structural diagram of the multimodal remote sensing information fusion module constructed in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0057] The embodiment of the present invention provides a large model driven high-resolution remote sensing image interpretation method, which is intended to solve the problem of poor remote sensing image interpretation effect.

[0058] As a possible implementation, see Figure 1 The method provided in the embodiment of the present invention comprises the following steps:

[0059] Step 1: Build a large-scale multimodal high-resolution remote sensing image dataset, and perform image preprocessing on the image data to build a training set and a test set. The multimodal high-resolution remote sensing image dataset includes visible light images and thermal infrared images, and the image preprocessing includes the unification of the image data format of each modality, image size normalization, and image cropping. The constructed training set is used for learning and training model parameters, and the test set is used to test the performance of the trained model.

[0060] In this embodiment, large-scale remote sensing image data, including visible light images and thermal infrared images, are collected from multiple remote sensing data sources and uniformly converted into a standardized format. In addition, the data can be cleaned and annotated to remove noise, correct distortion, and improve data quality through data enhancement technology to prepare for subsequent processing and analysis. The processed data set is then divided into a training set and a test set according to a predetermined ratio.

[0061] In one embodiment, step 1 specifically includes the following steps:

[0062] Step 1.1: Resize the image resolution in all datasets to 2000×2000;

[0063] Step 1.2: All visible light images and thermal infrared images can be cropped into 16-25 sub-images with a resolution of 512×512 from top to bottom and from left to right according to length and width, with an overlap rate of 10%-30%. In addition, the original image is also resized to 512×512. All cropped sub-images of the original image and the scaled original image are regarded as a complete data sample.

[0064] Step 1.3: Divide the processed samples into training set and test set in a 1:1 ratio, which are used for training and testing of the network model respectively.

[0065] Step 2: Construct a large model for intelligent extraction of multimodal remote sensing information to realize the extraction and coordination of multimodal remote sensing information. The large model for intelligent extraction of multimodal remote sensing information is divided into a visible light information encoder and a thermal infrared information encoder, which process visible light images and thermal infrared images respectively.

[0066] In one embodiment, step 2 includes the following steps:

[0067] In this embodiment, the structures of the visible light information encoder and the thermal infrared information encoder in step 2 are consistent, and each includes n (rounded to the number of cropped sub-images of each original image plus 1) parallel global high-order information embedding Transformer branches, such as Figure 2 As described above. The image in each data sample in the dataset is extracted by an independent global high-order information embedding visual Transformer branch. At each stage of the global high-order information embedding visual Transformer branch, the interaction of different branch features is realized through the cross-region information joint intelligent perception module.

[0068] Preferably, the above-mentioned global high-order information embedding Transformer is an enhancement of the original ViT, which can capture richer global information and identification information in the ground objects in the remote sensing large scene, thereby improving the model's perception of similar ground objects and constantly changing ground objects, and laying the foundation for accurate and comprehensive interpretation. The global high-order information embedding Transformer adds the embedding of high-order information to the original ViT.

[0069] See also Figure 3 , the global high-order information embedding Transformer module in this embodiment is specifically:

[0070] (1) The input vector of the Transformer module (the first-order input vector is directly the cropped sub-images of the corresponding sample and the cropped image after scaling) is embedded in the global high-order information by querying the weight matrix W Q , key weight matrix W K , value weight matrix W V The linear mapping is query vector Q, key vector K and value vector V;

[0071] (2) Perform covariance pooling operations on the query vector Q, key vector K, and value vector V, respectively, and then perform a dot multiplication operation on the obtained pooled vector and the corresponding vector before pooling to obtain a new vector Q n , K n and V n ;

[0072] (3) The vector Q n , K n and V n Send it to the multi-head self-attention module for self-attention mechanism calculation to get the output of the multi-head self-attention module Among them, softmax() represents the softmax activation function, d k Representative vector K n Dimensions;

[0073] (4) Through the first stacking & normalization layer, the output of the multi-head self-attention module and the input vector of the global high-order information embedding Transformer module are feature superimposed and then normalized to obtain the first self-attention fusion feature;

[0074] (5) The first self-attention fusion feature is sent to the fully connected layer, and then the output of the fully connected layer is superimposed on the output of the fully connected layer and the first self-attention fusion feature through the second superposition & normalization layer, and then a normalization operation is performed to obtain the second self-attention fusion feature, that is, the output feature of the global high-order information embedding Transformer module.

[0075] Among them, the covariance pooling operation is specifically:

[0076] The vectors Q, K, and V are respectively subjected to channel reduction operations through a convolution layer with a convolution kernel of 1×1 to obtain new feature vectors Q′, K′, and V′ with the same number of channels. Define C M Indicates the number of channels after reduction;

[0077] Calculate the correlation between channels in each feature vector Q′, K′ and V′ respectively, and get the dimension C M ×C M The correlation matrix M Q 、M K 、M V ; Among them, the calculation of the correlation of the feature graphs can be carried out by Pearson correlation coefficient, cosine similarity, element-by-element multiplication, etc. In the present invention, cosine similarity is preferred, that is, each correlation matrix is ​​obtained by calculating the cosine similarity between the feature graphs;

[0078] The new eigenvectors Q′, K′ and V′ are respectively correlated with the corresponding correlation matrix M Q 、M K 、M V Perform matrix multiplication to obtain the eigenvector S Q , S K , S V ;

[0079] The feature vector S Q , S K , S V Three new feature vectors S′ are obtained through a global average pooling layer and a convolution layer with a convolution kernel of 1×1 respectively. Q , S′ K , S′ V ;

[0080] Then the eigenvector S′ Q , S′ K , S′ V Perform dot multiplication with the corresponding vectors Q, K and V respectively to obtain the new vector Q n , K n and V n .

[0081] Preferably, see Figure 4 ,The processing process of joint intelligent perception of information in different regions is as follows:

[0082] First, each independent feature map of the global high-order information embedding Transformer branch will undergo global average pooling and global maximum pooling respectively, and the obtained vectors will be added pixel by pixel to obtain a vector of size 1×1×C′ (C′ is the set number of channels). Then these n vectors will be added element by element and passed through multiple layers of fully connected layers and ReLU activation functions to obtain a new vector S. The size of this global vector is also 1×1×C′. The above process formula is expressed as follows:

[0083]

[0084] Among them, f represents the fully connected layer, GAP represents the global average pooling, and F i Represents the output feature of the current stage of the i-th global high-order information embedding Transformer branch.

[0085] Next, the output feature map of each branch at the current stage will be multiplied by the global vector S to obtain a new feature map as the input of the next stage of each branch. The above process is expressed as follows:

[0086] V i =F i ⊙S+F i

[0087] Among them, V i Represents the input feature of the next stage of the i-th global high-order information embedding visual Transformer branch, and ⊙ represents the dot product operation.

[0088] Step 3: Construct a multimodal remote sensing information fusion module to achieve the fusion of visible light coding features and thermal infrared code features. The coding features from the visible light information encoder and the coding features from the thermal infrared information encoder are used to achieve the fusion of remote sensing multimodal features through the multimodal remote sensing information fusion module.

[0089] Preferably, the multimodal remote sensing information fusion module in step 3 is composed of multiple (e.g., 12) layers of multi-head self-attention mechanism layers. In each layer of the self-attention mechanism layer, two sets of parallel self-attention calculations are performed, one set of visible light coding features as Value, and thermal infrared coding features as Query and Key; the other set of thermal infrared coding features as Value, and visible light coding features as Query and Key. After each set of self-attention calculations is completed, the output feature maps of the two sets are added element by element, such as Figure 5 shown.

[0090] Step 4: construct a multimodal decoder for high-resolution remote sensing image interpretation based on the network structure of the encoder of the large language model, use the fused multimodal features as the input of the multimodal decoder, and obtain the interpretation result of the high-resolution remote sensing image based on its output;

[0091] Step 5: Based on the training set composed of multiple samples, the remote sensing image interpretation model composed of the multimodal remote sensing information extraction intelligent large model, the multimodal remote sensing information fusion module and the multimodal decoder is trained to learn the model parameters. When the preset training convergence conditions are met (the number of times reaches the upper limit or the set loss function value converges), the interpretation result of the input target image is obtained based on the trained remote sensing image interpretation model. The input target image includes the visible light image and the thermal infrared image of the same target object.

[0092] Example

[0093] The present embodiment provides a large model driven high-resolution remote sensing image interpretation method comprising the following steps:

[0094] Step S1: construct a large-scale multimodal high-resolution remote sensing image dataset (including visible light images and thermal infrared images), and perform image preprocessing on the image data therein to construct a training set and a test set.

[0095] In this embodiment, the shape of the visible light image and the thermal infrared image is 3×H×W, where H and W represent the height and width of the image.

[0096] In this embodiment, the image preprocessing is specifically set as follows:

[0097] The resolution of all visible light images and thermal infrared images was adjusted to 2000 × 2000.

[0098] All visible light images and thermal infrared images are cropped from top to bottom and from left to right into 16 sub-images with a resolution of 512×512, with an overlap rate of 10%. In addition, the original images with a resolution of 2000×2000 are also individually adjusted to 512×512. These 17 images are regarded as a complete modal data sample. The processed samples are divided into training and test sets in a 1:1 ratio, which are used for network training and testing, respectively.

[0099] Step S2, construct a large model for intelligent extraction of multimodal remote sensing information to realize the extraction of multimodal remote sensing information. The large model for intelligent extraction of multimodal remote sensing information is divided into a visible light information encoder and a thermal infrared information encoder, which process visible light images and thermal infrared images respectively. In this example, the dimensions of the visible light and thermal infrared input images are 3×512×512. The large model for intelligent extraction of multimodal remote sensing information consists of a visible light information encoder and a thermal infrared information encoder. The structures of these two encoders are consistent, and each encoder consists of 17 global high-order information embedded in the Transformer branch. The processing process of visible light images is explained below.

[0100] For each independent global high-order information embedding Transformer branch, the input image first passes through the embedding layer to obtain a tensor of dimension 32×32×768. Then, the tensor is processed continuously through 4 stages of multi-head self-attention mechanism, and the input tensor dimension before and after each stage is consistent. Each stage consists of 8 multi-head self-attentions. Finally, each independent global high-order information embedding Transformer branch will output a tensor of dimension 32×32×768. The 10 tensors are added pixel by pixel to obtain the final visible light coding feature tensor with dimension 32×32×768.

[0101] In each global high-order information embedding Transformer module of the global high-order information embedding Transformer branch, this embodiment will query the weight matrix W for Q, K and V vectors (the input vectors of the module are respectively Q , key weight matrix W K , value weight matrix W V Linear mapping is obtained) to mine and embed high-order information. Taking feature Q as an example to describe the processing process: First, feature Q will reduce the number of channels through 1×1 convolution to obtain a new feature Q / , with a dimension of 256×32×32, where 256 is the number of channels after reduction. Then, the feature Q is calculated / The correlation between the channels in the matrix M is obtained, and the dimension is 256×256. In this embodiment, it is obtained by calculating the cosine similarity between the feature maps. Next, the feature Q / Transform the dimension to (32×32)×256 and perform matrix multiplication with matrix M to obtain feature S Q , with a dimension of (32×32)×256. Finally, the feature S Q After a global average pooling and 1×1 convolution, a new feature with a dimension of 768×1×1 is obtained. This feature will be dot-multiplied with the input feature Q to obtain the output Q n The output feature Q n Keep the dimension consistent with the input feature Q.

[0102] In addition, the output of each stage of the 17 parallel branches will interact and perceive cross-regional information through the cross-regional information joint intelligent perception module, and the output of this module will be used as the input of the next stage of each branch. In the cross-regional information joint intelligent perception module, the tensors with a dimension of 32×32×768 from the 17 global high-order information embedded in the Transformer branch will first pass through global average pooling and global maximum pooling and add pixel by pixel to obtain 17 vectors with a dimension of 1×1×768. Then, these 17 1×1×768 vectors will be added pixel by pixel and passed through 3 layers of fully connected layers and ReLU activation function to obtain a new vector S, which also has a dimension of 1×1×768. The above process formula is expressed as follows:

[0103]

[0104] Among them, f represents the fully connected layer, GAP represents the global average pooling, and F i Represents the output feature of the i-th global high-order information embedding Transformer branch.

[0105] Next, the output feature map of each branch at the current stage will be multiplied by the global vector S to obtain a new feature map as the input of the next stage of each branch. The above process is expressed as follows:

[0106] V i =F i ⊙S+F i

[0107] Among them, V i Represents the input feature of the next stage of the i-th global high-order information embedding visual Transformer branch, and ⊙ represents the dot product operation.

[0108] Step S3, the coded features from the visible light information encoder and the coded features from the thermal infrared information encoder are fused through the multimodal remote sensing information fusion module. The multimodal remote sensing information fusion module consists of 12 layers of bidirectional interactive attention modules. The dimensions of all tensors remain unchanged before and after passing through the bidirectional interactive attention module. In the bidirectional interactive attention module, two sets of parallel self-attention calculations are performed. One set of visible light coded features is used as Value, with a dimension of 32×32×768, and the thermal infrared coded features are used as Query and Key, both with a dimension of 32×32×768; the other set of thermal infrared coded features is used as Value, with a dimension of 32×32×768, and the visible light coded features are used as Query and Key, both with a dimension of 32×32×768. After each set of self-attention calculations is completed, the two tensors with a dimension of 32×32×768 are added element by element to produce the final output with a dimension of 32×32×768.

[0109] Step S4, based on the encoder of the BERT model, the interpretation of high-resolution remote sensing images is realized. The tensor dimension after multimodal feature fusion is 32×32×768. Through the dimension change operation, the tensor is changed to a vector of dimension 1024×768. First, the tensor is element-by-element added through the Embedding layer and the position encoding vector to obtain the input of the BERT encoder. Then, the vector passes through 12 consecutive Transformer encoding layers, each layer consisting of 12 multi-head self-attention mechanisms. When the vector passes through the encoding layers of these 12 consecutive Transformers, the dimension always remains 1024×768. Finally, the softmax function is used to predict the word to which each Token belongs to produce the final interpretation result.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0111] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A large model driven high-resolution remote sensing image interpretation method, characterized in that: The following steps are involved: Step 1: construct a large-scale multimodal high-resolution remote sensing image dataset, which includes visible light images and thermal infrared images; and perform image preprocessing on the image data in the multimodal high-resolution remote sensing image dataset to obtain a training set for model training; The image preprocessing includes unifying the image data formats of each modality, normalizing the image size, and cropping the image; the size and number of the cropped sub-images of each image are consistent, and then the cropped image is scaled to the size of the cropped sub-image; for each modality, all the cropped sub-images and the cropped image scaled to the size of the cropped sub-image are regarded as one sample; Step 2, constructing a multimodal remote sensing information extraction intelligent large model for multimodal remote sensing information encoding; The multimodal remote sensing information extraction intelligent large model includes a visible light information encoder and a thermal infrared information encoder. The visible light information encoder is used to extract visible light coding features of visible light images, and the thermal infrared information encoder is used to extract thermal infrared coding features of thermal infrared images. Step 3, construct a multimodal remote sensing information fusion module to perform multimodal feature fusion on visible light coding features and thermal infrared coding features, and output the fused multimodal features; Step 4: construct a multimodal decoder for high-resolution remote sensing image interpretation based on the network structure of the encoder of the large language model, use the fused multimodal features as the input of the multimodal decoder, and obtain the interpretation result of the high-resolution remote sensing image based on its output; Step 5: Based on the training set composed of multiple samples, the remote sensing image interpretation model composed of the multimodal remote sensing information extraction intelligent large model, the multimodal remote sensing information fusion module and the multimodal decoder is trained on the model parameters. When the preset training convergence conditions are met, the interpretation result of the input target image is obtained based on the trained remote sensing image interpretation model.

2. The method according to claim 1, characterized in that In step 1, image cropping includes: based on a specified overlap ratio, cropping the image after the image size is normalized according to a specified cropping order.

3. The method according to claim 2, characterized in that In step 2, the visible light information encoder and thermal infrared information encoder in the multimodal remote sensing information extraction intelligent large model have the same structure, which respectively include n parallel global high-order information embedding Transformer branches, where the value of n is the number of cropped sub-images of each image plus 1; Among them, each global high-order information embedding Transformer branch includes m cascaded global high-order information embedding Transformer modules, where m is an integer greater than 2; each global high-order information embedding Transformer module corresponds to one stage; For the output feature map of the same-order global high-order information embedding Transformer module in n branches, information joint perception is performed through the cross-regional information joint intelligent perception module, and the information joint perception features of each branch are output, and then output to the next-order global high-order information embedding Transformer module of the corresponding branch.

4. The method according to claim 3, characterized in that The global high-level information embedding Transformer module is specifically set as: The input vector of the global high-order information embedding Transformer module is respectively obtained by querying the weight matrix W Q , key weight matrix W K , value weight matrix W V The linear mapping is query vector Q, key vector K and value vector V; Perform covariance pooling operations on the query vector Q, key vector K, and value vector V respectively, and then perform a dot multiplication operation on the obtained pooled vector and the corresponding vector before pooling to obtain a new vector Q n , K n and V n ; The vector Q n , K n and V n Send it to the multi-head self-attention module for self-attention mechanism calculation to get the output of the multi-head self-attention module Among them, softmax() represents the softmax activation function, d k Representative vector K n Dimensions; The output of the multi-head self-attention module and the input vector of the global high-order information embedding Transformer module are superimposed and normalized through the first superposition & normalization layer to obtain the first self-attention fusion feature; The first self-attention fusion feature is sent to the fully connected layer, and then the output of the fully connected layer is normalized after feature superposition of the output of the fully connected layer and the first self-attention fusion feature through the second superposition & normalization layer to obtain the second self-attention fusion feature, that is, the output feature of the global high-order information embedded in the Transformer module.

5. The method according to claim 4, characterized in that The covariance pooling operation is as follows: The vectors Q, K, and V are respectively subjected to channel reduction operations through a convolution layer with a convolution kernel of 1×1 to obtain new feature vectors Q′, K′, and V′ with the same number of channels. Define C M Indicates the number of channels after reduction; Calculate the correlation between channels in each feature vector Q′, K′ and V′ respectively, and get the dimension C M ×C M The correlation matrix M Q 、M K 、M V ; Among them, the calculation of the correlation of the feature graphs can be carried out by Pearson correlation coefficient, cosine similarity, element-by-element multiplication, etc. In the present invention, cosine similarity is preferred, that is, each correlation matrix is ​​obtained by calculating the cosine similarity between the feature graphs; The new eigenvectors Q′, K′ and V′ are respectively correlated with the corresponding correlation matrix M Q 、M K 、M V Perform matrix multiplication to obtain the eigenvector S Q , S K , S V ; The feature vector S Q , S K , S V Three new feature vectors S′ are obtained through a global average pooling layer and a convolution layer with a convolution kernel of 1×1 respectively. Q , S′ K , S′ V ; Then the eigenvector S′ Q , S′ K , S′ V Perform dot multiplication with the corresponding vectors Q, K and V respectively to obtain the new vector Q n , K n and V n .

6. The method according to claim 3, characterized in that The specific execution process of the cross-region information joint intelligent perception module includes: (1) Each feature map of the input global high-order information embedded in the Transformer branch is subjected to global average pooling and global maximum pooling respectively, and then the two pooling results are added element by element to obtain the first feature vector of each branch; (2) The first eigenvectors of all branches are added element by element and then passed through several layers of fully connected layers and a layer of ReLU activation function to obtain the global eigenvector; (3) Each feature map of the global high-order information embedded in the Transformer branch is point-multiplied with the global feature vector to obtain the information joint perception feature of each branch, which is used as the input feature of the global high-order information embedded in the next stage of the corresponding branch Transformer module.

7. The method according to claim 1, characterized in that In the multimodal remote sensing information fusion module, visible light coding features and thermal infrared features are fused through a cross-headed self-attention mechanism to achieve multimodal remote sensing feature fusion.

8. The method according to claim 1, characterized in that The multimodal remote sensing information fusion module includes multiple layers of multi-head self-attention mechanism layers. In each layer of the multi-head self-attention mechanism layer, two sets of parallel self-attention calculations are performed, one of which uses visible light encoding features as value vectors and thermal infrared encoding features as query vectors and key vectors; the other set uses thermal infrared encoding features as value vectors and visible light encoding features as query vectors and key vectors; and after each set of self-attention calculations is completed, the output feature maps of the two sets are added element by element as the input features of the next layer of the multi-head self-attention mechanism layer.

9. The method according to claim 1, characterized in that In step 5, the multimodal remote sensing information extraction intelligent large model in the remote sensing image interpretation model is first trained through the random size mask self-supervised learning strategy, which specifically includes: A training object is selected from the visible light information encoder and the thermal infrared information encoder, and a universal decoder is connected after the training object, and the decoder is used to reconstruct the input image of the training object; For the input image of the training object, randomly select one of the specified multiple image scales as the size of the current processing image; On the image of this size, a certain proportion of image blocks are randomly selected for mask processing. The scale of the mask matrix M used in the mask processing cannot exceed the size of the currently selected processing image; The loss function of the random size mask self-supervised learning strategy for the training object is set to: Where H, W, C represent the height, width and number of channels of the input image respectively; ∑M represents the total number of unmasked images in the mask matrix M, I hwc and Represent the pixel values ​​of the original image and the reconstructed image at the image position (h, w, c) respectively, where the original image is the input image of the training object. When the preset training convergence conditions are met, the trained visible light information encoder and thermal infrared information encoder are obtained in the form of parameter sharing based on the trained training objects.

10. The method according to claim 1, characterized in that In step 5, weight parameter training is performed on the multimodal remote sensing information fusion module and the multimodal decoder in the remote sensing image interpretation model based on the pre-trained multimodal remote sensing information extraction intelligent large model.

Images