A multi-source remote sensing data-based visual language model cross-modal alignment method

Abstract

The embodiment of the application discloses a visual language model cross-modal alignment method based on multi-source remote sensing data, which can improve the completion effect of remote sensing intelligent tasks. The method adopts a position coding technology to extract visual features of visible light images and invisible light images in multi-modal remote sensing data, then performs cross-modal feature fusion based on a cross attention mechanism, and then performs image-text alignment processing through a projection layer to obtain target features, and takes the target features as input data of a large language model to support the large language model to complete remote sensing intelligent tasks.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for cross-modal alignment of visual language models based on multi-source remote sensing data. Background Technology

[0002] Remote sensing technology, as a crucial means of acquiring information about the Earth's surface, is widely used in key areas such as agriculture, environmental monitoring, disaster early warning, and urban planning. With the continuous increase in the number of various remote sensing satellites, the resolution of remote sensing data across multiple dimensions, including time, space, spectrum, and radar, is constantly improving, forming a massive, multi-source, and high-dimensional remote sensing data system. This data includes not only traditional visible light images such as red-green-blue (RGB) optical images, but also multispectral images, hyperspectral images, and invisible light images such as synthetic aperture radar (SAR) images with strong penetration and all-weather acquisition capabilities, greatly enriching the sources of information for Earth observation.

[0003] However, current large remote sensing models have limited processing capabilities for non-visible light remote sensing modalities such as SAR and multispectral, making it difficult to deeply integrate natural language with multi-source remote sensing data (i.e., multimodal remote sensing data, which includes visible light images and invisible light images), resulting in poor performance in completing remote sensing intelligent tasks (such as cross-modal retrieval, intelligent question answering, and automatic interpretation). Summary of the Invention

[0004] The purpose of this application is to provide a cross-modal alignment method for visual language models based on multi-source remote sensing data, which can deeply integrate natural language with multi-modal remote sensing data, thereby improving the completion effect of remote sensing intelligent tasks.

[0005] In a first aspect, embodiments of this application provide a method for cross-modal alignment of visual language models based on multi-source remote sensing data, the method comprising: A multimodal remote sensing large model is constructed, which includes a data preprocessing module, a feature fusion module, and a feature alignment module; In response to receiving multimodal remote sensing data, the data preprocessing module divides the multimodal remote sensing data into multiple image patches and inputs the multiple patches into the feature fusion module. The multimodal remote sensing data includes at least one visible light image and at least one invisible light image. The feature fusion module performs position encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain the first modality visual features associated with the invisible light image and the second modality visual features associated with the visible light image; The feature fusion module is based on a cross-attention mechanism. It uses the second modality visual features as the core representation basis and supplements the core representation basis with the first modality visual features to obtain multimodal visual fusion features. The multimodal visual fusion features are then input into the feature alignment module. The feature alignment module performs image-text alignment processing on the input text and the multimodal visual fusion features through a pre-trained projection layer to obtain target features, and uses the target features as input data for the large language model to support the large language model in completing remote sensing intelligent tasks.

[0006] A second aspect of this application provides a multimodal remote sensing large model, the multimodal remote sensing large model including a data preprocessing module, a feature fusion module, and a feature alignment module, wherein: The data preprocessing module is used to, in response to receiving multimodal remote sensing data, divide the multimodal remote sensing data into multiple image patches and input the multiple patches into the feature fusion module. The multimodal remote sensing data includes at least one visible light image and at least one invisible light image. The feature fusion module is used to perform position encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain a first modal visual feature associated with the invisible light image and a second modal visual feature associated with the visible light image; and to use the second modal visual feature as the core representation basis based on the cross-attention mechanism, and to supplement the core representation basis with the first modal visual feature to obtain a multimodal visual fusion feature, and input the multimodal visual fusion feature to the feature alignment module; The feature alignment module is used to perform image-text alignment processing on the input text and the multimodal visual fusion features through a pre-trained projection layer to obtain target features, and use the target features as input data for the large language model to support the large language model in completing remote sensing intelligent tasks.

[0007] A third aspect of this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the cross-modal alignment method for visual language models based on multi-source remote sensing data as described in the first aspect.

[0008] A fourth aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the cross-modal alignment method for visual language models based on multi-source remote sensing data as described in the first aspect.

[0009] A fifth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the cross-modal alignment method for visual language models based on multi-source remote sensing data as described in the first aspect.

[0010] As can be seen from the above technical solution, this application employs positional encoding technology to extract visual features from visible and invisible light images in multimodal remote sensing data. Then, based on a cross-attention mechanism, the second modality visual features are used as the core representation basis, and the first modality visual features are used to supplement the core representation basis. This maintains the dominant position of information in the visible light image as the main modality and effectively integrates information such as the spectral structure implied in the invisible light image into the optical semantic space, thereby achieving a unified representation of multimodal remote sensing data and providing a technical foundation for subsequent deep fusion. Furthermore, this application introduces a projection layer between the visual layer and the language layer for image-text alignment processing, that is, performing a unified mapping from visual space to language space on the input text and multimodal visual fusion features to obtain target features. This enables deep fusion of natural language and multimodal remote sensing data, allowing the model to have the ability to model remote sensing visual language for multimodal remote sensing data (visible and invisible light images), thereby effectively improving the completion effect of remote sensing intelligent tasks. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the implementation of a cross-modal alignment method for visual language models based on multi-source remote sensing data, provided in this application embodiment; Figure 2 A schematic diagram of an image after patch division provided in an embodiment of this application; Figure 3 A schematic diagram of aligned visible light and invisible light images provided for an embodiment of this application; Figure 4 A schematic diagram of an MRSChat model architecture provided for an embodiment of this application; Figure 5 A schematic diagram illustrating a learnable positional encoding provided in an embodiment of this application; Figure 6 A schematic diagram of a multi-turn question-and-answer format of MRSChat provided for an embodiment of this application; Figure 7 A schematic diagram of a data processing flow provided in an embodiment of this application; Figure 8 This is a schematic diagram illustrating a patch selection method provided in an embodiment of this application. Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] The modal differences in remote sensing data (also known as remote sensing images) are significant. How to efficiently unify the representation and understanding of multimodal remote sensing data has become a key challenge in current intelligent remote sensing processing. Especially in the field of remote sensing visual language modeling, the deep integration of natural language and remote sensing images holds promise for further improving application performance such as automatic interpretation capabilities, cross-modal retrieval, and intelligent question answering, and is currently a research hotspot. Current large-scale remote sensing models mainly include: 1. RSGPT model: It is obtained by fine-tuning general large-scale visual language models (such as BLIP, BLIP-2, etc.) on small-scale, high-quality RGB remote sensing images and text pairs.

[0015] 2. GeoChat model: It is a remote sensing visual language model that supports regional reasoning and visual positioning capabilities. It has certain multimodal capabilities and can combine images and regional text for question answering and reasoning.

[0016] 3. SpectralGPT model: It is the first large-scale generative pre-trained model specifically designed for multi / hyperspectral remote sensing images. It uses techniques such as 3D token masking and multi-target reconstruction to effectively model spatial-spectral joint features.

[0017] The RSGPT model has the following main drawbacks: 1) Poor generalization effect: RSGPT models typically rely on a limited number of training data, primarily building multimodal perception capabilities based on standard RGB visible light images and general natural language. However, in the field of remote sensing, imaging conditions are highly complex. For example, different satellite platforms, different imaging times (seasons, day and night), different geographical regions (cities, farmland, mountains), and even different task types (disaster monitoring, ground feature identification, environmental assessment) can all lead to significant differences in images in terms of illumination, resolution, and texture structure.

[0018] 2) Individual model fine-tuning is required for each specific task: When dealing with different remote sensing tasks, RSGPT models typically require individual fine-tuning of model parameters for each task. This approach not only consumes significant time and computational resources but also limits the model's versatility and adaptability in open environments.

[0019] The GeoChat model has the following main drawbacks: 1) Prone to hallucinations: The addition of visual localization and region-level inference significantly increases the demands on the model's detail processing and spatial understanding capabilities. In complex remote sensing scenarios, the model may overinterpret or incorrectly infer selected areas, especially when lacking real-world geographical knowledge or specialized remote sensing training data. This can easily lead to erroneous responses and the so-called multimodal illusion phenomenon. For example, in areas with similar textures such as vegetation, water bodies, and cities, the model might identify buildings as roads or paddy fields as water, resulting in analytical errors.

[0020] 2) Unable to recognize multimodal remote sensing images: The GeoChat model architecture only supports RGB visible light image input. Its underlying visual encoders (such as CLIP and LLaVA) are primarily trained on natural images, lacking an understanding mechanism for the non-visible bands unique to remote sensing images (corresponding to invisible light images, such as near-infrared (NIR), short-wave infrared (SWIR), SAR, and hyperspectral images (HSI)). However, these bands often carry crucial information for identifying ground features. Due to the GeoChat model architecture's lack of multispectral (i.e., multimodal) perception capabilities, it cannot effectively model and fuse these data types, nor can it understand the physical meaning and spatiotemporal relationships between multiple bands. Therefore, when processing tasks such as flood monitoring, forest fire identification, and urban change detection, the GeoChat model's performance is significantly limited, making it unsuitable for high-precision, multi-source heterogeneous remote sensing data fusion analysis and decision-making tasks.

[0021] The main drawback of the SpectralGPT model is its lack of language modeling capabilities, making it unable to achieve text-image interaction. The architecture of the SpectralGPT model only focuses on spatial-spectral modeling of spectral images and does not incorporate text modality. It cannot perform language-related tasks (i.e., remote sensing intelligent tasks) such as remote sensing image-text matching, description generation, and cross-modal question answering. This makes it lack practical application capabilities in areas such as intelligent remote sensing understanding and semantic parsing.

[0022] Based on the above analysis, and addressing the problem in related technologies that large remote sensing models struggle to deeply integrate natural language with multimodal remote sensing data (including visible and invisible light images), this application provides a cross-modal alignment method for visual language models based on multi-source remote sensing data. Based on a cross-attention mechanism, it achieves multimodal visual feature fusion for visible and invisible light images and introduces a projection layer for image-text alignment processing. This enables deep integration of natural language with multimodal remote sensing data, allowing the model to possess the ability to model remote sensing visual language for multimodal remote sensing data (visible and invisible light images), thereby effectively improving the completion of remote sensing intelligent tasks.

[0023] See Figure 1 The diagram shown is an implementation flowchart of a cross-modal alignment method for visual language models based on multi-source remote sensing data provided in this application. The method may include the following steps: Step S101: Construct a multimodal remote sensing large model, which includes a data preprocessing module, a feature fusion module, and a feature alignment module.

[0024] In practical implementation, the large-scale multimodal remote sensing model described in this application (also known as MrsChat) can be constructed by introducing a cross-attention mechanism in the visual layer and adding a projection layer between the visual layer and the language layer, based on the Large Language and Vision Assistant (LLaVA) architecture.

[0025] Step S102: In response to receiving multimodal remote sensing data, the data preprocessing module divides the multimodal remote sensing data into multiple image patches and inputs the multiple patches into the feature fusion module. The multimodal remote sensing data includes at least one visible light image and at least one invisible light image.

[0026] The multimodal remote sensing large model uses the multimodal remote sensing data as input to the model's image dimension. For example, the input to the multimodal remote sensing large model in the image dimension (i.e., the multimodal remote sensing data) can be represented as RGB+X, where RGB represents the RGB optical image (corresponding to a visible light image), and X represents the image composed of other spectra besides R, G, and B spectra in the multispectral system (corresponding to an invisible light image), such as a SAR image, a multispectral image, or an infrared image.

[0027] In practical implementation, each image in the multimodal remote sensing data can be divided into multiple patches according to a preset number of patches or patch size. Taking a preset image patch size of 14×14 as an example, dividing a 504×504 resolution image into patches can divide the image into 1296 patches. The image after patch division is shown below. Figure 2 As shown.

[0028] Optionally, the data preprocessing module can adjust each image in the multimodal remote sensing data to a set resolution before dividing the data into patches (which can be determined based on the input size of the subsequent encoder).

[0029] For example, for a specific image in multimodal remote sensing data, the data preprocessing module can use a visual encoder (such as CLIP-VIT-336) to increase the resolution of the image to a set resolution (such as 504×504) through bi-triphase interpolation.

[0030] For example, for a certain image in multimodal remote sensing data, the data preprocessing module can crop the image to adjust its resolution to a set resolution (e.g., 504×504).

[0031] Step S103: The feature fusion module performs position encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain the first modal visual feature associated with the invisible light image and the second modal visual feature associated with the visible light image.

[0032] In practice, after each image in the multimodal remote sensing data is divided into multiple patches, the multiple patches associated with each image are used as multiple tokens (which are the basic units used by the large language model to represent natural language text) and input into the encoder for position encoding to obtain the visual features associated with each image (i.e., the first modal visual features associated with the invisible light image and the second modal visual features associated with the visible light image), so as to facilitate subsequent cross-fusion and alignment.

[0033] Step S104: The feature fusion module uses the second modality visual features as the core representation basis based on the cross-attention mechanism, and uses the first modality visual features to supplement the core representation basis to obtain multimodal visual fusion features, and inputs the multimodal visual fusion features to the feature alignment module.

[0034] In practical implementation, unlike the self-attention mechanism which focuses on features within the same modality, this application employs cross attention to handle data interaction between different modalities. This allows visual features of two different modalities (i.e., the first modal visual features associated with the invisible light image and the second modal visual features associated with the visible light image, such as SAR features and optical features, or multispectral features and optical features, etc.) to pay attention to each other during computation (i.e., perform cross attention fusion), thereby achieving cross-modal alignment and information interaction.

[0035] When performing cross-attention fusion on the first modality visual features and the second modality visual features, this application further uses the second modality visual features as the core representation basis and uses the first modality visual features to supplement the core representation basis, thereby enhancing the perception of supplementary information from the first modality visual features by the second modality visual features (i.e., optical features). It should be noted that first-modality visual features, such as SAR features, can provide structural, textural, or hydrological information that differs from optical features. Therefore, optical features, through a cross-attention mechanism, can "search" for the most relevant information within first-modality visual features like SAR features to supplement themselves, thereby enhancing their semantic meaning.

[0036] Taking the first modality visual features as an example, which at least include SAR features and multispectral features, the above-mentioned cross-attention fusion design has at least the following advantages: 1) Maintain the dominant information mode of optical images (i.e., visible light images); 2) Use information such as texture structure, landform, and water body boundaries from SAR features to supplement optical features; 3) By utilizing the rich band information in multispectral features, the reflection characteristics of ground objects under different electromagnetic bands can be effectively captured, thereby providing stronger semantic discrimination capabilities.

[0037] 4) Makes the main modality more robust in remote sensing intelligent tasks such as semantic generation tasks (e.g., dialogue response).

[0038] Therefore, based on the above-mentioned cross-attention fusion design, the spectral structure hidden in invisible light images such as multispectral images can be fused into the optical semantic space, thereby enabling the model to have a stronger ability to distinguish between "farmland and grassland", "dry and wet soil", and "healthy vegetation and withered vegetation", which can improve the performance of remote sensing intelligent tasks.

[0039] Step S105: The feature alignment module performs image-text alignment processing on the input text and the multimodal visual fusion features through a pre-trained projection layer to obtain target features, and uses the target features as input data for the large language model to support the large language model in completing remote sensing intelligent tasks.

[0040] In specific implementation, the projection layer can be a linear projection layer built based on a multilayer perceptron (MLP) adapter, used to achieve a unified mapping from the visual to the language space, providing basic support for subsequent large language models to complete remote sensing intelligent tasks such as multimodal question answering, reasoning, and classification.

[0041] It should be noted that the multimodal remote sensing data processed in this application has the following characteristics: these data are highly consistent in terms of imaging time and geographical location, meaning they originate from observations in the same time period and region, but exhibit different modal characteristics due to being mounted on different types of remote sensing sensors. This type of data has already been aligned in terms of spatial resolution, image size, and pixel level, possessing a high degree of well-alignment, thus providing a good foundation for fusion.

[0042] In practical applications, the above-mentioned aligned remote sensing data (for example, see...) Figure 3 The diagram showing aligned visible and invisible light images can be obtained from a wealth of existing objective data resources in the field of remote sensing. Many remote sensing satellite systems (such as the Sentinel series, Landsat series, and Gaofen series) simultaneously acquire multiple modal images during imaging missions and perform registration and correction through ground systems, providing a rich data foundation for multimodal modeling. Therefore, the core objective of this application is to design an efficient, accurate, and generalizable multimodal perception and understanding system based on these aligned multi-source remote sensing data to support complex remote sensing intelligent tasks such as target detection, ground feature recognition, scene understanding, and natural language question answering. Through in-depth mining and collaborative modeling of the features of each modality, this application can effectively improve the robustness and adaptability of the remote sensing analysis system in complex environments.

[0043] As can be seen from the above technical solution, this application employs positional encoding technology to extract visual features from visible and invisible light images in multimodal remote sensing data. Then, based on a cross-attention mechanism, the second modality visual features are used as the core representation basis, and the first modality visual features are used to supplement the core representation basis. This maintains the dominant position of information in the visible light image as the main modality and effectively integrates information such as the spectral structure implied in the invisible light image into the optical semantic space, thereby achieving a unified representation of multimodal remote sensing data and providing a technical foundation for subsequent deep fusion. Furthermore, this application introduces a projection layer between the visual layer and the language layer for image-text alignment processing, that is, performing a unified mapping from visual space to language space on the input text and multimodal visual fusion features to obtain target features. This enables deep fusion of natural language and multimodal remote sensing data, allowing the model to have the ability to model remote sensing visual language for multimodal remote sensing data (visible and invisible light images), thereby effectively improving the completion effect of remote sensing intelligent tasks.

[0044] In an optional embodiment, the feature fusion module, based on a cross-attention mechanism, uses the second modality visual features as the core representation basis and supplements the core representation basis with the first modality visual features to obtain multimodal visual fusion features, including: The feature fusion module maps the second modality visual features into a query vector and the first modality visual features into a key vector and a numerical vector, based on a trainable parameter matrix. The feature fusion module determines the multimodal visual fusion feature based on the query vector, the key vector, and the numerical vector.

[0045] In this embodiment, a trainable parameter matrix (e.g.) is used. , , , The original feature dimension is used to project visual features from different modalities into a compatible computation space to obtain query vectors, key vectors, and value vectors. In other words, visual features from different sources are converted into "query-key-value" (QKV) triples in a unified semantic space so that cross-modal feature association and matching can be achieved through a cross-attention mechanism. This allows the model to calculate the semantic attention (i.e., weights) of the second modal visual features (i.e., optical features) to different parts of the first modal visual features (such as SAR features or multispectral features) through the cross-attention mechanism. Finally, a fused feature (i.e., multimodal visual fusion feature) is output, which is a result of the weighted fusion of the second modal visual features with respect to the first modal visual features, thereby achieving cross-modal fusion.

[0046] Optionally, the multimodal visual fusion features are determined by the following formula:

[0047] in, Represents multimodal visual fusion features. Represents the query vector. Represents the key vector. Represents a numerical vector. This represents the original feature dimension before mapping.

[0048] Understandably, Softmax is a commonly used activation function, primarily used to transform a set of real numbers (which can be positive, negative, or zero) into a probability distribution where each value is between (0, 1) and the sum is 1. For example, for a vector... The output of softmax is:

[0049] in, and express The i-th and j-th elements, where n represents The number of elements in the middle.

[0050] In this embodiment, optical features "search" for the most relevant information within the visual feature space of invisible light images, such as SAR features, and then weight and enhance them. This fusion mechanism effectively compensates for the shortcomings of optical modalities in terms of structure, penetration capability, and terrain recognition. Especially in the subsequent image-text dialogue generation stage, this application uses optical images as the main semantic benchmark, enabling the fused features to more comprehensively reflect the scene content. This allows the model to answer multimodal integrated understanding tasks such as "Is there a risk of flooding in this area?" and "Is the vegetation healthy?". In other words, in subsequent remote sensing intelligent tasks (such as natural language question answering tasks), the fused visual features can provide reliable information support for the large language model, thereby guiding the large language model to generate more remote sensing-specific answers to questions such as "What is the health status of the vegetation in this area?", "Are there potential areas of crop disease?", and "Is soil salinization likely?".

[0051] In an optional embodiment, the feature fusion module performs positional encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain a first modality visual feature associated with the invisible light image and a second modality visual feature associated with the visible light image, including: The feature fusion module uses an unlearnable sine and cosine absolute position encoding method to encode the position of each patch associated with each invisible light image, thereby obtaining the first modality visual features. The feature fusion module uses a learnable positional encoding method to encode the position of each patch associated with a visible light image, thereby obtaining the second modality visual features.

[0052] In practical implementation, for first-modality visual features such as multispectral and SAR features, a non-learnable sine and cosine absolute position encoding method is used for positional encoding. This method introduces explicit spatial positional information while maintaining cross-modal alignment consistency, thereby helping the model understand the semantic relationships of the same spatial regions in different modal images during the cross-fusion stage. Furthermore, the non-learnable sine and cosine absolute position encoding method has the advantages of requiring no training and being generalizable to any input size, thus also contributing to improving the model's stability and generalization ability in large-scale remote sensing image processing.

[0053] For the second modality visual features, a learnable positional encoding method is used to encode their positions, so as to better capture the spatial structural relationship between patches and facilitate automatic adaptation to image content and downstream task requirements through the training process, thereby improving the ability to model spatial context and enhancing the effect of visual semantic alignment.

[0054] In this embodiment, the present application adopts a processing strategy of "heterogeneous position coding + cross-modal cross-fusion", that is: a learnable position coding method is used for optical images, and an unlearnable sine and cosine absolute position coding method is used for microwave remote sensing images such as SAR images, and cross-modal cross-fusion is performed after the position coding is completed. This processing strategy has at least the following advantages: 1. For optical remote sensing images such as RGB images, due to their rich texture information and complex scene content, the use of learnable positional encoding methods can better adapt to the local structural differences in different regions, thereby improving the feature representation capability. 2. For microwave remote sensing images such as SAR images, since their imaging mechanism is different from that of optical images, they usually have obvious speckle noise. Using a fixed position coding method, namely the non-learnable sine and cosine absolute position coding method, is beneficial to more stably represent their spatial geometric structure information, thereby improving the stability and generalization ability of model training. 3. By performing cross-modal fusion after location encoding, features of different modalities can interact more fully based on spatial location information alignment, thereby effectively leveraging the complementary advantages of optical images and SAR images in terms of texture and structural information, and improving the expressive effect of fused features.

[0055] Therefore, this processing strategy combines the imaging characteristics of heterogeneous sensors and adopts a differentiated position encoding mechanism for different modal data. It can both preserve spatial structure information and enhance the alignment consistency between modalities, thereby improving the model's spatial perception capability and effectively enhancing the overall robustness and generalization ability of the model.

[0056] Optionally, the feature fusion module employs a non-learnable sine and cosine absolute position encoding method to perform position encoding on the patch associated with each invisible light image, obtaining the first modality visual features, including: The feature fusion module uses each patch associated with each invisible light image as a sequence element to construct the first sequence associated with each invisible light image; The feature fusion module uses a first encoder to perform position encoding on the first sequence associated with each invisible light image to obtain the first modal visual features associated with each invisible light image; The first encoder is configured to: encode the first sequence using a non-learnable sine absolute position encoding method when the preset encoding dimension is even; and encode the first sequence using a non-learnable cosine absolute position encoding method when the preset encoding dimension is odd.

[0057] In practical implementation, for each invisible light image in multimodal remote sensing data, after the invisible light image is divided into several non-overlapping or overlapping patches, each patch is treated as a sequence element to construct a first sequence. Then, the first encoder encodes the first sequence according to a preset encoding dimension using an unlearnable sine and cosine absolute position encoding method, thus obtaining the corresponding position encoding sequence (i.e., the first modality visual feature). The shape of the position encoding sequence can be represented as [seq_len, d_model], where d_model represents the total embedding dimension of the first encoder; seq_len represents the sequence length, which is equal to the number of elements contained in the first sequence input into the first encoder. That is, for a single invisible light image, the length of its corresponding position encoding sequence, seq_len, is equal to the number of patches divided into the invisible light image. For example, if the invisible light image is divided into patches of size P×P, and the size of the invisible light image is H×W, then: .

[0058] Optionally, the unlearnable sinusoidal absolute position encoding method is represented as follows:

[0059] The unlearnable cosine absolute position encoding method is represented as follows:

[0060] in, , Indicates the visual features of the first modality. This represents the index of a sequence element in the first sequence. Indicates the preset encoding dimension ( Sine coding is used when the number is even. (Cosine encoding is used when the number is odd). This represents the total embedding dimension of the first encoder, which can be 1024, etc.

[0061] Optionally, the feature fusion module employs a learnable positional encoding method to perform positional encoding on the patch associated with each visible light image, thereby obtaining second-modality visual features, including: The feature fusion module uses each patch associated with each visible light image as a sequence element to construct a second sequence associated with each visible light image. The feature fusion module uses a second encoder to perform position encoding on the second sequence associated with each visible light image, thereby obtaining the second modal visual features associated with each visible light image; The second encoder is configured to perform position encoding using the following learnable position encoding method:

[0062] in, Indicates the visual features of the second modality. This represents the location embedding matrix constructed based on learnable parameters. Matrix elements Indicates the position of the sequence element in the second sequence. The corresponding embedding vector.

[0063] For example, see Figure 4 The diagram shown illustrates the MRSChat model architecture. The model's visual layer (i.e., the feature fusion module) includes two encoders for positional encoding (i.e., the first encoder and the second encoder) and a cross-attention module for feature fusion, wherein: The first encoder can be constructed based on Transformer or a custom Convolutional Neural Network (CNN) and is used to encode the position of the first sequence. The second encoder utilizes a vision model based on the Transformer architecture (Vision Transformer, ViT), such as CLIP, to encode the input second sequence in a learnable positional encoding manner. Understandably, at the start of training, the model can randomly initialize a parameter matrix as a positional embedding matrix, where each element corresponds to an embedding vector at a different position, and update the positional embedding matrix during training using the contrastive learning loss of CLIP. The cross-attention module can be constructed based on multiple sequentially connected feedforward layers and cross-attention units, and is used to interact and fuse the first sequence features and the second sequence features after position encoding.

[0064] It should be noted that this application introduces a first encoder into the model to enhance its adaptability to geometric structures, texture patterns, and radar scattering features in invisible light images such as SAR images. This application also introduces a second encoder into the model to improve the model's visual language alignment with visible light images such as RGB images by leveraging the semantic understanding capabilities pre-trained on second encoders such as CLIP. Furthermore, the heterogeneous encoder design facilitates modal decoupling modeling, enabling each modality to extract effective features from its preferred information domain, thus creating conditions for subsequent fusion.

[0065] For example, refer to Figure 5 The diagram illustrates a learnable positional encoding. For a second sequence of length L (i.e., image patches), the embedding vector dimension for each position is d (1024). The positional embedding matrix constructed based on the learnable parameters... .

[0066] In an alternative embodiment, considering that in traditional large-scale remote sensing models (i.e., remote sensing dialogue models, such as GeoChat, LLaVA-RS, etc.), the model input is usually RGB images, and the training data is usually a single-turn format of "image + instruction + response," the design of instructions is mainly focused on tasks supported by visible light information, such as ground feature recognition, scene description, and target localization. This results in the model achieving certain results in visual-language alignment, but severely underutilizing remote sensing multimodal information.

[0067] To address the aforementioned issues and enhance the model's practicality and task generalization capabilities, this application introduces an instruction fine-tuning stage. This stage involves constructing multi-turn, multimodal question-answering data from the remote sensing domain to train the model to understand complex task semantics and output high-quality responses. It's important to note that this stage primarily designs multi-turn question-answering based on the fused visual features, and its design process differs significantly from the training data design processes of other remote sensing models.

[0068] Specifically, the multimodal remote sensing large model is trained through the following steps: Question-and-answer data are generated for the feature information represented by one or more modalities of remote sensing data in different modalities. Based on the task chain reasoning process required by the multimodal remote sensing big model when processing the multimodal visual fusion features, the generated question-and-answer data are sorted to obtain multi-round question-and-answer data; The pre-trained multimodal remote sensing large model was trained using the multi-turn question-and-answer data.

[0069] In this embodiment, reference is made to Figure 6 The diagram shown illustrates the MRSChat multi-turn question-answering mechanism. This application designs a task-oriented dialogue flow based on the fused multimodal visual features to obtain multi-turn question-answering data. Subsequently, the multi-turn question-answering data is used to train (i.e., fine-tune) a pre-trained multimodal remote sensing model, enabling the model to perform dynamic question-answering, contextual reasoning, and cross-modal judgment based on information from different sensor modalities.

[0070] It should be noted that traditional large-scale remote sensing models relying on single-modality (e.g., RGB) sensors are limited to tasks supported by optical features, such as ground feature recognition, color description, and target localization, because the input remote sensing images are optical images. Multimodal visual features, on the other hand, add invisible light images such as SAR and multispectral images to the input images, enabling the model to understand both visible and invisible light images simultaneously. Therefore, multi-turn question-and-answer design should not be limited to the optical dimension. This application generates question-and-answer data based on the feature information represented by one or more modalities of remote sensing data from different modalities. For example, question-and-answer data can be generated based on the feature information represented by RGB optical images from different modalities (for the case of one modality of remote sensing data); or, question-and-answer data can be generated by combining the feature information represented by RGB optical images and SAR images from different modalities (for the case of multiple modalities of remote sensing data).

[0071] Based on this, this application sorts the generated question-and-answer data according to the task chain reasoning process required by the multimodal remote sensing big model when processing the multimodal visual fusion features, so that the obtained multi-round question-and-answer data can simulate the task chain reasoning process (e.g., first identify crops → then determine whether they are diseased → then analyze soil type), thereby improving the generalization performance and practicality of the model in complex tasks.

[0072] Understandably, compared to previous large remote sensing models that relied on a single modal sensor, this application focuses more on guiding the model to focus on key information of different modalities (such as RGB and SAR) from the fused multimodal visual features when designing multi-round question answering, so as to dynamically mine modal complementary information.

[0073] For example, the multi-round question-and-answer data can be designed using the following questioning order: 1. The first round of questions is "Is this area farmland?", to guide the model to perform farmland identification-related reasoning based on features such as texture and color information in RGB optical images; 2. The second round of follow-up questions, “Should rice be planted?”, guides the model to combine vegetation indices in the multispectral bands (such as the Normalized Difference Vegetation Index (NDVI) and other characterization information) to make inferences related to crop type determination. 3. The third round of questions, “Is there a risk of disease?”, guides the model to further combine backscattering features and moisture inversion information in SAR images to help determine abnormal crop growth.

[0074] Optionally, the multimodal remote sensing large model is pre-trained through the following steps: The multimodal remote sensing large model is pre-trained using a preset training set, and only the projection layer is adjusted during the pre-training process. The preset training set includes image-text pairs consisting of visible light image samples and text samples.

[0075] The preset training set can be constructed using publicly available image-text pair datasets in the field of remote sensing (such as RGB-text image-text pair datasets).

[0076] In this embodiment, the projection layer between the encoder in the visual layer and the large language model in the language layer is optimized so that the fused multimodal features can correctly activate the generation capability of the large language model. This is taken as the training objective. The multimodal remote sensing large model is pre-trained using a preset training set (such as an RGB-text image-text pair training set) to initialize the visual-language interface layer, so that the visual features can be effectively understood and used by the language model.

[0077] For example, with Figure 4 Taking the MRSChat model architecture shown as an example, during the pre-training process, the MRSChat model is optimized based on its performance on remote sensing intelligent tasks such as image description generation, land cover type question answering, and image semantic completion using a preset training set. During optimization, the main body of the large language model (i.e., Vicuna-7b) is frozen, and only the projection layer after the encoder (i.e., the MLP Adaptor) is fine-tuned. This ensures the model has the basic ability to "generate natural language output using remote sensing visual input as context." The data processing flow can be referenced. Figure 7 As shown.

[0078] Optionally, based on the training objective, a training loss function corresponding to the pre-training is constructed using Causal Language Modeling Loss (Causal LM Loss).

[0079] In the above embodiments, in order to enable the model to have a unified ability to understand and generate text and images, this application introduces alignment training, which includes a pre-training stage and an instruction tuning stage, to achieve efficient integration of multimodal information and language generation capabilities.

[0080] In an optional embodiment, multiple patches are input to the feature fusion module, including: The data preprocessing module extracts the patch associated with each visible light image and each invisible light image in the user-preselected image region from the multiple patches as the patch to be processed; The data preprocessing module inputs each of the patches to be processed into the feature fusion module.

[0081] For specific implementation, see Figure 8 The diagram shown illustrates patch selection. During the data preprocessing stage, the model allows users to select a region in a remote sensing image using a bounding box. This region (i.e., the user-selected image region) can be identified using the bounding box's position coordinates. Characterize it.

[0082] After obtaining the user-preselected image regions corresponding to each image in the multimodal remote sensing data, the model selects the patches in the user-preselected image region from the patch sequence (which contains all patches obtained from the division of the multimodal remote sensing data) for visual feature extraction. For example, visual feature extraction is performed on the patches in the user-preselected image regions of invisible light images such as SAR images and multispectral images to obtain the first modality visual features associated with the user-preselected image regions in invisible light images. Visual feature extraction is also performed on the patches in the user-preselected image regions of visible light images such as RGB images to obtain the second modality visual features associated with the user-preselected image regions in visible light images. Then, subsequent steps such as Cross Attentionr fusion and Large Language Model (LLM) interaction are performed on the first and second modality visual features associated with the user-preselected image regions.

[0083] In this embodiment, this application selects only the patches located within the user-specified bounding box for visual features to participate in subsequent dialogue and reasoning. This applies the Region Patch Focus technology to cross-modal alignment of visual language models based on multi-source remote sensing data, which can avoid cropping remote sensing images and achieve region perception directly at the visual level.

[0084] For example, with an image input size of The patch size is For example, there are a total of There are 1 patch, of which:

[0085] The coordinate index of each patch is represented as follows: Then the coordinates of its center point It can be represented as: For the user-defined bounding box The model selects the coordinates of all center points in the image. A patch that satisfies the following constraints (i.e., satisfies the following target conditions) is considered as a patch associated with the user-preselected image region:

[0086] Then, the mask attention method is used to add mask attention to the unselected patches so that the subsequent model can only pay attention to the user-selected region. In this way, the model only focuses attention on the patches in that region, while other regions are masked by the model, thereby achieving visual localization and region-level inference.

[0087] Based on the above embodiments, this application proposes a multimodal remote sensing large model (MrsChat) based on the LLaVA architecture. Through the above-mentioned image feature extraction, cross-modal fusion and image-text alignment, the model realizes the ability to identify and understand invisible bands (such as near-infrared NIR, short-wave infrared SWIR, synthetic aperture radar SAR, etc.) and align these features with text. In addition, for downstream tasks, this application also conducts multiple rounds of question-answering training on the model, thereby improving its application capability in real-world scenarios.

[0088] Therefore, MrsChat has the ability to deeply integrate multimodal remote sensing data (such as SAR images and RGB images) with natural language, thereby supporting multiple downstream tasks (i.e., remote sensing intelligent tasks) covering the field of remote sensing, such as multi-turn question answering, visual localization, land cover classification, and index calculation (such as calculating NDVI, Normalized Difference Water Index (NDWI)). Moreover, it does not require specific fine-tuning for a particular task, thus improving the model's adaptability.

[0089] Compared to traditional dialogue models that only support remote sensing image input, this application significantly improves the ability to understand multimodal images. Specifically, this application enhances the feature extraction and understanding capabilities for invisible light bands by introducing a cross-attention mechanism at the visual layer to fuse features from different modalities and adding a projection layer between the visual and linguistic layers for image-text alignment. Furthermore, using a large-sample, high-quality invisible light image (e.g., MS and SAR images)-text dataset for feature alignment improves the model's generalization ability. In addition, compared to models that only support remote sensing image input, MrsChat, through multi-turn question-answering training, possesses the ability to support multi-turn dialogues, thereby significantly expanding the model's interactivity and practicality.

[0090] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0091] This application also provides a multimodal remote sensing large model, such as Figure 4 As shown, The multimodal remote sensing large model includes a data preprocessing module, a feature fusion module, and a feature alignment module, wherein: The data preprocessing module is used to, in response to receiving multimodal remote sensing data, divide the multimodal remote sensing data into multiple image patches and input the multiple patches into the feature fusion module. The multimodal remote sensing data includes at least one visible light image and at least one invisible light image. The feature fusion module is used to perform position encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain a first modal visual feature associated with the invisible light image and a second modal visual feature associated with the visible light image; and to use the second modal visual feature as the core representation basis based on the cross-attention mechanism, and to supplement the core representation basis with the first modal visual feature to obtain a multimodal visual fusion feature, and input the multimodal visual fusion feature to the feature alignment module; The feature alignment module is used to perform image-text alignment processing on the input text and the multimodal visual fusion features through a pre-trained projection layer to obtain target features, and use the target features as input data for the large language model to support the large language model in completing remote sensing intelligent tasks.

[0092] Optionally, the feature fusion module is further configured to perform the following steps: Based on the trainable parameter matrix, the second modality visual features are mapped to query vectors, and the first modality visual features are mapped to key vectors and numerical vectors; The multimodal visual fusion features are determined based on the query vector, the key vector, and the numerical vector.

[0093] Optionally, the multimodal visual fusion features are determined by the following formula:

[0094] in, Represents multimodal visual fusion features. Represents the query vector. Represents the key vector. Represents a numerical vector. This represents the original feature dimension before mapping.

[0095] Optionally, the feature fusion module is further configured to perform the following steps: A non-learnable sine and cosine absolute position encoding method is used to encode the position of each patch associated with an invisible light image to obtain the first modality visual features; A learnable positional encoding method is used to encode the position of each patch associated with a visible light image to obtain the second modality visual features.

[0096] Optionally, the feature fusion module is further configured to perform the following steps: Each patch associated with each invisible light image is used as a sequence element to construct the first sequence associated with each invisible light image; The first encoder is used to positionally encode the first sequence associated with each invisible light image to obtain the first modal visual features associated with each invisible light image; The first encoder is configured to: encode the first sequence using a non-learnable sine absolute position encoding method when the preset encoding dimension is even; and encode the first sequence using a non-learnable cosine absolute position encoding method when the preset encoding dimension is odd.

[0097] Optionally, the unlearnable sinusoidal absolute position encoding method is represented as follows:

[0098] The unlearnable cosine absolute position encoding method is represented as follows:

[0099] in, , Indicates the visual features of the first modality. This represents the index of a sequence element in the first sequence. Indicates the preset encoding dimension. This represents the total embedding dimension of the first encoder.

[0100] Optionally, the feature fusion module is further configured to perform the following steps: Each patch associated with a visible light image is used as a sequence element to construct a second sequence associated with each visible light image; The second encoder is used to positionally encode the second sequence associated with each visible light image to obtain the second modal visual features associated with each visible light image; The second encoder is configured to perform position encoding using the following learnable position encoding method:

[0101] in, Indicates the visual features of the second modality. This represents the location embedding matrix constructed based on learnable parameters. Matrix elements Indicates the position of the sequence element in the second sequence. The corresponding embedding vector.

[0102] Optionally, the multimodal remote sensing large model is trained through the following steps: Question-and-answer data are generated for the feature information represented by one or more modalities of remote sensing data in different modalities. Based on the task chain reasoning process required by the multimodal remote sensing big model when processing the multimodal visual fusion features, the generated question-and-answer data are sorted to obtain multi-round question-and-answer data; The pre-trained multimodal remote sensing large model was trained using the multi-turn question-and-answer data.

[0103] Optionally, the multimodal remote sensing large model is pre-trained through the following steps: The multimodal remote sensing large model is pre-trained using a preset training set, and only the projection layer is adjusted during the pre-training process. The preset training set includes image-text pairs consisting of visible light image samples and text samples.

[0104] Optionally, the data preprocessing module is further configured to perform the following steps: From the multiple patches, extract the patches associated with each visible light image and each invisible light image in the user-preselected image region as the patches to be processed; Each of the patches to be processed is input into the feature fusion module; The patch associated with the user-preselected image region includes all patches in the image that satisfy the target conditions, which are expressed as follows:

[0105] in,( , () represents the center coordinates of the patch with coordinate index (i, j) in the image. , This indicates the position coordinates of the bounding box corresponding to the user-selected image region, and the image contains... One patch, , Indicates the size of the image. This indicates the size of the patch in the image.

[0106] As can be seen from the above technical solution, this application employs positional encoding technology to extract visual features from visible and invisible light images in multimodal remote sensing data. Then, based on a cross-attention mechanism, the second modality visual features are used as the core representation basis, and the first modality visual features are used to supplement the core representation basis. This maintains the dominant position of information in the visible light image as the main modality and effectively integrates information such as the spectral structure implied in the invisible light image into the optical semantic space, thereby achieving a unified representation of multimodal remote sensing data and providing a technical foundation for subsequent deep fusion. Furthermore, this application introduces a projection layer between the visual layer and the language layer for image-text alignment processing, that is, performing a unified mapping from visual space to language space on the input text and multimodal visual fusion features to obtain target features. This enables deep fusion of natural language and multimodal remote sensing data, allowing the model to have the ability to model remote sensing visual language for multimodal remote sensing data (visible and invisible light images), thereby effectively improving the completion effect of remote sensing intelligent tasks.

[0107] This application also provides an electronic device, which is described in reference to... Figure 9 , Figure 9 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 9 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the cross-modal alignment method for visual language models based on multi-source remote sensing data disclosed in the embodiments of this application.

[0108] This application also provides a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the cross-modal alignment method for visual language models based on multi-source remote sensing data disclosed in this application.

[0109] This application also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the cross-modal alignment method for visual language models based on multi-source remote sensing data disclosed in this application.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, systems, devices, storage media, and program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0116] The above provides a detailed description of a cross-modal alignment method for visual language models based on multi-source remote sensing data provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for cross-modal alignment of visual language models based on multi-source remote sensing data, characterized in that, The method includes: A multimodal remote sensing large model is constructed, which includes a data preprocessing module, a feature fusion module, and a feature alignment module; In response to receiving multimodal remote sensing data, the data preprocessing module divides the multimodal remote sensing data into multiple image patches and inputs the multiple patches into the feature fusion module. The multimodal remote sensing data includes at least one visible light image and at least one invisible light image. The feature fusion module performs position encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain the first modality visual features associated with the invisible light image and the second modality visual features associated with the visible light image; The feature fusion module is based on a cross-attention mechanism. It uses the second modality visual features as the core representation basis and supplements the core representation basis with the first modality visual features to obtain multimodal visual fusion features. The multimodal visual fusion features are then input into the feature alignment module. The feature alignment module performs image-text alignment processing on the input text and the multimodal visual fusion features through a pre-trained projection layer to obtain target features, and uses the target features as input data for the large language model to support the large language model in completing remote sensing intelligent tasks.

2. The method according to claim 1, characterized in that, The feature fusion module, based on a cross-attention mechanism, uses the second modality visual features as the core representation basis and supplements the core representation basis with the first modality visual features to obtain multimodal visual fusion features, including: The feature fusion module maps the second modality visual features into a query vector and the first modality visual features into a key vector and a numerical vector, based on a trainable parameter matrix. The feature fusion module determines the multimodal visual fusion feature based on the query vector, the key vector, and the numerical vector; wherein, the multimodal visual fusion feature is determined by the following formula: in, Represents multimodal visual fusion features. Represents the query vector. Represents the key vector. Represents a numerical vector. This represents the original feature dimension before mapping.

3. The method according to claim 1, characterized in that, The feature fusion module performs positional encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain the first modality visual features associated with the invisible light image and the second modality visual features associated with the visible light image, including: The feature fusion module uses an unlearnable sine and cosine absolute position encoding method to encode the position of each patch associated with each invisible light image, thereby obtaining the first modality visual features. The feature fusion module uses a learnable positional encoding method to encode the position of each patch associated with a visible light image, thereby obtaining the second modality visual features.

4. The method according to claim 3, characterized in that, The feature fusion module employs a non-learnable sine and cosine absolute position encoding method to perform position encoding on the patch associated with each invisible light image, obtaining the first modality visual features, including: The feature fusion module uses each patch associated with each invisible light image as a sequence element to construct the first sequence associated with each invisible light image; The feature fusion module uses a first encoder to perform position encoding on the first sequence associated with each invisible light image to obtain the first modal visual features associated with each invisible light image; The first encoder is configured to: encode the first sequence using a non-learnable sine absolute position encoding method when the preset encoding dimension is even; and encode the first sequence using a non-learnable cosine absolute position encoding method when the preset encoding dimension is odd; the non-learnable sine absolute position encoding method is expressed as follows: The unlearnable cosine absolute position encoding method is represented as follows: in, , Indicates the visual features of the first modality. This represents the index of a sequence element in the first sequence. Indicates the preset encoding dimension. This represents the total embedding dimension of the first encoder.

5. The method according to claim 3, characterized in that, The feature fusion module employs a learnable positional encoding method to perform positional encoding on the patch associated with each visible light image, thereby obtaining second-modality visual features, including: The feature fusion module uses each patch associated with each visible light image as a sequence element to construct a second sequence associated with each visible light image. The feature fusion module uses a second encoder to perform position encoding on the second sequence associated with each visible light image, thereby obtaining the second modal visual features associated with each visible light image; The second encoder is configured to perform position encoding using the following learnable position encoding method: in, Indicates the visual features of the second modality. This represents the location embedding matrix constructed based on learnable parameters. Matrix elements Indicates the position of the sequence element in the second sequence. The corresponding embedding vector.

6. The method according to claim 1, characterized in that, The multimodal remote sensing large model is trained through the following steps: Question-and-answer data are generated for the feature information represented by one or more modalities of remote sensing data in different modalities. Based on the task chain reasoning process required by the multimodal remote sensing big model when processing the multimodal visual fusion features, the generated question-and-answer data are sorted to obtain multi-round question-and-answer data; The pre-trained multimodal remote sensing large model was trained using the multi-turn question-and-answer data.

7. The method according to claim 6, characterized in that, The multimodal remote sensing large model is pre-trained through the following steps: The multimodal remote sensing large model is pre-trained using a preset training set, and only the projection layer is adjusted during the pre-training process. The preset training set includes image-text pairs consisting of visible light image samples and text samples.

8. The method according to any one of claims 1-7, characterized in that, Inputting multiple patches into the feature fusion module includes: The data preprocessing module extracts the patch associated with each visible light image and each invisible light image in the user-preselected image region from the multiple patches as the patch to be processed; The data preprocessing module inputs each of the patches to be processed into the feature fusion module; The patch associated with the user-preselected image region includes all patches in the image that satisfy the target conditions, which are expressed as follows: in,( , () represents the center coordinates of the patch with coordinate index (i, j) in the image. , This indicates the position coordinates of the bounding box corresponding to the user-selected image region, and the image contains... One patch, , Indicates the size of the image. This indicates the size of the patch in the image.

9. A large-scale multimodal remote sensing model, characterized in that, The multimodal remote sensing large model includes a data preprocessing module, a feature fusion module, and a feature alignment module, wherein: The data preprocessing module is used to, in response to receiving multimodal remote sensing data, divide the multimodal remote sensing data into multiple image patches and input the multiple patches into the feature fusion module. The multimodal remote sensing data includes at least one visible light image and at least one invisible light image. The feature fusion module is used to perform position encoding on the patches associated with the invisible light image and the visible light image respectively, to obtain a first modal visual feature associated with the invisible light image and a second modal visual feature associated with the visible light image; and is used to use the second modal visual feature as the core representation basis based on the cross-attention mechanism, and to supplement the core representation basis with the first modal visual feature to obtain a multimodal visual fusion feature, and input the multimodal visual fusion feature to the feature alignment module; The feature alignment module is used to perform image-text alignment processing on the input text and the multimodal visual fusion features through a pre-trained projection layer to obtain target features, and use the target features as input data for the large language model to support the large language model in completing remote sensing intelligent tasks.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the cross-modal alignment method for visual language models based on multi-source remote sensing data as described in any one of claims 1-8.

Images