Multi-scale cross-modal alignment remote sensing visual language training method and related product

Abstract

The invention discloses a multi-scale cross-modal alignment remote sensing visual language training method and a related product. The method comprises the following steps: selecting a training task and a training data set corresponding to the training task; extracting a visual feature sample by using the visual pre-training model and a visual encoder, and calculating a first loss function; extracting a language feature sample through a language pre-training model, and calculating a second loss function; the visual feature sample and the language feature sample are input into a multi-scale cross-modal alignment module comprising a visual-language interaction sub-module and a language-visual interaction sub-module which are parallel, the two sub-modules both adopt a multi-head attention mechanism to achieve interaction between language and visual embedding, and a third loss function is calculated; and comprehensively calculating a total loss function, training and updating the model weight until the total loss function converges, and outputting a trained model. The problems that when an existing visual language model processes complex tasks, modal alignment precision is insufficient, training targets are single, and the feature fusion effect is poor are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image multi-modal task, and relates to a multi-scale cross-modal alignment remote sensing visual language training method and related products. BACKGROUND

[0002] In visual language tasks such as image recognition and retrieval (RIS), visual genome (VG), and visual question answering (VQA), although their goals and application scenarios are different, they all rely on the ability to extract and fuse features from images and texts. This commonality has prompted researchers to adopt a classic solution paradigm: on the one hand, convolutional neural networks (CNNs) such as ResNet, VGG, and EfficientNet are used to extract visual features from images; on the other hand, pre-trained language models such as BERT, RoBERTa, and GPT are used to extract semantic features from texts. Subsequently, the features of the two branches are integrated through cross-modal fusion techniques to achieve deep understanding and interaction between images and texts.

[0003] However, with the continuous development of technology, existing visual language models gradually reveal some limitations when dealing with complex tasks. First, the accuracy of modality alignment is a key factor that restricts the performance of the model. Traditional alignment strategies are often based on simple feature linear mapping, which is not sufficient when dealing with multi-scale and multi-modal complex information, and it is difficult to fully capture the fine-grained information between different modalities. This leads to deficiencies in the model's understanding and processing of deep relationships between images and texts.

[0004] Secondly, existing methods usually use fixed loss functions in the pre-training stage, which limits the diversity and adaptability of the model. For example, some methods may rely only on reconstruction loss or classification loss to guide the training of the model, ignoring the need for multiple losses within and between modalities. This single training objective not only limits the generalization ability of the model, but also may cause performance bottlenecks of the model on specific tasks.

[0005] In addition, in the feature fusion stage, existing models also face challenges. Especially when dealing with high-resolution and multi-scale information images such as remote sensing images, how to effectively combine information from different modalities becomes a difficult problem. Traditional fusion methods often fail to fully utilize the complementarity between images and texts, leading to difficulties in understanding and processing complex scenarios. SUMMARY

[0006] The purpose of the present application is to solve the technical problems of poor alignment accuracy of each modality, single training target, and ineffective feature fusion in the prior art, and to provide a multi-scale cross-modal alignment remote sensing visual language training method and related products.

[0007] To achieve the above objectives, the present invention employs the following technical solution: The first aspect of this invention provides a multi-scale, cross-modal aligned remote sensing visual language training method, comprising the following steps: S1, Select the training task and its corresponding training dataset; S2, Select a visual pre-trained model and a visual encoder, extract features from the training dataset using the visual pre-trained model and the visual encoder, generate visual feature samples, and calculate the first loss function; S3, determine the language pre-training model, extract features from the training dataset using the language pre-training model, generate language feature samples, and calculate the second loss function; S4, input visual feature samples and language feature samples into the multi-scale cross-modal alignment module to calculate the third loss function; S5, calculate the total loss function, train and update the model weights until the total function converges, and output the trained model; The multi-scale cross-modal alignment module includes a parallel visual-language interaction submodule and a language-visual interaction submodule; both the visual-language interaction submodule and the language-visual interaction submodule employ a multi-head attention mechanism to realize the interaction between language and visual embeddings.

[0008] Furthermore, the first loss function is the mean squared error between the visual feature samples and the training dataset.

[0009] Furthermore, the second loss function is the MLM loss function.

[0010] Furthermore, the third loss function is the cross-entropy loss.

[0011] Furthermore, the total loss function is the sum of the first loss function, the second loss function, and the third loss function.

[0012] Furthermore, in the visual-language interaction submodule, visual embedding serves as a query and text embedding serves as a key-value pair; in the language-visual interaction submodule, text embedding serves as a query and visual embedding serves as a key-value pair.

[0013] A second aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-scale cross-modal aligned remote sensing visual language training method.

[0014] A third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-scale cross-modal aligned remote sensing visual language training method.

[0015] The fourth method of the present invention provides a computer program product, the computer program product including computer instructions, the computer instructions instructing a computer to execute the multi-scale cross-modal aligned remote sensing visual language training method.

[0016] The fifth aspect of this invention provides a multi-scale, cross-modal aligned remote sensing visual language training system, comprising: The data selection module allows you to select the training task and its corresponding training dataset. The visual feature generation module selects a visual pre-trained model and a visual encoder, extracts features from the training dataset using the visual pre-trained model and the visual encoder, generates visual feature samples, and calculates the first loss function. The language feature generation module determines the language pre-training model, extracts features from the training dataset using the language pre-training model, generates language feature samples, and calculates the second loss function. The visual-language interaction module inputs visual feature samples and language feature samples into the multi-scale cross-modal alignment module to calculate the third loss function; The training module calculates the total loss function, trains and updates the model weights until the total function converges, and outputs the trained model. The multi-scale cross-modal alignment module includes a parallel visual-language interaction submodule and a language-visual interaction submodule; both the visual-language interaction submodule and the language-visual interaction submodule employ a multi-head attention mechanism to realize the interaction between language and visual embeddings.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention discloses a multi-scale cross-modal alignment remote sensing visual-language training method. By introducing a multi-scale cross-modal alignment module, comprising parallel visual-language interaction sub-modules and language-visual interaction sub-modules, and utilizing a multi-head attention mechanism to achieve fine-grained interaction between language and visual embeddings, the alignment accuracy between modalities is significantly improved. This method can effectively capture fine-grained information between different modalities, enhancing the model's understanding and processing capabilities of deep relationships between images and text. For images with high resolution and multi-scale information, such as remote sensing images, this invention achieves effective fusion of visual and linguistic features through the multi-scale cross-modal alignment module. This fusion method fully utilizes the complementarity between images and text, improving the model's accuracy and robustness in understanding and processing complex scenes. The method of this invention supports seamless transfer from pre-training to downstream tasks. By pre-training separately in the visual and linguistic branches, and then combining the feature fusion of the multi-scale cross-modal alignment module, the model can easily adapt to different downstream task requirements. Only simple fine-tuning and optimization are needed to achieve good performance for different training objectives, greatly reducing the threshold for model deployment and application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the multi-scale cross-modal alignment remote sensing visual language training method of the present invention; Figure 2 This is a structural diagram of the multi-scale cross-modal alignment module of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and marked in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 This invention discloses a multi-scale, cross-modal alignment remote sensing visual language training method, comprising the following steps: Step S1, Dataset Preparation and Preprocessing The RIS task uses the RefSegRS or RRSIS-D dataset; the VG task uses the DIOR-RSVG or Optical-OPT-RSVG dataset; and the VQA task uses the GQA or CLEVR dataset.

[0024] After determining the appropriate datasets and tasks, the datasets for different tasks undergo uniform formatting. Specifically, uniform formatting involves data augmentation, primarily using the following methods: Crop: Cropping images to a specified size. If the image is small, padding is applied to achieve the required size; Horizontal Flip: Randomly flipping the image and mask horizontally to increase data diversity; Vertical Flip: Randomly flipping the image and mask vertically; Normalize: Normalizing the image to the range of 0 to 1 and performing standardization; Resize: Adjusting the size of the image and mask to fit the model's input requirements. For language data, standardization and text cleaning are performed to ensure consistent input format.

[0025] Step S2, pre-training of visual and language branches; Step S21, visual branch pre-training; determine the visual pre-training model and visual encoder, extract features from the training dataset using the visual pre-training model and visual encoder, generate visual feature samples, and calculate the first loss function. The MAE method is employed, a pre-training method for unsupervised learning, particularly in vision tasks, especially in image reconstruction and representation learning. It is primarily used to address image encoding problems in self-supervised learning. Its core idea is to randomly mask a portion of the image and train the model to recover these masked regions, thereby learning an effective representation of the image.

[0026] The steps of this visual branch are as follows: Based on step S1, a dataset is selected, a certain proportion of the image is randomly masked, and the masked image is input into the encoder to extract features. This encoder supports more than 50 encoder options in the framework, including Vit, Swin, Vmamba, and Mit. Then, the decoder is used to reconstruct the masked region. The reconstruction error of the masked region is calculated.

[0027] The training objective of the decoder is to minimize the difference between the reconstructed image and the original image in the masked region. Commonly used loss functions include mean squared error (MSE) loss, the formula for which is shown below:

[0028] in, It is the number of samples. It is the actual value of the i-th sample. This is the predicted value of the i-th sample. The loss function of MAE primarily focuses on the reconstruction error. The loss function calculates the degree of error the model makes in recovering the masked region. During training, this mean squared error is used as the loss function Loss1.

[0029] Step S22, Language Branch Pre-training: The language branch uses a BERT model pre-trained, which is a pre-trained deep learning model for natural language processing (NLP) tasks. It learns general language representations by pre-training on large-scale corpora and can then be fine-tuned for specific tasks.

[0030] Based on step S1, a dataset is selected and input into the BERT model to obtain language features. BERT uses the MLM loss function, which masks some tokens in a sentence. The model is trained to predict these masked tokens, and for each masked token, the model's predicted token distribution is compared with the actual masked tokens. The formula for the loss function is:

[0031] in, It is the actual token that is being concealed. It masks the context of the token. This is the probability distribution predicted by the model. The MLM loss function is used as Loss2.

[0032] Step S3, Multi-scale cross-modal alignment module Based on step S21, visual features are extracted from the visual branch to obtain visual features. Based on step S22, linguistic features are extracted from the language branch to obtain linguistic features. The visual and linguistic features are then input into the multi-scale cross-modal alignment module. The multi-scale visual and text features are aligned using the multi-scale cross-modal alignment module. Then, the sentence features are used as a language query to retrieve and aggregate reference object information from the multi-scale visual features in the multi-modal Transformer. The Transformer generates object embeddings for the final object coordinate prediction.

[0033] Specifically, the multi-scale cross-modal alignment module in this invention comprises two sub-modules: a language-visual interaction sub-module and a visual-language interaction sub-module. The language-visual interaction sub-module refines text features by progressively executing cross-modal interactions between text features and visual features at different levels, constructing the correlation between visual and text features. The visual-language interaction sub-module has a similar structure to the other sub-module but executes a different interaction process. The visual-language interaction sub-module allows the original multi-scale visual features and text embeddings to interact through a multi-head attention layer and a multiplication layer, respectively. Both the language-visual interaction and the visual-language interaction sub-module use a multi-head attention mechanism to achieve interaction between text and visual embeddings. In language-visual interaction, the text embedding acts as the query (Q), and the visual features act as key-value pairs (K, V); while in visual-language interaction, the visual embedding acts as the query, and the text embedding acts as the key-value pair. A similarity matrix is ​​obtained by calculating the similarity between the text embedding and the visual pixel embedding. An attention weight matrix is ​​calculated between the text embedding (query) and the visual pixel embedding (key). Technical terms have corresponding formulas. The similarity matrix is ​​used to measure the correlation between text embeddings and visual pixel embeddings, and serves as a weighting factor for feature aggregation. In cross-attention mechanisms: Text embedding as query Q; Visual features serve as keys K and values ​​V; The matrix obtained by calculating Q×KTQ × K^T (usually after softmax normalization) is the "similarity matrix".

[0034] Similarity matrices are used to aggregate language- or visually relevant object information, which is then multiplied with the input features to obtain refined text or visual features. To obtain positional information, a fixed two-dimensional sinusoidal positional code is added to the visual features before cross-attention processing, and a fixed one-dimensional sinusoidal positional code is added to the text features. In the language-visual interaction submodule, after multiple iterations of interaction, the final refined text features are transformed into sentence features through a projection layer and used as language query input to the multimodal DETR. The cross-entropy loss is calculated by comparing the changes in feature similarity before and after the interaction. This cross-entropy loss is denoted as Loss3.

[0035] The total loss function is the sum of the visual branch loss Loss1 from step S21, the language branch loss Loss2 from step S22, and the feature alignment loss Loss3 from step S3, to obtain the overall pre-training loss.

[0036] Step S4, Model Training The framework is trained using the AdamW optimizer with a Poly learning rate decay strategy. The initial learning rate for the visual branch is 6e-5, and for the language branch and alignment module, it is 6e-4. Training is performed for 50 epochs. Model performance is evaluated using a validation dataset, and the optimal model weights based on mIoU are saved.

[0037] Step S5, model transfer to downstream tasks The trained model is then transferred to downstream tasks such as RIS, VG, and VQA. The model is fine-tuned and optimized according to the requirements of the downstream tasks. Predictions are made on new data to obtain visual and linguistic feature outputs. Multi-scale feature fusion and alignment are performed using the pre-trained framework from training to generate the final prediction results.

[0038] Step S6, Result Evaluation oIoU and mIoU are used for result evaluation. mIoU is a commonly used evaluation metric for semantic segmentation tasks. It evaluates the performance of the model by calculating the ratio of the intersection to the union between the predicted results and the true labels.

[0039] This invention provides a multi-scale cross-modal alignment remote sensing visual-language training method. By introducing a multi-scale cross-modal alignment module, this invention achieves semantic alignment between visual and linguistic features. This multi-scale feature interaction not only strengthens the correlation between visual and linguistic features but also enhances the complementarity between different modalities, enabling the model to better handle complex remote sensing image scenes. Through a joint optimization strategy using multiple loss functions of the visual branch, linguistic branch, and feature alignment module, the model's generalization ability is significantly improved. This pre-training method can better capture the potential correlation information in multi-modal data, allowing the model to exhibit superior performance when transferred to specific downstream tasks (such as RIS, VG, VQA). By pre-training separately on the visual and linguistic branches and then combining feature fusion with the multi-scale cross-modal alignment module, this invention can effectively handle the data requirements of different tasks, achieving seamless transfer from pre-training to downstream tasks, and providing a more efficient solution for multi-task processing in remote sensing image analysis.

[0040] One embodiment of the present invention provides a multi-scale cross-modal alignment remote sensing visual language training system, comprising: The data selection module allows you to select the training task and its corresponding training dataset. The visual feature generation module selects a visual pre-trained model and a visual encoder, extracts features from the training dataset using the visual pre-trained model and the visual encoder, generates visual feature samples, and calculates the first loss function. The language feature generation module determines the language pre-training model, extracts features from the training dataset using the language pre-training model, generates language feature samples, and calculates the second loss function. The visual-language interaction module inputs visual feature samples and language feature samples into the multi-scale cross-modal alignment module to calculate the third loss function; The training module calculates the total loss function, trains and updates the model weights until the total function converges, and outputs the trained model. The multi-scale cross-modal alignment module includes a parallel visual-language interaction submodule and a language-visual interaction submodule; both the visual-language interaction submodule and the language-visual interaction submodule employ a multi-head attention mechanism to realize the interaction between language and visual embeddings.

[0041] One embodiment of the present invention provides a computer program product, the computer program product including computer instructions that instruct a computer to execute the multi-scale cross-modal aligned remote sensing visual language training method.

[0042] In one embodiment of the present invention, an electronic device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a multi-scale cross-modal alignment remote sensing visual language training method.

[0043] In one embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that more specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0044] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0045] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0046] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the multi-scale cross-modal aligned remote sensing visual language training method in the above embodiments.

[0047] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-scale cross-modal alignment remote sensing visual language training method based on multi-scale cross-modal alignment, comprising the following steps: The multi-scale cross-modal alignment remote sensing visual language model comprises a visual pre-training model, a visual encoder and a multi-scale cross-modal alignment module; characterized in that, S1, selecting a training task and its corresponding training data set; S2, selecting a visual pre-training model and a visual encoder, extracting features from the training data set through the visual pre-training model and the visual encoder to generate visual feature samples, and calculating a first loss function; S3, determining a language pre-training model, extracting features from the training data set through the language pre-training model to generate language feature samples, and calculating a second loss function; S4, inputting the visual feature samples and the language feature samples into a multi-scale cross-modal alignment module to calculate a third loss function; S5, calculating a total loss function, training and updating model weights until the total function converges, and outputting a trained model; Wherein, the multi-scale cross-modal alignment module includes visual-linguistic interaction sub-modules and linguistic-visual interaction sub-modules arranged side by side; the visual-linguistic interaction sub-modules and the linguistic-visual interaction sub-modules both use a multi-head attention mechanism to realize the interaction between language and visual embedding. The first loss function is the mean square error of the visual feature samples and the training data set.

2. The multi-scale cross-modal alignment remote sensing visual language training method according to claim 1, characterized in that, The second loss function is an MLM loss function.

3. The multi-scale cross-modal alignment remote sensing visual language training method according to claim 1, wherein, The third loss function is a cross-entropy loss.

4. The multi-scale cross-modal alignment remote sensing visual language training method according to claim 1, characterized in that, The total loss function is the sum of the first loss function, the second loss function and the third loss function.

5. The multi-scale cross-modal alignment remote sensing visual language training method according to claim 1, characterized in that, In the visual-linguistic interaction sub-module, the visual embedding is used as the query and the text embedding is used as the key-value pair; in the linguistic-visual interaction sub-module, the text embedding is used as the query and the visual embedding is used as the key-value pair.

6. The multi-scale cross-modal alignment remote sensing visual language training method according to claim 1, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the multi-scale cross-modal alignment remote sensing visual language training method of any one of claims 1-6.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the multi-scale cross-modal alignment remote sensing visual language training method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer instructions instruct the computer to execute the multi-scale cross-modal alignment remote sensing visual language training method of any one of claims 1-6.

9. A computer program product comprising computer instructions, characterized in that, Comprising:

10. A multi-scale cross-modal alignment remote sensing visual language training system, characterized in that, a data selection module for selecting a training task and its corresponding training data set; a visual feature generation module for selecting a visual pre-training model and a visual encoder, extracting features from the training data set through the visual pre-training model and the visual encoder to generate visual feature samples, and calculating a first loss function; a language feature generation module for determining a language pre-training model, extracting features from the training data set through the language pre-training model to generate language feature samples, and calculating a second loss function; a visual-linguistic interaction module for inputting the visual feature samples and the language feature samples into a multi-scale cross-modal alignment module to calculate a third loss function; a training module for calculating a total loss function, training and updating model weights until the total function converges, and outputting a trained model; ​ The multiscale cross-modal alignment module comprises side-by-side visual-linguistic interaction submodules and linguistic-visual interaction submodules; the visual-linguistic interaction submodules and the linguistic-visual interaction submodules both adopt a multi-head attention mechanism to realize interaction between linguistic and visual embeddings.

Images