Remote sensing visual question answering method based on large language model and multi-level attention mechanism

Abstract

This invention belongs to the interdisciplinary field of remote sensing visual question answering and artificial intelligence, specifically involving a remote sensing visual question answering method based on a large language model and a multi-level attention mechanism. It includes: Step 1: Constructing a semantic segmentation network based on MogaNet; Step 2: Text encoding using a large language model; Step 3: Constructing a hybrid guided attention module; Step 4: Constructing a bidirectional gated cross-attention module; Step 5: Building and co-training a remote sensing visual question answering network. The MLaVQA framework proposed in this invention can achieve efficient and accurate visual question answering processing of remote sensing images. It can quickly answer complex semantic questions when processing large-scale multimodal remote sensing data, enabling refined monitoring and intelligent management of Earth's resources, significantly improving data utilization efficiency and decision support capabilities, and providing technical support for natural resource management, ecological protection, and sustainable development.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of remote sensing visual question answering and artificial intelligence, specifically involving a remote sensing visual question answering method based on a large language model and a multi-level attention mechanism. Background Technology

[0002] The field of Remote Sensing Visual Question Answering (RSVQA) primarily involves extracting meaningful visual features from remote sensing images and combining them with natural language questions to generate semantically reasonable answers. Current technologies mainly extract visual features through visual feature extraction networks, extract natural language features through language coding models, and achieve joint image-text modeling through multimodal fusion modules. However, existing technologies still face significant challenges in remote sensing visual question answering tasks, including insufficient semantic understanding, weak spatial relationship reasoning capabilities, and low efficiency in multimodal information fusion. These limitations prevent them from fully meeting the practical needs of high-precision semantic reasoning and question answering in remote sensing scenarios. Specific problems are as follows:

[0003] 1. In terms of visual feature extraction, existing methods mostly utilize Convolutional Neural Networks (CNNs) and Transformer architectures. CNN-based visual feature extraction architectures such as ResNet and EfficientNet can efficiently extract visual information from remote sensing images. However, due to the fixed receptive field of the convolutional kernel, they struggle to capture global contextual information across regions and scales in remote sensing images, resulting in performance bottlenecks for identifying small targets and complex ground structures. To compensate for this deficiency, Transformer-based visual feature extraction models such as DeiT and PVT enhance global dependency modeling capabilities through self-attention mechanisms. However, these methods generally suffer from high computational resource consumption and insufficient multi-scale modeling, limiting their effectiveness in multi-resolution processing of remote sensing images.

[0004] 2. In language modeling, early methods primarily used RNNs and LSTMs. While these could capture some sequence dependencies, their ability to understand the semantics of long texts was limited, and their computational efficiency was low. With the development of pre-trained language models, large language models based on the Transformer architecture, such as BERT, have demonstrated stronger semantic understanding and reasoning capabilities, and are gradually being applied in multimodal tasks. However, in remote sensing question-answering scenarios, existing methods often fail to adequately model text features and lack semantically guided text-image collaboration mechanisms, thus limiting the model's ability to understand complex spatial semantic problems.

[0005] 3. Existing remote sensing visual question answering models such as SOBA and EarthVQANet mostly improve the accuracy of image-text alignment and reasoning ability through direct projection or simple attention mechanisms. Although they have achieved certain results, they still lack the ability to deeply express the features of small targets and fine-grained ground objects in remote sensing images. They are difficult to model the complex spatial and semantic relationships between geographic objects, and lack dynamic guidance mechanisms in cross-modal fusion strategies, resulting in redundancy and bias in the interaction of multi-source information. Summary of the Invention

[0006] The purpose of this invention is to provide a remote sensing visual question answering method based on a large language model and a multi-level attention mechanism, which aims to solve the problems of insufficient semantic understanding, weak spatial relationship reasoning ability, and low efficiency of multimodal information fusion in current intelligent analysis of remote sensing images.

[0007] A remote sensing visual question answering method based on a large language model and a multi-level attention mechanism includes the following steps:

[0008] Step 1: Construct a semantic segmentation network based on MogaNet

[0009] To address the challenge of fixed receptive fields in convolutional neural networks (CNNs) failing to adapt to significant scale differences in land cover types within complex remote sensing scenes, MogaNet is employed as the visual encoder, and FPN as the decoder. Given an input remote sensing image... The MogaNet encoder performs a four-stage progressive feature extraction to achieve multi-scale representation learning. At each stage, an embedding stem with a stride of 4, along with spatial and channel aggregation modules, is applied to achieve a 4x downsampling. This embedding stem can be represented by the following formula.

[0010]

[0011] Here, Embed represents the embedded backbone, i represents the i-th consecutive 3 × 3 convolution, and BN represents batch normalization, which can accelerate convergence and mitigate gradient vanishing. The spatial and channel aggregation module consists of a multi-level gated aggregation part and a channel aggregation part. The former extracts multi-scale spatial features through global average pooling and residual connections, while the latter adaptively adjusts the distribution of channel information through normalization and convolution operations.

[0012] After four stages, image features that retain rich spatial details are finally obtained. This is then used as a visual cue. During the decoding phase, the model employs an FPN structure for cross-layer feature fusion. Features from different levels are integrated by first upsampling the deeper feature maps to match the spatial resolution of the shallower feature maps, and then performing element-wise concatenation to obtain the output features of the FPN. , The semantic segmentation results are generated by processing the data through the argmax operation to obtain category predictions. These results are then converted into one-hot encoded masks and resized to match visual cues. The shape of the one-hot mask is then determined. Finally, the processed one-hot mask is combined with visual cues. By concatenating along the channel dimension, the guiding feature is obtained. ).

[0013] Step 2: Text encoding using a large language model

[0014] The input question text is encoded using the state-of-the-art large language model LLaMA 3, pre-trained on a large multilingual corpus with a Transformer architecture, for a given dimension of L×d. m The input (where L represents the number of tokens in the problem, d) m (This refers to the embedding dimension). LLaMA 3 first transforms discrete text into a continuous representation through an embedding layer. Then, it applies a multi-layer Transformer to model the context of the token sequence, as shown in the following formula:

[0015]

[0016] Here, Q represents the input question token sequence. During the encoding process, the model dynamically learns the semantic relationships between the tokens and generates context-rich representations. LLaMA3 employs an enhanced self-attention mechanism to capture fine-grained linguistic structure and long-range dependencies. This enables the model to better understand the semantics of complex problems and align them more accurately with visual information, thereby improving the overall performance of the Remote Sensing Visual Problem Classification (RSVQA) task.

[0017] Step 3: Construct a hybrid guided attention module

[0018] The hybrid guided attention module achieves efficient fusion of semantic and visual features through multi-stage collaborative optimization. First, it uses guided features... As input features, the original input is projected into a high-dimensional feature space through preconvolutional embedding. The preconvolutional embedding mainly consists of 3 × 3 convolutions, batch normalization, and ReLU activation. After embedding, the input features are fed into two parallel branches for feature optimization: a dynamic feature selection branch and a feature enhancement branch.

[0019] In the dynamic feature selection branch, input features are processed by max pooling and average pooling respectively, and then fed into two fully connected (FC) layers for backprojection to further refine the features. These two paths are then aggregated to capture their contextual information. The aggregated features are then passed through a sigmoid activation function to generate channel attention weights, thereby achieving adaptive learning of channel importance. The attention weights are calculated using the following formula:

[0020]

[0021] in , is the dimension reduction and projection restoration matrix, w is the attention weight, and r is the compression ratio. Represents the ReLU activation function. This represents the Sigmoid function. The attention weights are then multiplied by the feature channels to achieve dynamic feature selection.

[0022] In the feature enhancement branch, global average pooling is used to capture the statistical distribution characteristics of the features. The pooled features are then passed through two fully connected layers to model the correlation between channels and perform dimensionality reduction. Next, the ReLU activation function and another fully connected layer are used to restore the features to their original dimensions. Finally, a normalization-based weight adjustment operation is used to scale the features, applying the learned weights to each channel to achieve feature enhancement. The data flow processing is illustrated in the formula.

[0023]

[0024] in, Represents the ReLU activation function. This indicates a scaling operation. This represents the enhanced output feature map. Finally, the outputs from the two branches are concatenated and fed into a multi-head self-attention mechanism to perform spatial-channel co-modeling, further enhancing the fusion capability of multi-granular semantic cues. Subsequently, the spatial dimension of the output feature map is unfolded into a length of... Several tokens are passed through a fully connected layer, and the channel dimension of each position is projected as... Thus, the final representation is obtained. .

[0025] Step 4: Construct a bidirectional gated cross-attention module

[0026] The bidirectional gated cross-attention module employs a multi-stage cascaded structure to achieve dynamic cross-modal interaction. The visual features X, enhanced by the hybrid guided attention module, are... f and text features Y fSemantic associations within a modality are modeled using layer normalization and multi-head self-attention mechanisms, respectively. The features are then passed through another Layer Normalization layer, and a bidirectional information transfer channel is constructed to achieve cross-modal feature alignment: in visual... Text direction refers to the self-attention features of the text. As a key / value pair, visual self-attention features As a query, it generates cross-modal feature representations. In the text Visual direction, will generate As a key / value pair, and Continuing as a query, it can be specifically expressed as a formula:

[0027]

[0028]

[0029]

[0030] Here, Self Attn represents multi-head self-attention, and Cross Attt represents bidirectional cross-attention. This bidirectional interaction ensures comprehensive semantic alignment between the two modalities. Furthermore, to further refine cross-modal fusion, the model introduces an independent multi-level gated aggregation module for each interaction direction. Original intra-modal features and cross-participation features are fed along the channel dimension through connections to the multi-level gated aggregation module, which uses Global Average Pooling (GAP) to generate adaptive gate weights. Specifically, the input features are first processed by GAP, then by a learnable fully connected (FC) layer. A SiLU activation function is then applied to introduce a non-linear transformation. Finally, the output features are obtained through gated residual fusion, which can be expressed by the formula:

[0031]

[0032]

[0033]

[0034] in, and Let represent the gated input features, g represent the gate weights learned through the multi-level gated aggregation module, W be the weight matrix of the fully connected layer, LN denotes layer normalization, and ⊙ denotes element-wise multiplication. After completing the gated fusion step, the model applies a bidirectional cross-attention mechanism again to further refine the alignment and interaction between visual and textual features, resulting in the final prediction result z.

[0035] Step 5: Build and collaboratively train a remote sensing visual question-answering network

[0036] The MLaVQA framework, a remote sensing visual question answering network based on a large language model and a multi-level attention mechanism, adopts a two-stage collaborative training paradigm. The model uses MogaNet as the visual encoder and LLaMA3 as the language encoder, achieving feature alignment and fusion through a hybrid guided attention module and a bidirectional gated cross-attention module. The entire training process includes two stages: a semantic segmentation network training stage and a hybrid attention training stage, aiming to fully utilize multimodal data features and improve the model's understanding and reasoning capabilities. In the semantic segmentation network training stage, the model trains the semantic segmentation network based on MogaNet, performing end-to-end training to generate land cover classification results. The model simultaneously generates pseudo-masks and visual cue vectors with semantic guidance capabilities. These outputs provide fine-grained geospatial cues for subsequent remote sensing visual question answering (VQA) tasks. In the hybrid attention training stage, the focus is on training the hybrid attention mechanism for visual question answering, which consists of two components: a hybrid guided attention module and a bidirectional gated cross-attention module. The hybrid guided attention module enhances the ability to extract visual features through a dual-branch attention structure, while the bidirectional gated cross-attention module adopts a hierarchical attention design to achieve efficient alignment and deep fusion of visual and textual features.

[0037] MLaVQA is a novel composite remote sensing visual question answering method that combines a large language model with a multi-level attention mechanism. It uses MogaNet as a visual encoder to achieve four-level progressive feature extraction, taking into account both local details and global contextual features. It achieves deep semantic understanding of natural language through the LLaMA3 large language model. The hybrid guided attention module and bidirectional gated cross-attention module of this invention realize bidirectional feature alignment between visual and textual aspects, which significantly improves the model's performance in reasoning about complex spatial and semantic relationships.

[0038] The technical effects of this invention are as follows:

[0039] (1) The hybrid guided attention module adopts a dual-branch structure to achieve collaborative modeling of local details and overall semantic distribution of remote sensing images, thereby guiding and refining visual features and avoiding information loss caused by relying solely on single-dimensional attention.

[0040] (2) The bidirectional cross-attention module introduces a gating unit into the ordinary cross-attention mechanism to adaptively adjust the feature contribution from different modalities, enhance the fusion of multimodal features and cross-modal interaction, thereby improving the model’s ability to understand complex remote sensing scenes.

[0041] (3) MLaVQA uses MogaNet as the visual encoder and LLaMA3 large language model as the composition language encoder to fully extract cross-modal features. It also introduces a hybrid guided attention module and a bidirectional gated cross attention module to strengthen the alignment and interaction of cross-modal information, improve the accuracy of visual question answering, and increase the robustness of the model.

[0042] (4) The MLaVQA framework proposed in this invention can achieve efficient and accurate visual question answering processing of remote sensing images. It can quickly complete the answering of complex semantic questions when processing large-scale multimodal remote sensing data, realize refined monitoring and intelligent management of Earth resources, significantly improve data utilization efficiency and decision support capabilities, and provide technical support for natural resource management, ecological protection and sustainable development. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the overall structure of the hybrid guided attention module of the present invention;

[0044] Figure 2 This is a schematic diagram of the overall structure of the bidirectional gated cross-attention module of the present invention;

[0045] Figure 3 This is a schematic diagram of the overall structure of the remote sensing visual question-answering network of the present invention;

[0046] Figure 4 This is a visualization of the comparative experimental results of the present invention;

[0047] Figure 5 A heatmap visualization of the attention mechanism of this invention. Detailed Implementation

[0048] A remote sensing visual question answering method based on a large language model and a multi-level attention mechanism includes the following steps:

[0049] Step 1: Construct a semantic segmentation network based on MogaNet:

[0050] The overall structure of remote sensing visual question answering networks is as follows: Figure 3 As shown, the overall structure of the semantic segmentation network based on MogaNet is as follows: Figure 3 As shown in (a).

[0051] Step 2: Text encoding using a large language model:

[0052] The overall structure of the Large Language Model (LLaMA 3) is as follows: Figure 3 As shown in (2).

[0053] Step 3: Construct a hybrid guided attention module:

[0054] The overall structure of the hybrid guided attention module is as follows: Figure 1 As shown.

[0055] Step 4: Construct a bidirectional gated cross-attention module:

[0056] The overall structure of the bidirectional gated cross-attention module is as follows: Figure 2 As shown.

[0057] Step 5: Build and collaboratively train the remote sensing visual question-answering network:

[0058] The overall structure of the remote sensing visual question answering network MLaVQA is as follows: Figure 3 As shown.

[0059] Experimental method in this embodiment: The model is trained on the EarthVQA multimodal dataset to verify the effectiveness of the method.

[0060] Evaluation Metrics: During the semantic segmentation network training phase, mIoU is used to evaluate the performance of the segmentation task. mIoU calculates the average ratio of the intersection to the union of the predicted and ground truth regions for all categories, comprehensively reflecting the overall performance of the model. During the hybrid attention training phase, commonly used metrics such as accuracy and root mean square error (RMSE) are used to evaluate the performance of various types of VQA tasks, and OA (overall accuracy) and OR (overall RMSE) are used to further evaluate the overall performance of the model on the entire dataset. The calculation formulas are shown below.

[0061]

[0062]

[0063]

[0064] Where C represents the total number of categories used in semantic segmentation. TP represents the number of true positive samples (correctly predicted as positive), TN represents the number of true negative samples (correctly predicted as negative), FP represents the number of false positive samples (incorrectly predicted as positive), and FN represents the number of false negative samples (incorrectly predicted as negative). Let represent the true value of the i-th sample. Let N represent the predicted value of the i-th sample, where N is the total number of samples.

[0065] Experimental Results: Three classic models that have demonstrated strong performance in remote sensing visual question answering tasks in recent years—RSVQA, RSIVQA, and SOBA—were selected for comparison. The results are shown in Table 1 and... Figure 4 As shown.

[0066] Table 1 Comparative Experiments

[0067]

[0068] The results show that the proposed MLaVQA model performs excellently on the EarthVQA dataset, achieving an overall accuracy (OA) of 79.02%, outperforming the other three models. For all six types of visual question answering tasks, MLaVQA demonstrates superior performance compared to other models. The highest accuracy (91.42%) is achieved on the basic judgment task, indicating stable and accurate performance in handling simple discrimination problems. The lowest accuracy (48.56%) is achieved on the understanding and analysis tasks, but this still outperforms other models, demonstrating that MLaVQA is more effective than existing methods in complex reasoning tasks. Furthermore, MLaVQA's RMSE (0.7644) is significantly lower than other models, indicating smaller prediction errors. This suggests that the model is more stable and robust in handling complex remote sensing VQA tasks, exhibiting stronger generalization ability.

Claims

1. A remote sensing visual question answering method based on a large language model and a multi-level attention mechanism, characterized in that, Includes the following steps: Step 1: Construct a semantic segmentation network based on MogaNet MogaNet was used as the visual encoder and FPN as the decoder; a four-level progressive feature extraction was performed; finally, image features 𝐹 that retain rich spatial details were obtained. 𝑣 And use it as a visual cue; During the decoding phase, semantic segmentation results are generated, which are then converted into one-hot encoded masks and resized to match the visual cue 𝐹. 𝑣 The shape; the processed one-hot mask and visual cue 𝐹 𝑣 The guiding features are obtained by concatenating them along the channel dimension. ; Step 2: Text encoding using a large language model The input question text is encoded using the large language model LLaMA 3 with a Transformer architecture. First, discrete text is converted into a continuous representation through an embedding layer; then, a multi-layer Transformer is applied to model the context of the token sequence. Step 3: Construct a hybrid guided attention module The hybrid guided attention module achieves efficient fusion of semantic and visual features through multi-stage collaborative optimization; First, use guiding features As input features, the original input is projected into a high-dimensional feature space through pre-convolutional embedding; After embedding, the input features are fed into two parallel branches for feature optimization: a dynamic feature selection branch and a feature enhancement branch. In the dynamic feature selection branch, the input features are processed by max pooling and average pooling respectively, and then fed into two fully connected FC layers for back projection to further refine the features; these two paths are then aggregated to capture their contextual information; then, the aggregated features are used to generate channel attention weights through a sigmoid activation function, thereby achieving adaptive learning of channel importance; Attention weights are then multiplied by feature channels to enable dynamic feature selection; In the feature enhancement branch, global average pooling is used to capture the statistical distribution characteristics of the features; the pooled features are then passed through two fully connected layers to model the correlation between channels and perform dimensionality reduction; next, the ReLU activation function and another fully connected layer are used to restore the features to their original dimensions; finally, the features are scaled through a normalization-based weight adjustment operation, and the learned weights are applied to each channel to achieve feature enhancement. Finally, the outputs from the two branches are concatenated and fed into a multi-head self-attention mechanism to perform spatial-channel co-modeling, further enhancing the fusion capability of multi-granular semantic cues. Subsequently, the spatial dimension of the output feature map is unfolded into several tokens of length 𝑃 = 𝐻 / 32 × 𝑊 / 32, and the channel dimension of each position is projected onto a fully connected layer. Thus, the final representation X is obtained. f ∈ℝ P×dm ; Step 4: Construct a bidirectional gated cross-attention module The bidirectional gated cross-attention module employs a multi-stage cascaded structure to achieve dynamic cross-modal interaction. The visual features X, enhanced by the hybrid guided attention module, are... f and text features Y f Semantic associations within a modality are modeled using layer normalization and multi-head self-attention mechanisms, respectively. The features are then passed through another layer of Layer Norm, and a bidirectional information transmission channel is constructed to achieve cross-modal feature alignment. The model introduces an independent multi-level gated aggregation module for each interaction direction; the original intramodal features and cross-participation features are fed into the multi-level gated aggregation module along the channel dimension, and the multi-level gated aggregation module uses global average pooling (GAP) to generate adaptive gate weights. Step 5: Build and collaboratively train a remote sensing visual question-answering network The remote sensing visual question answering network MLaVQA framework based on a large language model and a multi-level attention mechanism adopts a two-stage collaborative training paradigm. The model uses MogaNet as the visual encoder and LLaMA3 as the language encoder, and achieves feature alignment and fusion through a hybrid guided attention module and a bidirectional gated cross-attention module.

2. The remote sensing visual question answering method based on a large language model and a multi-level attention mechanism according to claim 1, characterized in that, Step one specifically yields the following: MogaNet is used as the visual encoder, and FPN is used as the decoder; given the input remote sensing image F v ∈ℝ H×W×C The MogaNet encoder performs a four-stage progressive feature extraction. At each stage, an embedding structure with a stride of 4 and spatial and channel aggregation modules are applied to achieve a 4x downsampling. The embedding structure is represented by the following formula: ， Where Embed represents the embedded backbone, i represents the i-th consecutive 3 × 3 convolution, and BN represents batch normalization, which can accelerate convergence and alleviate gradient vanishing; the spatial and channel aggregation module consists of a multi-level gated aggregation part and a channel aggregation part; the former extracts multi-scale spatial features through global average pooling and residual connections, while the latter adaptively adjusts the distribution of channel information through normalization and convolution operations; After four stages, the final image feature 𝐹 that retains rich spatial details is obtained. 𝑣 And use it as a visual cue; During the decoding phase, the model employs an FPN structure for cross-layer feature fusion. Features from different levels are integrated by first upsampling the deeper feature maps to match the spatial resolution of the shallower feature maps, and then performing element-wise concatenation to obtain the FPN's output feature 𝐹. FPN , FPN The argmax operation is used to obtain category predictions, thereby generating semantic segmentation results; These results are then converted into one-hot encoded masks and resized to match visual cues (𝐹). 𝑣 The shape of ); Finally, the processed one-hot mask is combined with the visual cue 𝐹 𝑣 By concatenating along the channel dimension, the guiding feature is obtained. ).

3. The remote sensing visual question answering method based on a large language model and a multi-level attention mechanism according to claim 1, characterized in that, The specific method for step two is as follows: The input question text is encoded using the Large Language Model LLaMA 3 with a Transformer architecture, for a given dimension of L×d. m The input is given by , where L represents the number of tokens in the problem, and d m In terms of embedding dimension, LLaMA 3 first transforms discrete text into a continuous representation through an embedding layer; then, it applies a multi-layer Transformer to model the context of the token sequence, as shown in the following formula: ， Where Q represents the input question token sequence; during the encoding process, the model dynamically learns the semantic relationships between the tokens and generates context-rich representations. ∈ℝ Lxdm .

4. The remote sensing visual question answering method based on a large language model and a multi-level attention mechanism according to claim 1, characterized in that, In step three, the preconvolutional embedding consists of 3 × 3 convolution, batch normalization, and ReLU activation; The attention weight is calculated using the following formula: ， in , It is the dimension reduction and projection recovery matrix, and w is the attention weight; 𝛿 represents the ReLU activation function. It represents Sigmoid; In the feature enhancement branch, the data flow processing is as shown in the formula: ， Where 𝑅𝐸𝐿𝑈 represents the ReLU activation function, This indicates a scaling operation. This represents the enhanced output feature map.

5. The remote sensing visual question answering method based on a large language model and a multi-level attention mechanism according to claim 1, characterized in that, The specific method for step four is as follows: In the visual-to-text direction, the self-attention features of the text are... As a key / value pair, visual self-attention features As a query, it generates cross-modal feature representations. ; In the text-visual direction, the generated As a key / value pair, and Continuing as a query; specifically expressed as a formula: ， ， ， Where Self Attn represents bullish self-attention, and Cross Attt represents bidirectional cross attention; The model introduces an independent multi-level gated aggregation module for each interaction direction. Original intra-modal features and cross-participation features are fed along the channel dimension through connections into the multi-level gated aggregation module, which uses Global Average Pooling (GAP) to generate adaptive gate weights. Specifically, the input features are first processed by GAP, then by a learnable fully connected (FC) layer; then, the SiLU activation function is applied to introduce a non-linear transformation; finally, the output features are obtained through gated residual fusion, expressed by the formula: ， ， ， in, and denoted by g, g represents the gated input features, W is the weight matrix of the fully connected layer, LN represents layer normalization, and ⊙ represents element-wise multiplication. After completing the gating fusion step, the model applies a bidirectional cross-attention mechanism again to further refine the alignment and interaction between visual and textual features, resulting in the final prediction result z.

6. The remote sensing visual question answering method based on a large language model and a multi-level attention mechanism according to claim 1, characterized in that, Step 5's entire training process comprises two phases: a semantic segmentation network training phase and a hybrid attention training phase. In the semantic segmentation network training phase, the model trains the semantic segmentation network based on MogaNet, performing end-to-end training to generate land cover classification results. The model simultaneously generates pseudomasks and visual cue vectors with semantic guidance capabilities. These outputs provide fine-grained geospatial cues for subsequent remote sensing visual question answering (VQA) tasks. In the hybrid attention training phase, the focus is on training a hybrid attention mechanism for visual question answering, which consists of two components: a hybrid guided attention module and a bidirectional gated cross-attention module.

Images