A remote sensing visual question answering method based on bimodal transformation and consensus perception

Abstract

The present invention discloses a remote sensing visual question-answering method based on bimodal transformation and consensus perception, including: 1. Obtaining image and text feature representations through an image and text feature extraction network based on a CLIP model; 2. Obtaining text feature representations and picture feature representations with dependencies within and between visual and text representations through a self-attention processing module and a common attention mechanism module; 3. Constructing a knowledge feature extraction network based on a graph convolutional neural network to extract co-occurrence knowledge feature representations; 4. Obtaining comprehensive characterization features through a feature fusion network; 5. Updating network parameters through a cross entropy loss function to obtain an optimal matching model through training. The present invention extracts co-occurrence knowledge feature representations through a knowledge feature extraction network based on a graph convolutional neural network, excavates deep text knowledge, greatly improves the utilization efficiency of text information, and thus greatly improves the accuracy of remote sensing visual question-answering.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically is a remote sensing visual question answering method based on bimodal transformation and consensus perception. Background Art

[0002] Remote sensing image analysis has been receiving particular attention, as there may be a large amount of data in images, enabling many tasks such as land use / land cover classification and object detection. In this regard, effective image analysis often relies on the precise localization of objects therein. In addition to image captioning and text-to-image retrieval, visual question answering (VQA) is another novel querying paradigm, i.e., it involves providing feedback on questions related to an image. Thus, unlike previous image retrieval and captioning tasks, in visual question answering, both the image and the specific question are taken as input. Compared to image captioning, which typically generates a natural language description of the image as a whole, visual question answering offers the advantage of retrieving finer details (e.g., details of certain objects in the image, such as the area they occupy in the image, the relative position of an object with respect to another object, and the density of certain objects within a specific area of ​​the image, etc.).

[0003] The application of visual question answering in remote sensing images is still in its infancy, and the number of contributions is very small. Some existing models attempt to convert visual information into language form, but this conversion may lose key visual details, limiting the model's understanding of complex visual scenes. When dealing with problems, many models may only focus on literal keyword extraction and direct visual feature matching, while ignoring the deep semantic information hidden in the text and taking a long time to converge during training, affecting the efficiency of actual applications. Summary of the invention

[0004] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a remote sensing visual question answering method based on bimodal transformation and consensus perception, in order to capture the dependencies within and between visual and textual representations, and to more effectively utilize consensus knowledge, thereby improving the accuracy of remote sensing visual question answering.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0006] The remote sensing visual question answering method based on dual-modal transformation and consensus perception of the present invention is characterized in that it includes the following steps:

[0007] Step 1: Construct the answer index table AnsVoc and the true answer label O in visual question answering GT ;

[0008] Step 2: Build a feature extraction network based on the CLIP model, including: image feature extraction module and text feature extraction module;

[0009] Step 2.1: Get any image pic and its corresponding question text que and answer text from the visual dialogue dataset, and add it to the real answer label set O GT Extract the real answer label o corresponding to the answer text;

[0010] Step 2.2: The image feature extraction module uses the CLIP model to extract features from the image pic, and obtains a primary region feature set V of the image pic. pic = {v 1 ,...,v i ,...,v q},v i represents the i-th primary region feature of image pic, v i ∈R d ; d represents the dimension of the feature, q represents the total number of features, i=1,...,q;

[0011] Step 2.3: The text feature extraction module uses the CLIP model to extract word-level features from the question text que, and obtains a set of primary word feature sets T = {t 1 ,...,t j ,...,t J}, t j represents the jth primary word feature, t j ∈R m , j = 1, ..., J; m represents the dimension of the feature, J represents the total number of word features;

[0012] Step 3: Build the feature processing module and process V pic and T, and the final region-level feature V" of the image pic and the final word-level feature T" of the question text que are obtained accordingly;

[0013] Step 4: Build a knowledge feature extraction network based on graph convolutional neural network, and use it to process all question texts in the visual question answering dataset to obtain knowledge feature representation Z;

[0014] Step 5: Build a feature fusion network and obtain the instance-level features f of the image pic and its corresponding question text que based on T' and V' I ; According to Z, we get the knowledge features f of the image pic and its corresponding question text que C , thus obtaining the comprehensive representation information f U ;

[0015] Step 6: Optimize the parameters of the remote sensing visual question answering network composed of the feature extraction network, feature processing module, knowledge feature extraction network and feature fusion network to obtain the optimal remote sensing visual question answering model for visual question answering of the input multimodal remote sensing images.

[0016] The remote sensing visual question answering method based on bimodal transformation and consensus perception described in the present invention is also characterized in that the step 1 includes the following steps:

[0017] Step 1.1, obtain a visual question answering dataset, including: a question text set, an image set, and an answer text set; preprocess the answer text set in the visual question answering dataset to obtain a predicted answer set;

[0018] Step 1.2: Based on the predicted answer set, construct the answer index table AnsVoc = {a 1 :r 1 ,a 2 :r 2 ,...,a n :r n ,...,a N :r N}; where a n represents the nth predicted answer, r n Represents the index corresponding to the nth predicted answer, where N is the total number of predicted answers in the answer index table AnsVoc;

[0019] Step 1.3: Use the answer index table AnsVoc to map the answer text set to obtain the real answer label set O GT =[o 1 ,o 2 ,...,o s ,...,o S ], where o s Represents the index of the sth answer text in the answer index table AnsVoc, where S is the total number of answer texts.

[0020] The feature processing module constructed in step 3 includes: a self-attention processing module and a joint attention mechanism module;

[0021] Step 3.1: The self-attention processing module consists of multiple self-attention heads, and converts the region-level image features V pic As the key K, value V and query Q in the attention mechanism, we get the mid-term region-level feature set V' = {v' 1 ,...,v' i ,...,v' qAt the same time, the word-level feature set T is also used as the key K and value V in the attention mechanism to query Q and obtain the mid-term word-level feature set T' = {t' 1 ,...,t' j ,...,t' J}, where v' i represents the i-th mid-term regional-level characteristics, t' j represents the j-th mid-term word-level feature;

[0022] Step 3.2: The joint attention mechanism module uses V' as the query Q in the attention mechanism, and T as the key K and value V in the attention mechanism to obtain the final region-level feature V" = {v" 1 ,...,v” i ,...,v” q}, and use T' as the query Q in the attention mechanism, and V pic As the key K and value V in the attention mechanism, the final word-level feature T” = {t” 1 ,...,t” j ,...,t” J}, where v” i represents the i-th region-level feature, t” j represents the j-th word-level feature.

[0023] The knowledge feature extraction network in step 4 includes: a knowledge concept extraction module, a concept association graph construction module, and a knowledge feature representation module;

[0024] Step 4.1: The knowledge concept extraction module performs word segmentation on all question texts in the visual question answering dataset to obtain segmented words, and selects words with a frequency greater than a threshold and objective meaning from the segmented words as knowledge concepts; then uses the CLIP method to vectorize all knowledge concepts to obtain an instantiated concept representation set C = {c 1 ,...,c k ,...,c x}, c k ∈R d represents the kth concept representation, x represents the total number of concept representations, k = 1, ..., x;

[0025] Step 4.2: The concept association graph construction module constructs a concept association graph through formula (1):

[0026]

[0027] In formula (1), G αβ represents the element in the αth row and βth column of G, Oαβ is the αth concept representation c α and the βth concept represents c β The number of times a question text appears at the same time, θ represents the threshold of the number of occurrences; c α ∈C,c β ∈C;

[0028] Step 4.3: The knowledge feature representation module uses a graph convolutional neural network to learn the concept representation of the concept representation C and the concept related graph G, thereby obtaining the knowledge feature representation Z = [z 1 ,z 2 ,...,z k ,...,z x ],z k ∈R d represents the kth knowledge feature, and x represents the total number of knowledge feature representations.

[0029] The step 5 comprises:

[0030] Step 5.1: Take the average of the final word-level feature T' and the final region-level feature V' and add them together to obtain the instance-level feature f of the image pic and its corresponding question text que I ;

[0031] Step 5.2: Obtain the kth knowledge feature z through formula (2): k The importance score α k , thus obtaining the importance score vector α:

[0032]

[0033] In formula (2), α k represents the kth knowledge feature z k The importance score, W v ∈R d×d is the parameter matrix to be learned, represents the kth knowledge feature z k The transpose of , λ represents the parameter that controls the smoothness of the softmax function;

[0034] Step 5.3: Obtain the knowledge feature f of the image pic and its corresponding question text que through formula (3): C :

[0035]

[0036] Step 5.4: Obtain the comprehensive representation information f of the image pic and its corresponding question text que through formula (4): U ;

[0037] fU =αf C +(1-α)f I (4)

[0038] In formula (4), α is a parameter that determines the relative ratio between the two features.

[0039] The step 6 comprises:

[0040] Step 6.1: f U Input to the softmax layer for processing, and obtain the probability distribution y = [y 1 ,y 2 ,...,y n ,...,y N ], where y n Indicates the nth predicted answer a in the answer index table AnsVoc n The probability of, N is the total number of predicted answers in the answer index table AnsVoc;

[0041] Step 6.2: construct a cross entropy loss based on the probability distribution y of the predicted answer and the true answer label o, and use the Adam optimizer to train the remote sensing visual question answering network, calculate the cross entropy loss to update the network parameters, and stop training until the number of iterations reaches the maximum number of iterations or the cross entropy loss value reaches the minimum, thereby obtaining the optimal remote sensing visual question answering model after training.

[0042] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the remote sensing visual question-answering method, and the processor is configured to execute the program stored in the memory.

[0043] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the remote sensing visual question answering method when the computer program is executed by a processor.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention uses the question text set to acquire concepts, selects words that frequently appear in the question text as labels, uses a graph convolutional network to capture co-occurring concepts as consensus knowledge, designs a consensus perception module to obtain feature representations that incorporate consensus information, and then completes the visual question answering task based on this fused feature, thereby strengthening the semantic relationship between images and texts.

[0046] 2. The present invention proposes a dual-branch feature processing module, which processes image and text data respectively. Each branch independently uses the self-attention mechanism to explore the deep correlation between elements within its own field. It also introduces a common attention mechanism to connect the two branches of image and text, allowing the model to dynamically adjust the interaction between them while considering the image and text information at the same time, thereby achieving a comprehensive capture of the dependencies within and between visual and text representations, providing strong support for remote sensing visual question answering tasks.

[0047] 3. The present invention uses the image and text processing modules of the CLIP model to efficiently process the feature extraction tasks of text and images. With its cross-modal learning ability, it can deeply understand and associate complex information in the visual and language fields. In addition, the model used in the invention captures the regional feature set in the image, obtains each region in the image and its features, and generates a word-level feature set in the sentence, accurately capturing each word in the text and its contextual meaning, thereby achieving a significant improvement in the accuracy of remote sensing visual question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is an architecture diagram of the remote sensing visual question answering method based on dual-modal transformation and consensus perception of the present invention;

[0049] Figure 2 This is a feature processing module diagram of the present invention;

[0050] Figure 3 This is a diagram of the consensus-aware learning module of the present invention. DETAILED DESCRIPTION

[0051] In this embodiment, a remote sensing visual question answering method based on bimodal transformation and consensus perception extracts co-occurrence knowledge feature representation through a knowledge feature extraction network based on a graph convolutional neural network, mines deep text knowledge, greatly improves the utilization efficiency of text information, and thus greatly improves the accuracy of remote sensing visual question answering. Figure 1 As shown in the figure, the model used in this method is divided into three main parts: feature extraction part, consensus concept mining module and consensus perception learning module. Specifically, the method includes the following steps:

[0052] Step 1: Construct the answer index table and true answer label in visual question answering;

[0053] Step 1.1, obtain a visual question answering dataset, including: question text set, image set and answer text set; preprocess the answer text set in the visual question answering dataset to obtain a predicted answer set;

[0054] Step 1.2: Based on the predicted answer set, construct the answer index table AnsVoc = {a1 :r 1 ,a 2 :r 2 ,...,a n :r n ,...,a N :r N}; where a n represents the nth predicted answer, r n Represents the index corresponding to the nth predicted answer, where N is the total number of predicted answers in the answer index table AnsVoc;

[0055] Step 1.3: Use the answer index table AnsVoc to map the answer text set to obtain the real answer label set O GT =[o 1 ,o 2 ,...,o s ,...,o S ], where o s Represents the index of the sth answer text in the answer index table AnsVoc, where S is the total number of answer texts.

[0056] In this embodiment, feature extraction is performed on images in two datasets, RSVQA_LR and RSVQA_HR. RSVQA_LR is a remote sensing visual question answering dataset containing 772 images, each of which is 256×256 pixels in size. Then, 77,232 pairs of questions and answers are generated. For each image, many questions and answers are given (approximately 100 questions and answers for each image). 77.8% of the dataset is used for training, 11.1% and 11.1% are used for verification and testing respectively. RSVQA_HR is another high-resolution remote sensing visual question answering dataset, which includes a total of 10,659 images, each of which is 512×512 pixels in size. Similarly, 1,066,316 pairs of questions and answers are generated (approximately 100 questions and answers for each image). 61.5% of the dataset is used for training, and the rest is used as a test set (20.5% for test set 1 and 6.8% for test set 2).

[0057] Step 2: Build a feature extraction network based on the CLIP model, including: image feature extraction module and text feature extraction module;

[0058] Step 2.1: Get any image pic and its corresponding question text que and answer text from the visual dialogue dataset, and add it to the real answer label set O GT Extract the true answer label o corresponding to the answer text;

[0059] Step 2.2: The image feature extraction module uses the CLIP model to extract features from the image pic and obtains the primary regional feature set V of the image pic.pic = {v 1 ,...,v i ,...,v q},v i represents the i-th primary region feature of image pic, v i ∈R d ; d represents the dimension of the feature, q represents the total number of features, i=1,...,q;

[0060] Step 2.3: The text feature extraction module uses the CLIP model to extract word-level features from the question text que, and obtains a set of primary word feature sets T = {t 1 ,...,t j ,...,t J}, t j represents the jth primary word feature, t j ∈R m , j=1,...,J; m represents the dimension of the feature, and J represents the total number of word features.

[0061] Step 3: Build a feature processing module. The feature processing module is as follows: Figure 2 As shown, it includes: a self-attention processing module and a co-attention mechanism module, which use self-attention and co-attention mechanisms to capture the dependencies within and between visual and text representations;

[0062] Step 3.1: The self-attention processing module consists of multiple self-attention heads and converts the region-level image features V pic As the key K, value V and query Q in the attention mechanism, we get the mid-term region-level feature set V' = {v' 1 ,...,v' i ,...,v' q At the same time, the word-level feature set T is also used as the key K and value V in the attention mechanism to query Q and obtain the mid-term word-level feature set T' = {t' 1 ,...,t' j ,...,t' J}, where v' i represents the i-th mid-term regional-level characteristics, t' j represents the j-th mid-term word-level feature;

[0063] Step 3.2: The joint attention mechanism module takes V' as the query Q in the attention mechanism, T as the key K and value V in the attention mechanism, and obtains the final region-level feature V" = {v" 1 ,...,v” i ,...,v” q}, and use T' as the query Q in the attention mechanism, and Vpic As the key K and value V in the attention mechanism, the final word-level feature T” = {t” 1 ,...,t” j ,...,t” J}, where v” i represents the i-th region-level feature, t” j represents the j-th word-level feature.

[0064] Step 4: Build a knowledge feature extraction network based on graph convolutional neural network, including: knowledge concept extraction module, concept association graph construction module, knowledge feature representation module;

[0065] Step 4.1: The knowledge concept extraction module performs word segmentation on all question texts in the visual question answering dataset to obtain segmented words. In this example, knowledge concepts are extracted from the question text corpus in the dataset, and the vocabulary frequency is calculated using a probability-based method. Words with a frequency greater than a threshold and objective meaning are selected from the segmented words as knowledge concepts. Then, the CLIP method is used to vectorize all knowledge concepts to obtain an instantiated concept representation set C = {c 1 ,...,c k ,...,c x}, c k ∈R d represents the kth concept representation, x represents the total number of concept representations, k = 1,...,x. This module ensures that the model can use shared knowledge for more advanced reasoning when processing images and texts.

[0066] Step 4.2: The concept association graph construction module constructs the concept association graph through formula (1):

[0067]

[0068] In formula (1), G αβ represents the element in the αth row and βth column of G, O αβ is the αth concept representation c α and the βth concept represents c β The number of times a question text appears at the same time, θ represents the threshold of the number of occurrences; c α ∈C, c β ∈C;

[0069] Step 4.3: The knowledge feature representation module uses a graph convolutional neural network to learn the concept representation of the concept representation C and the concept-related graph G, thereby obtaining the knowledge feature representation Z = [z 1 ,z 2 ,...,z k ,...,z x],z k ∈R d represents the kth knowledge feature, and x represents the total number of knowledge feature representations.

[0070] Step 5: Build a feature fusion network; Figure 3 As shown in Figure 2, in this module, the model fuses consensus-level representation features with instance-level representation features. This step improves the model’s understanding of high-level semantic relationships between images and text by integrating consensus-aware concept features.

[0071] Step 5.1: Take the average of the final word-level feature T' and the final region-level feature V' and add them together to obtain the instance-level feature f of the image pic and its corresponding question text que I ;

[0072] Step 5.2: Obtain the kth knowledge feature z through formula (2): k The importance score α k , thus obtaining the importance score vector α:

[0073]

[0074] In formula (2), α k represents the kth knowledge feature z k The importance score, W v ∈R d×d is the parameter matrix to be learned, represents the kth knowledge feature z k , λ represents the parameter that controls the smoothness of the softmax function.

[0075] Step 5.3: Obtain the knowledge feature f of the image pic and its corresponding question text que through formula (3): C :

[0076]

[0077] Step 5.4: Obtain the comprehensive representation information f of the image pic and its corresponding question text que through formula (4): U ;

[0078] f U =αf C +(1-α)f I (4)

[0079] In formula (4), α is a parameter that determines the relative ratio between the two features.

[0080] Step 6, optimizing the parameters of the remote sensing visual question answering network based on bimodal transformation and consensus perception, which is composed of a feature extraction network, a feature processing module, a knowledge feature extraction network, and a feature fusion network;

[0081] Step 6.1: f U Input to the softmax layer for processing, and obtain the probability distribution y = [y 1 ,y 2 ,...,y n ,...,y N ], where y n Indicates the nth predicted answer a in the answer index table AnsVoc n The probability of, N is the total number of predicted answers in the answer index table AnsVoc;

[0082] Step 6.2: Construct a cross entropy loss based on the predicted value y and the true answer label o, and use the Adam optimizer to train the remote sensing visual question answering network. Calculate the cross entropy loss to update the network parameters until the number of iterations reaches the maximum number of iterations or the cross entropy loss value reaches the minimum. Stop training to obtain the optimal remote sensing visual question answering model after training, which is used to perform visual question answering on the input multimodal remote sensing images.

[0083] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0084] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A remote sensing visual question answering method based on bimodal transformation and consensus perception, characterized in that: The steps include: Step 1: Construct the answer index table AnsVoc and the true answer label in visual question answering ; Step 2: Build a feature extraction network based on the CLIP model, including: image feature extraction module and text feature extraction module; Step 2.1: Get any image pic and its corresponding question text que and answer text from the visual dialogue dataset, and find the answer in the real answer label set Extract the real answer label o corresponding to the answer text; Step 2.2: The image feature extraction module uses the CLIP model to extract features from the image pic to obtain a primary regional feature set of the image pic. , represents the i-th primary region feature of image pic, ; d represents the dimension of the feature, q represents the total number of features, ; Step 2.3: The text feature extraction module uses the CLIP model to extract word-level features from the question text que to obtain a set of primary word feature sets. , represents the jth primary word feature, , ; m represents the dimension of the feature, J represents the total number of word features; Step 3: Build the feature processing module, including: self-attention processing module, joint attention mechanism module, and and Processing is performed to obtain the final region-level features of the image pic and the final word-level features of the question text que ; Step 3.1: The self-attention processing module consists of multiple self-attention heads, and the region-level image features As the key K, value V and query Q in the attention mechanism, we get the mid-term region-level feature set of the image pic At the same time, the word-level feature set Also used as the key K and value V in the attention mechanism, query Q, and obtain the mid-term word-level feature set of the question text que ,in, represents the i-th mid-term regional-level characteristics, represents the j-th mid-term word-level feature; Step 3.2: The joint attention mechanism module will As the query Q in the attention mechanism, As the key K and value V in the attention mechanism, the final region-level features of the image pic are obtained , and at the same time As the query Q in the attention mechanism, As the key K and value V in the attention mechanism, the final word-level features of the question text que are obtained ,in, represents the i-th region-level feature, represents the jth word-level feature; Step 4: Build a knowledge feature extraction network based on graph convolutional neural network, including: knowledge concept extraction module, concept association graph construction module, knowledge feature representation module, and process all question texts in the visual question answering dataset to obtain knowledge feature representation Z; Step 4.1: The knowledge concept extraction module performs word segmentation on all question texts in the visual question answering dataset to obtain segmented words, and selects words with a frequency greater than a threshold and objective meaning from the segmented words as knowledge concepts; then uses the CLIP method to vectorize all knowledge concepts to obtain an instantiated concept representation set. , represents the kth concept representation, x represents the total number of concept representations, ; Step 4.2: The concept association graph construction module constructs a concept association graph through formula (1): (1) In formula (1), express Middle Line The elements of the column, It is Conceptual representation and Conceptual representation The number of times a question text appears at the same time, Indicates the threshold of the number of occurrences; , ; Step 4.3: The knowledge feature representation module uses a graph convolutional neural network to learn the concept representation of the concept representation C and the concept related graph G, thereby obtaining the knowledge feature representation , represents the kth knowledge feature, and x represents the total number of knowledge feature representations; Step 5: Build a feature fusion network and and Get the instance-level features of the image pic and its corresponding question text que ; According to Z, we can get the knowledge features of the image pic and its corresponding question text que , thus obtaining comprehensive characterization information ; Step 5.1: Final word-level features and final regional-level characteristics After taking the average value and adding them together, we get the instance-level features of the image pic and its corresponding question text que ; Step 5.2: Get the kth knowledge feature through formula (2): Importance score : (2) In formula (2), represents the kth knowledge feature The importance score of is the parameter matrix to be learned, represents the kth knowledge feature The transpose of Represents the parameter that controls the smoothness of the softmax function; Step 5.3: Get the knowledge features of the image pic and its corresponding question text que through formula (3) : (3) Step 5.4: Obtain the comprehensive representation information of the image pic and its corresponding question text que through formula (4) ; f U =ρf C +(1-ρ)f I (4) In formula (4), ρ is a parameter that determines the relative ratio between two features; Step 6: Optimize the parameters of the remote sensing visual question answering network composed of the feature extraction network, the feature processing module, the knowledge feature extraction network and the feature fusion network to obtain the optimal remote sensing visual question answering model for visual question answering of the input multimodal remote sensing images; Step 6.1: Input to the softmax layer for processing to obtain the probability distribution of each predicted answer , where y n Indicated in the answer index table The nth predicted answer a n The probability of, N is the total number of predicted answers in the answer index table AnsVoc; Step 6.2: construct a cross entropy loss based on the probability distribution y of the predicted answer and the true answer label o, and use the Adam optimizer to train the remote sensing visual question answering network, calculate the cross entropy loss to update the network parameters, and stop training until the number of iterations reaches the maximum number of iterations or the cross entropy loss value reaches the minimum, thereby obtaining the optimal remote sensing visual question answering model after training.

2. According to claim 1, a remote sensing visual question answering method based on dual-modal transformation and consensus perception is characterized in that: The step 1 comprises the following steps: Step 1.1, obtain a visual question answering dataset, including: a question text set, an image set, and an answer text set; preprocess the answer text set in the visual question answering dataset to obtain a predicted answer set; Step 1.2: Build the answer index table based on the predicted answer set ; Among them, a n represents the nth predicted answer, r n Represents the index corresponding to the nth predicted answer, where N is the total number of predicted answers in the answer index table AnsVoc; Step 1.3: Use the answer index table AnsVoc to map the answer text set to obtain the real answer label set ,in, Represents the index of the sth answer text in the answer index table AnsVoc, where S is the total number of answer texts.

3. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the remote sensing visual question answering method according to claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the remote sensing visual question answering method according to any one of claims 1 or 2 are executed.

Images