Position and semantic optimization method and system for remote sensing visual question answering

Abstract

The present invention belongs to the technical field of visual question answering, and discloses a position and semantic optimization method and system for remote sensing visual question answering, which is divided into a text stream and an ocean remote sensing image stream, and includes the following steps: step S1, inputting ocean remote sensing images and texts; step S2, cross-modal context fusion: integrating text and ocean remote sensing image information through a cross-modal position constraint attention mechanism to generate cross-modal features; step S3, semantic reasoning: extracting semantic information in a semantic graph Gs through a semantic graph Transformer to obtain the final refined semantic information; finally, outputting a prediction classification head to obtain a predicted answer. The present invention enhances the perception of multi-order knowledge of a visual question answering system, enhances reasoning ability, and obtains accurate prediction results.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual question answering, and in particular relates to a method and system for optimizing position and semantics for remote sensing visual question answering. Background Art

[0002] Visual question answering refers to answering questions corresponding to each image based on the image, which mainly includes two important steps: First, feature extraction of remote sensing images and texts to mine their deep / high-level / semantic representations. Second, prediction of the answer to the question based on the extracted deep / high-level / semantic representations. For process one, deep neural network representation models are currently mainly used, such as: convolution-based deep neural network feature extractors, such as: hollow convolution, deformable convolution, word embedding, etc. For process two, cascade, gating, attention mechanism and other methods are currently mainly used to obtain robust multimodal representations.

[0003] However, the existing remote sensing visual question answering methods have the following problems when applied to marine remote sensing images:

[0004] (1) It is unable to adaptively cope with the accumulation of complex noise, and it is difficult to establish a stable visual question answering model. Marine remote sensing images contain a large amount of complex noise (such as the striping phenomenon caused by the remote sensor detecting uniform surface targets (such as water bodies), waves on the ocean surface, and uneven water bodies in the ocean). Existing visual question answering methods perform feature extraction, context mining, deconvolution and other operations on the original image to generate feature representations, but these feature representations can only describe low-order information, such as color, shape, and area. When encountering complex noise, such as the striping phenomenon in the ocean and the uneven water bodies in the ocean, the low-order features such as image color and shape will not be stable, and complex noise cannot be adaptively eliminated / mitigated. The impact of noise may even be amplified, reducing the accuracy of the visual question answering model.

[0005] (2) The semantic gap between marine remote sensing images and natural language. Existing visual question answering methods generate feature representations by performing operations such as feature extraction, context mining, and deconvolution on the original image, and directly fuse them with text features through cascading, gating, or attention mechanisms. However, these multimodal feature representations themselves have the problem of a semantic gap. When the foreground and background of remote sensing images are unbalanced and there are too many non-semantic words in natural language, it is very easy to cause the representation to be non-robust, which further widens the semantic gap between marine remote sensing visual features and text features, and reduces the accuracy of the visual question answering model.

[0006] (3) Accumulation of redundant information in multimodal representations. Most previous studies simply concatenate multimodal (ocean remote sensing images and text) representations, resulting in the accumulation of redundant information and limited contextual interactions between modes. When encountering complex noise, such as banding in the ocean and uneven water bodies in the ocean, the image color, shape and other features will not be stable, and complex noise cannot be adaptively eliminated / mitigated. The impact of noise may even be amplified, which further widens the semantic gap between ocean remote sensing visual features and text features and reduces the accuracy of visual question answering models. In addition, other work only considers simple contextual features and ignores semantic clues, resulting in insufficient capture of the semantic coherence of remote sensing visual question answering and the consistency of multimodal semantic co-reference. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a method and system for position and semantic optimization for remote sensing visual question answering, following the structure of a dual-tower network, which is a text stream and an ocean remote sensing image stream. First, a cross-modal position constraint module is designed, which integrates text and ocean remote sensing image information through cross-modal position constraint attention. The module takes into account the influence of unimodal semantics and cross-modal semantic associations. Secondly, this patent designs a semantic reasoning module to extract effective semantic features and contextual information, including the interactive relationship between contextual information and multimodal objects.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] First, the present invention provides a position and semantic optimization method for remote sensing visual question answering, which is divided into a text stream and an ocean remote sensing image stream, and includes the following steps:

[0010] Step S1, inputting ocean remote sensing images and text;

[0011] Step S2, cross-modal context fusion: Integrate text and ocean remote sensing image information through the cross-modal position constrained attention mechanism to generate cross-modal features; the details are as follows:

[0012] Step S21, single mode characterization:

[0013] The marine remote sensing image and text data are respectively extracted through the image encoder and text encoder to extract image and text features, represented as X i , then X i The unimodal representation is obtained by encoding through the Transformer-based multi-head attention mechanism and residual mechanism. , where i represents the modality, and the image modality is Indicates that the text mode adopts express;

[0014] S22. Multimodal Fusion:

[0015] For the image modality representation obtained in step S21 and text modality representation , weighted constraints are performed through the cross-modal position constraint attention mechanism, and the constrained multimodal representation X is output c , X c Image modality representation , text modality representation Fusion to obtain cross-modal features , used for subsequent graph modeling;

[0016] The cross-modal position constrained attention mechanism is implemented by cross-attention of image and text and weighted constrained attention LCA of category position information, wherein the weighted constrained attention LCA of category position information is obtained by computing category information SA and position information RP of the remote sensing image. Specifically, the category information SA and position information RP of the remote sensing image are element-wise multiplied with the weight matrix of the traditional attention mechanism to obtain the final attention weight matrix as the weighted constraint of category position information;

[0017] Step S3: semantic reasoning:

[0018] First, the cross-modal features obtained in step S2 The semantic graph Gs is obtained by graph modeling through the relational graph convolution network; then, the semantic encoding is calculated. and relative position encoding , and input it into the semantic graph Transformer, through which the semantic information in the semantic graph Gs is extracted to obtain the final refined semantic information; finally, the prediction classification head is output to obtain the prediction answer.

[0019] Furthermore, step S21 is specifically as follows: i The attention map of the attention weights of a single attention head is obtained through the attention mechanism of Transformer, and the weighted sum is calculated using these attention weights to obtain the output; finally, a residual connection and a layer norm are added to obtain a unimodal representation ;

[0020] (1);

[0021] (2);

[0022] Where i represents the sequence number of the i-th input feature, , , represent the query, key, and value vectors of the h-th attention head, which are composed of the input features X iObtained through linear transformation, softmax is the softmax function, which is used to convert the attention weight matrix into a probability distribution to ensure that the sum of all weights is 1, and k represents the dimension of the key vector matrix. Used to scale the key vector matrix;

[0023] represents the weighted sum of the outputs of the hth attention head, N represents the total number of multi-head attentions, , , ..., Represent the weighted sum of the outputs of the 1st, 2nd, ..., Nth attention heads, respectively. represents an element-wise addition operation, and W is a learnable weight matrix used to map the concatenated matrix to the original feature dimension.

[0024] Furthermore, in step S22, the cross-modal position constrained attention mechanism is implemented through cross-attention of image and text and category position information weighted constrained attention LCA, wherein the category position information weighted constrained attention LCA is obtained after operation of category information SA and position information RP of the remote sensing image. Specifically: the category information SA and position information RP of the remote sensing image are element-wise multiplied with the weight matrix of the traditional attention mechanism to obtain the final attention weight matrix as the category position information weighted constraint.

[0025] Further, in step S22, the category information SA is determined by the image category and the entity category in the image, and the position information RP is determined by the relative position; specifically:

[0026] The category information SA is used to focus on the category semantics of the current remote sensing image and simulate the semantic inertia of the category information on the current remote sensing image:

[0027] (3);

[0028] Among them, s m is the image category of the current remote sensing image, s n is the entity category in the remote sensing image, i.e., class information, It is the strength of association between the image category and entity category of the current remote sensing image. A value of 1 means that the representation remains unchanged, and a value of 0 means that it is discarded.

[0029] The position information RP is represented by calculating the visual distance between the image regions um and un. If the visual distance is short, the semantic metric RP value is large, indicating that the relative importance between the image regions um and un is high.

[0030] Furthermore, in step S22, the image modality representation and text modality representation The weighted constraint through the cross-modal position-constrained attention mechanism is achieved in the following way: first, and The weighted representation is then weighted by LCA weighted by the weighted attention constraint of the category position information, and the constrained visual representation and text representation are output. The two are added to obtain the constrained multimodal representation X c ;

[0031] Among them, in step S22, and The cross-attention weighting is to simulate the interaction between the two modalities by cross-attention of the two streams of image and text. The cross-attention combines the multi-modal image and text information and transforms the text features using the representation of the image modality.

[0032] For a single attention head, the cross-modal position-constrained attention mechanism is expressed as follows:

[0033] (4);

[0034] in, Represents the weight matrix in the traditional Transformer. LCA represents the weighted constrained attention of category position information. It is an attention matrix constrained by category information and position information, which is used to emphasize local context information. represents the j-th value vector, It represents the output value corresponding to the i-th query vector Q and the j-th key vector K after calculation by the cross-modal position constraint attention mechanism; represents the element-wise multiplication operator, which combines the traditional Transformer attention weights and the local constrained attention weights. represents the matrix multiplication operator, which multiplies the final weight with the value vector to obtain the output value.

[0035] Furthermore, step S3 is specifically as follows:

[0036] Step S31: construct a semantic graph:

[0037] Cross-modal features The semantic graph Gs is constructed through the relational graph convolutional network, which maps text and images to an entity space as points. The graph structure of the semantic graph includes points and edges. Points represent the semantics of image entities and text entities; edges represent the interactive relationship between entities, including the relationship between text and text, the relationship between text and image, the relationship between text and multimodal representation, the relationship between multimodal representation and image, and the relationship between text and image.

[0038] Step S32: semantic information refinement:

[0039] Compute semantic encoding through entity feature vector and relative position encoding , input into the semantic graph Transformer to obtain the final refined semantic information;

[0040] Step S33: predict the classification head.

[0041] Furthermore, step S31 is specifically as follows:

[0042] Defining a semantic graph , each entity is represented by a node, different connecting edges represent directed relationships, Vs represents a set of entity nodes, A set of relations representing the semantic similarity between entities is defined as formula (5),

[0043] (5);

[0044] in, represents the semantic similarity between entity m and entity n, and Represent the feature vectors of entity m and entity n respectively, T represents transposition, represents the Euclidean norm, that is, the length of a vector; arccos is the inverse cosine function, which is used to convert a cosine value into an angle value.

[0045] Further, step S32 is specifically as follows:

[0046] Step S321, determine semantic and position coding information:

[0047] Given the node features of the semantic graph Gs , Represent the 1st, 2nd, ..., nth node features of the semantic graph respectively; define two codes to represent the semantic relationship between two nodes in the semantic graph Gs, the first one is the relative position code , represents the relative position information of the entity in the semantic graph Gs, Each vector in represents the distance between nodes and is normalized by a sigmoid function to be in the range of [0, 1]; the second is the semantic encoding , calculated by formula (10) It is represented by calculating the cosine similarity of the two node features and converting it into a value in the range of [0, 1]; Each vector in represents the topological relationship represented by the shortest path distance between two nodes. and Add the two together to get the semantic position encoding , Use a vector to represent the shortest path distance between two nodes, thereby reflecting the semantic similarity between them;

[0048] Step S322: The semantic graph Transformer encodes node features and semantic positions Combined with the semantic graph transformer, semantic information is extracted from node features to obtain refined semantic representation, i.e., updated node features. , specifically:

[0049] The node features of the semantic graph Gs are converted into the query vector Q, key vector K, and value vector V of the attention mechanism, and the semantic similarity and attention scores between nodes are calculated and compared with the semantic position encoding. Combine, calculate , the formula is as follows:

[0050] (6);

[0051] (7);

[0052] (8);

[0053] In formula (6), represents the attention score between node i and node j, and is a learnable weight matrix used to transform node features and Converted to query and key vectors, d is a predefined dimension used to normalize the attention scores; is the semantic and position information item of node ij, indicating the semantic similarity between node i and node j, Represents the calculation of query vector and key vector The dot product between them is used to measure the similarity between node i and node j;

[0054] In formula (8), represents the updated node features, represents the feature vector of node j, represents the normalized attention score, G N represents the number of adjacent nodes, Manipulate the feature vectors representing neighboring nodes, adding positional encoding and semantic information.

[0055] The present invention also provides a position and semantic optimization system for remote sensing visual question answering, which is used to implement the position and semantic optimization method for remote sensing visual question answering as described above. The system adopts a dual-tower network structure, which is a text stream and an ocean remote sensing image stream, and includes a single-modal representation module, a cross-modal position constraint module and a semantic reasoning module.

[0056] The unimodal representation module includes a visual embedding block and a text embedding block, each including an encoder, a Transformer-based multi-head attention mechanism and a residual mechanism, and the encoder includes an image encoder and a text encoder, which are used to extract image and text features X of remote sensing images and text data, respectively. i , X i After being processed by the Transformer-based multi-head attention mechanism and residual mechanism, the unimodal representation of image and text is output. ;

[0057] The cross-modal position constraint module uses the cross-modal position constraint attention mechanism to perform weighted constraints and fusion on the image modality representation and the text modality representation, integrates the text and ocean remote sensing image information, and generates cross-modal features. , used for subsequent graph modeling;

[0058] The semantic reasoning module is used to construct a semantic graph and refine the graph semantic information, and output a prediction classification head.

[0059] Compared with the prior art, the present invention has the advantages of:

[0060] (1) Adaptively eliminate / mitigate noise interference. A cross-modal position constraint module is designed to integrate text and ocean remote sensing image information through cross-modal position constraint attention, learn multi-modal features under mutual guidance, and play a role in eliminating ambiguity and denoising. By utilizing effective class information and cross-attention, a more stable and accurate robust representation constrained by class relations is constructed.

[0061] (2) The compactness of unimodal features is improved while the discriminability of multimodal features is expanded. A semantic graph is defined to model the relationship between adjacent utterances, and a semantic graph-based reasoning encoder is used to capture the global underlying semantics, making the model more sensitive to intra-class and inter-class representations.

[0062] (3) Improved feature robustness. A semantic reasoning module is designed to extract effective semantic features and contextual information, including the interaction between contextual information and multimodal objects. By adding a reasoning mechanism through position encoding weighting, the interaction between spatial position and multimodal features is realized. The introduction of additional weighting further improves the robustness of the features. Design of semantic information refinement submodule: By introducing topological relationship supervision information, it aims to enhance the question-answering system's perception of multi-order knowledge, and then enhance the model's reasoning ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0064] Figure 1 It is a structural diagram of the present invention;

[0065] Figure 2 It is a schematic diagram of the single-modal embedding representation of the present invention;

[0066] Figure 3 Schematic diagram of the cross-modal position-constrained attention mechanism of the present invention;

[0067] Figure 4 Schematic diagram of the semantic graph Transformer of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0069] Combination Figure 1-Figure 4 As shown, this embodiment provides a location and semantic optimization method for remote sensing visual question answering, which is divided into a text stream and an ocean remote sensing image stream, and includes the following steps:

[0070] Step S1, inputting ocean remote sensing images and text;

[0071] Step S2, cross-modal context fusion: Integrate text and ocean remote sensing image information through the cross-modal position constrained attention mechanism to generate cross-modal features; the details are as follows:

[0072] Step S21, single mode characterization:

[0073] The marine remote sensing image and text data are respectively extracted through the image encoder and text encoder to extract image and text features, represented as X i , then X iThe unimodal representation is obtained by encoding through the Transformer-based multi-head attention mechanism and residual mechanism. , where i represents the modality, and the image modality is Indicates that the text mode adopts express.

[0074] As a specific implementation, Figure 2 As shown, step S21 is specifically as follows: i The attention map of the attention weights of a single attention head is obtained through the attention mechanism of Transformer, and the weighted sum is calculated using these attention weights to obtain the output; finally, a residual connection and a layer norm are added to obtain a unimodal representation ;

[0075] (1);

[0076] (2);

[0077] Where i represents the sequence number of the i-th input feature, , , represent the query, key, and value vectors of the h-th attention head, which are composed of the input features X i Obtained through linear transformation, softmax is the softmax function, which is used to convert the attention weight matrix into a probability distribution to ensure that the sum of all weights is 1, and k represents the dimension of the key vector matrix. Used to scale the key vector matrix to avoid the gradient vanishing problem. Figure 2 The PE in the figure represents position encoding. The feedforward neural network transmits information through layers. The residual connection solves the degradation problem of deep networks. Normalization stabilizes the output distribution. The above content is a commonly used technique in the Transformer model to retain the position information of elements in the sequence. I will not go into details here.

[0078] represents the weighted sum of the outputs of the hth attention head, N represents the total number of multi-head attentions, , , ..., Represent the weighted sum of the outputs of the 1st, 2nd, ..., Nth attention heads, respectively. represents an element-wise addition operation, and W is a learnable weight matrix used to map the concatenated matrix to the original feature dimension.

[0079] The two formulas together describe the calculation process of the self-attention mechanism in the Transformer model. The attention weight is obtained by calculating the similarity between the query and key vectors and normalizing them using the softmax function. Then, the attention weight is multiplied with the value vector and weighted summed to obtain the final output feature.

[0080] S22. Multimodal Fusion:

[0081] like Figure 2 As shown, for the image modality representation obtained in step S21 and text modality representation , weighted constraints are performed through the cross-modal position constraint attention mechanism, and the constrained multimodal representation X is output c , X c Image modality representation , text modality representation Fusion to obtain cross-modal features , used for subsequent graph modeling.

[0082] The cross-modal position constrained attention mechanism is implemented through cross-attention of image and text and category position information weighted constrained attention LCA, wherein the category position information weighted constrained attention LCA is obtained through operation of category information SA and position information RP of the remote sensing image. Specifically: the category information SA and position information RP of the remote sensing image are element-wise multiplied with the weight matrix of the traditional attention mechanism to obtain the final attention weight matrix as the category position information weighted constraint.

[0083] In step S22, the category information SA is determined by the image category and the entity category in the image, and the position information RP is determined by the relative position; specifically:

[0084] The category information SA is used to focus on the category semantics of the current remote sensing image and simulate the semantic inertia of the category information on the current remote sensing image:

[0085] (3);

[0086] Among them, s m is the image category of the current remote sensing image, s n is the entity category in the remote sensing image, i.e., class information, It is the strength of association between the image category and the entity category of the current remote sensing image. A value of 1 means that the representation remains unchanged, and a value of 0 means it is discarded.

[0087] The position information RP is represented by calculating the visual distance between the image regions um and un. If the visual distance is short, the semantic metric RP value is large, indicating that the relative importance between the image regions um and un is high.

[0088] In step S22, the image modality is represented and text modality representation The weighted constraint through the cross-modal position-constrained attention mechanism is achieved in the following way: first, and The weighted representation is then weighted by LCA weighted by the weighted attention constraint of the category position information, and the constrained visual representation and text representation are output. The two are added to obtain the constrained multimodal representation X c .

[0089] Among them, in step S22, and The cross-attention weighting is to simulate the interaction between the two modalities by performing cross-attention on the two streams of image and text. The cross-attention combines the multimodal image and text information and uses the representation of the image modality to transform the text features.

[0090] in Figure 1 In It represents the attention heads of two modalities: image and text. It represents the attention head after constraint.

[0091] For a single attention head, the cross-modal position-constrained attention mechanism is expressed as follows:

[0092] (4);

[0093] in, Represents the weight matrix in the traditional Transformer. LCA represents the weighted constrained attention of category position information. It is an attention matrix constrained by category information and position information, which is used to emphasize local context information. represents the j-th value vector, It represents the output value corresponding to the i-th query vector Q and the j-th key vector K after calculation by the cross-modal position constraint attention mechanism; represents the element-wise multiplication operator, which combines the traditional Transformer attention weights and the local constrained attention weights. represents the matrix multiplication operator, which multiplies the final weight with the value vector to obtain the output value.

[0094] Step S3: semantic reasoning:

[0095] First, the cross-modal features obtained in step S2 The semantic graph Gs is obtained by graph modeling through the relational graph convolution network; then, the semantic encoding is calculated. and relative position encoding , and input it into the semantic graph Transformer, through which the semantic information in the semantic graph Gs is extracted to obtain the final refined semantic information; finally, the prediction classification head is output to obtain the prediction answer.

[0096] Step S3 is as follows:

[0097] Step S31: construct a semantic graph:

[0098] Cross-modal features Constructing a semantic graph Gs through a relational graph convolutional network is to map text and images to an entity space as points. The graph structure of the semantic graph includes points and edges. Points represent the semantics of image entities and text entities; edges represent the interactive relationships between entities, including the relationship between text and text, the relationship between text and image, the relationship between text and multimodal representation, the relationship between multimodal representation and image, and the relationship between text and image.

[0099] In order to establish semantic relations between adjacent multimodal semantic features and capture the effects within a single modality and between multimodalities, a semantic graph is defined. , each entity is represented by a node, different connecting edges represent directed relationships, Vs represents a set of entity nodes, A set of relations representing the semantic similarity between entities is defined as formula (5),

[0100] (5);

[0101] in, represents the semantic similarity between entity m and entity n, and Represent the feature vectors of entity m and entity n respectively, T represents transposition, represents the Euclidean norm, that is, the length of a vector; arccos is the inverse cosine function, which is used to convert a cosine value into an angle value.

[0102] Step S32: semantic information refinement:

[0103] Compute semantic encoding through entity feature vector and relative position encoding , input into the semantic graph Transformer to obtain the final refined semantic information.

[0104] Step S32 is specifically as follows:

[0105] Step S321, determine semantic and position coding information:

[0106] Given the node features of the semantic graph Gs , Represent the 1st, 2nd, ..., nth node features of the semantic graph respectively; define two codes to represent the semantic relationship between two nodes in the semantic graph Gs, the first one is the relative position code , represents the relative position information of the entity in the semantic graph Gs, Each vector in represents the distance between nodes and is normalized by a sigmoid function to be in the range of [0, 1]; the second is the semantic encoding , calculated by formula (10) It is represented by calculating the cosine similarity of the two node features and converting it into a value in the range of [0, 1]; Each vector in represents the topological relationship represented by the shortest path distance between two nodes. and Add the two together to get the semantic position encoding , A vector is used to represent the shortest path distance between two nodes, thereby reflecting the semantic similarity between them.

[0107] Step S322: The semantic graph Transformer encodes node features and semantic positions Combined with the semantic graph transformer, semantic information is extracted from node features to obtain refined semantic representation, i.e., updated node features. , specifically:

[0108] like Figure 4 As shown in the figure, the node features of the semantic graph Gs are converted into the query vector Q, key vector K, and value vector V of the attention mechanism, and the semantic similarity and attention score between nodes are calculated and compared with the semantic position encoding Combine, calculate , the formula is as follows:

[0109] (6);

[0110] (7);

[0111] (8).

[0112] First, Q, K and Calculation (Formula (7)), secondly, Q, K and Calculate and get (Formula (6)), finally v, , The refined representation is calculated (Formula (13)). Through Formula (6)-Formula (13), the semantic graph Transformer combines the node features and semantic relationship encoding to form a new node feature , thereby capturing the semantic relationship between nodes.

[0113] In formula (6), represents the attention score between node i and node j, and is a learnable weight matrix used to transform node features and Converted to query and key vectors, d is a predefined dimension used to normalize the attention scores; is the semantic and position information item of node ij, indicating the semantic similarity between node i and node j, Represents the calculation of query vector and key vector The dot product between them is used to measure the similarity between node i and node j.

[0114] Formula (6) calculates the attention score between node i and node j, taking into account node features and semantic similarity. The higher the attention score, the higher the degree of attention node i pays to node j, and the greater the contribution of node j to the semantic information of node i.

[0115] In formula (7), and Represents the position encoding vector, which indicates the relative position of node i and node j in the semantic space.

[0116] Formula (7): Defined The calculation method of , which comprehensively considers relative position encoding and semantic similarity. It can reflect the semantic relationship between node i and node j and is used to calculate the attention score.

[0117] In formula (8), represents the updated node features, represents the feature vector of node j, represents the normalized attention score, G N represents the number of adjacent nodes, Manipulate the feature vectors representing neighboring nodes, adding positional encoding and semantic information.

[0118] Formula (8): The updated feature vector of node i is calculated by combining node i’s own features, node i’s attention score to other nodes, node j’s feature vector, relative position encoding and semantic similarity. In this way, the global semantic relationship between nodes can be captured and the feature vector of the node can be updated, thereby extracting richer semantic information.

[0119] Step S33: predict the classification head.

[0120] As another embodiment of the present invention, a position and semantic optimization system for remote sensing visual question answering is provided, which is used to implement the position and semantic optimization method for remote sensing visual question answering as described above. The system adopts a dual-tower network structure, which is a text stream and an ocean remote sensing image stream, respectively, and includes a single-modal representation module, a cross-modal position constraint module and a semantic reasoning module.

[0121] The unimodal representation module includes a visual embedding block and a text embedding block, each including an encoder, a Transformer-based multi-head attention mechanism and a residual mechanism, and the encoder includes an image encoder and a text encoder, which are used to extract image and text features X of remote sensing images and text data, respectively. i , X i After being processed by the Transformer-based multi-head attention mechanism and residual mechanism, the unimodal representation of image and text is output. .

[0122] The cross-modal position constraint module uses the cross-modal position constraint attention mechanism to perform weighted constraints and fusion on the image modality representation and the text modality representation, integrates the text and ocean remote sensing image information, and generates cross-modal features. , used for subsequent graph modeling.

[0123] The semantic reasoning module is used to construct a semantic graph and refine the graph semantic information, and output a prediction classification head.

[0124] The functions of each module and the data processing process can be found in the various steps of the position and semantic optimization method for remote sensing visual question answering described above, which will not be repeated here.

[0125] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for position and semantic optimization for remote sensing visual question answering, characterized in that: It is divided into text stream and ocean remote sensing image stream, including the following steps: Step S1, inputting ocean remote sensing images and text; Step S2, cross-modal context fusion: Integrate text and ocean remote sensing image information through the cross-modal position constrained attention mechanism to generate cross-modal features; the details are as follows: Step S21, single mode characterization: The marine remote sensing image and text data are respectively extracted through the image encoder and text encoder to extract image and text features, represented as X i , then X i Through the Transformer-based multi-head attention mechanism and residual mechanism encoding, the unimodal representation X′ is obtained i , where i represents the modality, and the image modality is X′ a Indicates that the text mode adopts X′ t express; S22. Multimodal Fusion: For the image modality representation X′ obtained in step S21 a and the text modality representation X′ t , weighted constraints are performed through the cross-modal position constraint attention mechanism: First, X′ a and X′ t The weighted representation is then weighted by LCA weighted by the weighted attention constraint of the category position information, and the constrained visual representation and text representation are output. The two are added to obtain the constrained multimodal representation X c , X c and image modality representation X′ a , text modality representation X′ t Fusion is performed to obtain the cross-modal feature X′ c , used for subsequent graph modeling; The cross-modal position constrained attention mechanism is implemented by cross-attention of image and text and weighted constrained attention LCA of category position information, wherein the weighted constrained attention LCA of category position information is obtained by computing category information SA and position information RP of the remote sensing image. Specifically, the category information SA and position information RP of the remote sensing image are element-wise multiplied with the weight matrix of the traditional attention mechanism to obtain the final attention weight matrix as the weighted constraint of category position information; The category information SA is determined by the strength of association between the image category and the entity category in the image, and the position information RP is determined by the relative position of the image region; Step S3: semantic reasoning: First, the cross-modal feature X′ obtained in step S2 c The semantic graph Gs is obtained by graph modeling through the relational graph convolution network; then, the semantic encoding is calculated. and relative position encoding And input it into the semantic graph Transformer, extract the semantic information in the semantic graph Gs through the semantic graph Transformer, and obtain the final refined semantic information; finally, output the prediction classification head to get the prediction answer.

2. The position and semantic optimization method for remote sensing visual question answering according to claim 1, characterized in that: Step S21 is as follows: i The attention map of the attention weights of a single attention head is obtained through the attention mechanism of Transformer, and the weighted sum is calculated using these attention weights to obtain the output; finally, a residual connection and a layer norm are added to obtain the unimodal representation X′ i ; Where i represents the sequence number of the i-th input feature, represent the query, key, and value vectors of the h-th attention head, which are composed of the input features X i It is obtained through linear transformation. Softmax is the softmax function used to convert the attention weight matrix into a probability distribution to ensure that the sum of all weights is 1. K represents the dimension of the key vector matrix. Used to scale the key vector matrix; represents the weighted sum of the outputs of the hth attention head, N represents the total number of multi-head attentions, Represent the weighted sum of the outputs of the 1st, 2nd, ..., Nth attention heads, respectively. represents an element-wise addition operation, and W is a learnable weight matrix used to map the concatenated matrix to the original feature dimension.

3. The position and semantic optimization method for remote sensing visual question answering according to claim 1, characterized in that: In step S22, specifically: The category information SA is used to focus on the category semantics of the current remote sensing image and simulate the semantic inertia of the category information on the current remote sensing image: Among them, s m is the image category of the current remote sensing image, s n is the entity category in the remote sensing image, i.e., class information, SA m，n It is the strength of association between the image category and entity category of the current remote sensing image. A value of 1 means that the representation remains unchanged, and a value of 0 means that it is discarded. The position information RP is represented by calculating the visual distance between the image regions um and un. If the visual distance is short, the semantic metric RP value is large, indicating that the relative importance between the image regions um and un is high.

4. The method for position and semantic optimization for remote sensing visual question answering according to claim 3, characterized in that: In step S22, X′ a and X′ t The cross-attention weighting is to simulate the interaction between the two modalities by cross-attention of the two streams of image and text. The cross-attention combines the multi-modal image and text information and transforms the text features using the representation of the image modality. For a single attention head, the cross-modal position-constrained attention mechanism is expressed as follows: O ij (Q,K)=(W ij ☉LCA)·V j (4); Among them, W ij represents the weight matrix in the traditional Transformer, LCA represents the weighted constraint attention of category position information, which is an attention matrix constrained by category information and position information, used to emphasize local context information, V j represents the jth value vector, O ij (Q, K) represents the output value corresponding to the i-th query vector Q and the j-th key vector K after calculation by the cross-modal position constrained attention mechanism; ⊙ represents the element-by-element multiplication operator, which combines the traditional Transformer attention weights and the local constrained attention weights; · represents the matrix multiplication operator, which multiplies the final weight with the value vector to obtain the output value.

5. The method for position and semantic optimization for remote sensing visual question answering according to claim 1, characterized in that: Step S3 is as follows: Step S31: construct a semantic graph: Cross-modal feature X′ c The semantic graph Gs is constructed through the relational graph convolutional network, which maps text and images to an entity space as points. The graph structure of the semantic graph includes points and edges. Points represent the semantics of image entities and text entities; edges represent the interactive relationship between entities, including the relationship between text and text, the relationship between text and image, the relationship between text and multimodal representation, and the relationship between multimodal representation and image. Step S32: semantic information refinement: Compute semantic encoding through entity feature vector and relative position encoding Input into the semantic graph Transformer to obtain the final refined semantic information; Step S33: predict the classification head.

6. The method for position and semantic optimization for remote sensing visual question answering according to claim 5, characterized in that: Step S31 is specifically as follows: Define a semantic graph G s =(V s , E s ), each entity is represented by a node, different connecting edges represent directed relationships, Vs represents a set of entity nodes, A set of relations representing the semantic similarity between entities is defined as formula (5), Among them, sim m，n represents the semantic similarity between entity m and entity n, g m and g n They represent the feature vectors of entity m and entity n respectively, T represents transpose, || || represents the Euclidean norm, that is, the length of the vector; arccos is the inverse cosine function, which is used to convert the cosine value into an angle value.

7. The method for position and semantic optimization for remote sensing visual question answering according to claim 6, characterized in that: Step S32 is specifically as follows: Step S321, determine semantic and position coding information: Given a semantic graph Gs with node features G = [g1, g2, ..., g n ], g1, g2, ..., g n Represent the 1st, 2nd, ..., nth node features of the semantic graph respectively; Define two codes to represent the semantic relationship between two nodes in the semantic graph Gs. The first is the relative position code Represents the relative position information of the entity in the semantic graph Gs, Each vector in represents the distance between nodes and is normalized by a sigmoid function to be in the range of [0,1]. The second is the semantic encoding The sim calculated by formula (10) m，n To represent, it is obtained by calculating the cosine similarity of the two node features and converting it into a value in the range of [0,1]; Each vector in represents the topological relationship represented by the shortest path distance between two nodes. and Add the two together to get the semantic position encoding Use a vector to represent the shortest path distance between two nodes, thereby reflecting the semantic similarity between them; Step S322: The semantic graph Transformer encodes node features and semantic positions Combined with the semantic graph transformer, semantic information is extracted from the node features to obtain the refined semantic representation, i.e. the updated node feature h′ i , specifically: The node features of the semantic graph Gs are converted into the query vector Q, key vector K, and value vector V of the attention mechanism, and the semantic similarity and attention scores between nodes are calculated and compared with the semantic position encoding. Combine and calculate h′ i , the formula is as follows: In formula (6), a ij represents the attention score between node i and node j, W q and W k is a learnable weight matrix used to transform node features g i and g j Converted to query and key vectors, d is a predefined dimension used to normalize the attention scores; is the semantic and position information item of node ij, indicating the semantic similarity between node i and node j, (W q h i ) T (W k h j ) represents the calculation of the query vector q i and key vector k j The dot product between them is used to measure the similarity between node i and node j; In formula (8), h′ i represents the updated node features, v j represents the feature vector of node j, represents the normalized attention score, G N represents the number of adjacent nodes, Manipulate the feature vectors representing neighboring nodes, adding positional encoding and semantic information.

8. Position and semantic optimization system for remote sensing visual question answering, characterized by: The method for implementing the position and semantic optimization method for remote sensing visual question answering as described in any one of claims 1 to 7, wherein the system adopts a dual-tower network structure, which is a text stream and an ocean remote sensing image stream, respectively, and includes a single-modal representation module, a cross-modal position constraint module and a semantic reasoning module. The unimodal representation module includes a visual embedding block and a text embedding block, each including an encoder, a Transformer-based multi-head attention mechanism and a residual mechanism, and the encoder includes an image encoder and a text encoder, which are used to extract image and text features X of remote sensing images and text data, respectively. i , X i After processing by the Transformer-based multi-head attention mechanism and residual mechanism, the unimodal representation X′ of the image and text is output i ; The cross-modal position constraint module uses the cross-modal position constraint attention mechanism to perform weighted constraints and fusion on the image modality representation and the text modality representation, integrates the text and ocean remote sensing image information, and generates the cross-modal feature X′ c , used for subsequent graph modeling; The semantic reasoning module is used to construct a semantic graph and refine the graph semantic information, and output a prediction classification head.

Images