A method for visual question answering in ocean remote sensing based on high-level knowledge enhancement

Abstract

The present invention belongs to the technical field of visual question answering, and discloses a marine remote sensing visual question answering method based on high-order knowledge enhancement, including: step S1, extracting unimodal features; step S2, extracting multimodal features: using cross-modal local constrained attention to generate high-order cross-modal features, and inputting attention selection blocks to obtain multimodal features; step S3, extracting knowledge-enhanced entity association representation: identifying visual and text entities in image-question pairs through knowledge-enhanced entity association modules, and using knowledge relevance reasoning strategies to learn knowledge-enhanced entity representations, and then fusing entities to obtain knowledge-enhanced entity association representations; step S4, generating visual question answering results: connecting the multimodal features generated in step S2 with the knowledge-enhanced entity association representations generated in step S3 to form a representation of generated answers. The present invention can effectively extract contextual image-question pair information and generate accurate visual question answering results.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual question answering, and in particular relates to an ocean remote sensing visual question answering method based on high-order knowledge enhancement. Background Art

[0002] Marine remote sensing images have the characteristics of all-day, all-weather, and all-round, and are widely used in important fields such as military reconnaissance, maritime traffic control, marine pollution monitoring, and resource exploration. As one of the important means of processing and analyzing marine remote sensing images, visual question answering is to answer the corresponding questions based on the image, which mainly includes two important steps: First, feature extraction of remote sensing images and texts, and mining their deep / high-level / semantic representations. Second, predict the answer to the question based on the extracted deep / high-level / semantic representations. The cutting-edge visual question answering network uses contextual information mining methods such as multi-scale, attention mechanism, and graph convolutional network to adaptively model the connection between natural language text and remote sensing visual images, and improve the multimodal feature representation capabilities. However, the above remote sensing visual question answering methods have the following problems when used in marine remote sensing images:

[0003] First, the spatial layout of high-order semantics in marine remote sensing images is seriously unbalanced, which is mainly reflected in the imbalance of high-order semantic features. Existing methods face two problems in high-order semantic modeling. On the one hand, errors are generated and there is a lot of noise in the modeled high-order features. On the other hand, in the modeling process of the marine remote sensing question-answering system, its wider and smaller areas will cause the signal-to-noise ratio of the target signal to be low, affecting the multimodal feature fusion effect. The reasons for the above two problems are that there is too much ambiguous noise, including confusion of high-order features of remote sensing imaging, which makes it impossible to model the best visual question-answering model for marine remote sensing images.

[0004] Second, ocean remote sensing images contain a lot of complex noise. Existing visual question answering methods perform feature extraction, context mining, deconvolution and other operations on the original image to generate feature representations, but these feature representations can only describe low-order information, such as color, shape, area, etc., and do not focus on introducing high-order information from external knowledge graphs. However, when encountering complex noise, low-order features such as image color and shape will not be stable, and complex noise cannot be adaptively eliminated / mitigated, and the impact of noise may even be amplified, reducing the accuracy of visual question answering.

[0005] Third, there is a semantic gap between marine remote sensing images and natural language. Existing visual question-answering methods generate feature representations by performing operations such as feature extraction, context mining, and deconvolution on the original image, and directly fuse them with text features through cascading, gating, or attention mechanisms. However, these multimodal feature representations themselves have a semantic gap problem. In the absence of knowledge graph association, the multimodal information interaction, alignment, and linking process can easily lead to the non-robust nature of representation, which further widens the semantic gap between marine remote sensing visual features and text features, and reduces the accuracy of visual question answering. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides an ocean remote sensing visual question answering method based on high-order knowledge enhancement.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A marine remote sensing visual question answering method based on high-order knowledge enhancement includes the following steps:

[0009] Step S1: Extracting unimodal features:

[0010] Input image-question pair, first use backbone network to encode visual features and text features, pass the visual feature and text feature sequences to visual and text embedding blocks respectively, and output unimodal visual embedding and text embedding to obtain unimodal representation;

[0011] Step S2: extract multimodal features:

[0012] The visual embedding and text embedding obtained in step S1 are input into the cross-modal context fusion module, and high-order cross-modal features are generated using cross-modal local constrained attention. The high-order cross-modal features are input into the attention selection block to obtain multi-modal features.

[0013] Step S3: Extract knowledge to enhance entity association representation:

[0014] The knowledge-enhanced entity association module identifies visual and textual entities in image-question pairs and adopts a knowledge relevance reasoning strategy to learn knowledge-enhanced entity representations, measure the semantic relevance distance of each pair of entities, and select the top k consistent or inconsistent entity pairs, which are further fused through a signed attention mechanism to obtain consistent or inconsistent knowledge-enhanced entity association representations;

[0015] Step S4: Generate visual question answering results:

[0016] The multimodal features generated in step S2 are connected with the knowledge-enhanced entity association representation generated in step S3 to form a representation of the generated answer.

[0017] Furthermore, in step 1, the backbone network is a dual-stream structure, including a text encoder and a visual encoder, which are used to extract basic features from the text T and the image V to obtain text feature sequence and visual feature sequence layers respectively.

[0018] Furthermore, given the visual question answering where N is the number of image-question pairs, are the 1st, 2nd, ..., m, ..., Nth image-question pairs in visual question answering D. Each image-question pair involves two data sources, corresponding to the visual modality and the textual modality, respectively, and is expressed as ,in Represents an image, Represents text, and represents the feature dimension, v represents vision, t represents text, and the combined input feature matrix of all image-question pairs in visual question answering is ,in , representing two characteristic modes, are the combined input feature matrices of the 1st, 2nd, ..., Nth image-question pairs in visual question answering D.

[0019] Furthermore, in step S1, the visual feature and text feature sequences are passed to the visual and text embedding blocks respectively, and the specific steps of obtaining the unimodal representation are as follows:

[0020] First, the triangle position is embedded in PE and added directly to X i , and defines the encoding for the input feature X i Query Q i , key K i Sum value V i vector, a single attention head is obtained through the attention mechanism Attention map of the attention weights of a single attention head The query is , key is , the value is Vector, h is the serial number of the attention head, which is used to calculate the weighted sum of the values ​​of a single attention head:

[0021] ;

[0022] Where k is the number of heads in the self-attention mechanism, is the weighted sum of the values;

[0023] Next, the outputs of multiple attention heads are concatenated and linearly transformed to obtain the final output:

[0024] ;

[0025] in, Indicates cascade operation, is a weight matrix, A is the total number of attention heads, , ,……, It is the 1st, 2nd, ..., Ath attention head;

[0026] Finally, adding residual connections and layer normalization, we get .

[0027] Furthermore, in step S2, the single-mode representation Input into the cross-modal context fusion module, where , represents two feature modalities, and uses the cross-modal local constrained attention module to generate high-order cross-modal features , and then the cross-modal features Input into an attention selection block to obtain multimodal feature representation ;

[0028] The cross-modal local constrained attention module includes a cross-modal local constrained transformer block and a cross-modal local constrained attention layer. After the unimodal transformer, after obtaining the global temporal dependency of the unimodal features, the cross-modal local constrained transformer block is applied to obtain local context information, focusing on the interaction between different modalities, wherein each cross-modal local constrained transformer block is combined with the cross-modal local constrained attention layer;

[0029] The cross-modal local constrained attention layer integrates information from different data sources and calculates the correlation or weight between text features and visual features to determine the importance of text or visual features in the conversion process;

[0030] The query, key and value of the visual feature are defined as , , , For visual input, , , are the weight matrices for visual feature queries, keys, and values ​​respectively; the query, keys, and values ​​for text features are defined as , , , For text input, , , are the weight matrices of text feature query, key, and value respectively.

[0031] Furthermore, cross-modal local constrained attention is used to assign weights to image-question pairs with different question categories and relative positions in the context, including question category weights QC and relative position weights RP. Specifically:

[0032] Corresponding weights are assigned to question categories to focus on questions of the same category as the question currently being asked and to model questions of different categories. The question category weights are expressed as follows:

[0033] ;

[0034] in and are image-question pairs and is the question category, α is the parameter value of the question category weight;

[0035] For the relative position weight RP, we calculated and The relative position weight is expressed as follows:

[0036] ;

[0037] Where N is the actual number of visual question-answer pairs, c and d are the image-question pairs and The relative position weight RP is input into the sigmoid function, and finally it is element-wise multiplied with the question category weight QC to obtain the cross-modal local constrained attention LCA;

[0038] Weighted sum of weight mapping of visual features to text features for:

[0039] ;

[0040] in is the weight matrix;

[0041] Weighted sum of text features and weight mapping of visual features for:

[0042] ;

[0043] in is the weight matrix.

[0044] Furthermore, in step S2, the attention selection block fuses the features of the two modalities through a model-level fusion strategy, assigns different importance to different modalities, and combines the internal and cross-modal context information of the two modalities to obtain a multimodal feature representation. Specifically:

[0045] Extracting image-question pairs Visual features , text features , cross-modal features ,in is the mth image-question pair in visual question answering; then, by equalizing the feature dimensions of all inputs and concatenating them together;

[0046] At a given time, for the input features , where K is the number of modes, Represents the input features of different modes, and the input features Perform an affine transformation, Represents the modal number, and performs nonlinear mapping through the ReLU activation function to generate an activation value :

[0047] ;

[0048] Affine transformation is to linearly combine the input features through the weight matrix and add the bias; calculate the attention score of each modality :

[0049] ;

[0050] Attention scores for each modality , where K=3, the final multimodal features The generation method is as follows:

[0051] ;

[0052] in, is the attention score of each modality.

[0053] Furthermore, in step S3, the specific process of the knowledge enhancement entity association module is as follows:

[0054] Step S31, entity recognition and linking: identifying visual and textual entities from images and texts, and then linking them together to generate a cross-modal entity pair set and two intra-modal entity pair sets;

[0055] Step S32, semantic relevance distance measurement: measuring the semantic relevance distance of each entity pair in the three entity pair sets by using a knowledge relevance reasoning strategy, and constructing a knowledge-enhanced entity representation;

[0056] Step S33, modeling knowledge-enhanced entity associations: select the top k semantically consistent and inconsistent entity pairs from each set, and apply multimodal features from the cross-modal context fusion module The positive or negative weighted attention of each entity is used to obtain the respective knowledge and enhance the entity association.

[0057] Further, in step S32, define is a set of text-text entity pairs, is a set of image-image entity pairs, is a set of text-image entity pairs, for , or Any set of entity pairs ,in and are the two entities in an entity pair, measured by To measure the semantic relevance distance between two entities on the pre-trained knowledge graph; measurement method Modeling semantic relevance using feature distance in vector space and graph distance on knowledge graph topology;

[0058] in The calculation method is as follows:

[0059] (1) First find the connection in the knowledge graph and The shortest semantically relevant path ;

[0060] (2) After obtaining the shortest semantically relevant path Then, the entity is Knowledge-enhanced entity representation To refine:

[0061] ;

[0062] in Is an entity in the path The feature embedding of is the weight coefficient, ρ is the entity number;

[0063] Symmetrically, for the entity Representation The details are as follows:

[0064] ;

[0065] (3) Semantic relevance distance is calculated as and The Euclidean distance between:

[0066] ;

[0067] Features to be connected As a pair Knowledge-enhanced entity representation.

[0068] Further, in step S33, the high-order entity relevance is obtained by selecting the top k most relevant or irrelevant entity pairs and applying positive or negative signed connection attention with multimodal features to model the high-order knowledge enhanced entity association in each entity pair set;

[0069] by This process is described in detail using an example to learn knowledge-enhanced entity associations between image and text modalities, as follows:

[0070] exist Two subsets are constructed in , which contain the top k most related entity pairs or the top k least related entity pairs respectively. This is done by The entity pairs in the The k entity pairs with the smallest distance are selected as the associated subset. , the k entity pairs with the largest distance are selected as irrelevant subsets ;

[0071] use express The semantic relevance distance of entity pairs in , Indicates the sequence number of the entity pair, using represents the corresponding knowledge-enhanced entity representation; express The semantic relevance distance of entity pairs in , represents the corresponding knowledge-enhanced entity representation;

[0072] The knowledge-enhanced entity association module uses the signed attention mechanism to further integrate multimodal features. , the positive sign indicates consistency, indicating that the entities have similar linguistic features or relations; the negative sign indicates inconsistency, indicating that there are dissimilar or opposite semantic features or relations between entities; specifically, the knowledge-enhanced entity association module uses positive attention to capture the consistent association between image and question; it combines multimodal features As a query, semantically related knowledge is added to entity representation As keys and values, the consistent association is calculated as follows:

[0073] ;

[0074] ;

[0075] in yes The dimension of Represents the positive attention coefficient, using Reweighting coefficients to incorporate semantically related distances into consistent representations ;

[0076] At the same time, the knowledge-enhanced entity association module uses negative attention to obtain inconsistent representations. , as shown below:

[0077] ;

[0078] ;

[0079] Then it will be expressed and Connect to form a set of entity pairs Knowledge-enhanced entity association representation ;

[0080] Similarly, the knowledge-enhanced entity association module uses the same mechanism to obtain and Relationship and , these representations are used for the final answer generation.

[0081] Compared with the prior art, the present invention has the following advantages:

[0082] First, make full use of the background semantic relevance contained in the external knowledge graph. The present invention constructs a knowledge-enhanced entity association module to identify visual and textual entities in image-question pairs, and designs a knowledge relevance reasoning strategy to find the shortest semantically relevant path between each pair of entities in the external knowledge graph and absorb all complementary contextual knowledge connecting the entities in the path to learn knowledge-enhanced entity representations. Then the semantic relevance distance of each pair of entities is measured, and the top k consistent / inconsistent entity pairs are selected, which are further fused through the signed attention mechanism to obtain consistent / inconsistent knowledge-enhanced entity associations.

[0083] Second, the injection of high-order knowledge of categories is introduced. The present invention introduces the injection of high-order knowledge of categories by introducing cross-modal local constrained attention in the cross-modal context fusion module. First, the question category is masked, and the context information with the same category as the current question is given a higher weight. Secondly, the weight is assigned according to the relative position of the context information in the fixed window, and the weighted multiplication is performed with the question category mask to obtain the weight of the question category in the fixed window. The full injection of high-order knowledge of categories can effectively extract contextual image-question pair information, enhance the robustness of the model, and generate accurate visual question and answer results. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0085] Figure 1 It is a flowchart of the method of the present invention;

[0086] Figure 2 It is a schematic diagram of the single-mode embedding of the present invention;

[0087] Figure 3 Schematic diagram of a cross-modal context fusion module of the present invention;

[0088] Figure 4 Schematic diagram of cross-modal local constrained attention LCA of the present invention;

[0089] Figure 5 Schematic diagram of the knowledge enhancement entity association module of the present invention. DETAILED DESCRIPTION

[0090] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0091] like Figure 1-Figure 5 As shown, this embodiment provides an ocean remote sensing visual question answering method based on high-order knowledge enhancement, comprising the following steps:

[0092] Step S1: Extracting unimodal features:

[0093] Input image-question pairs, for example, the image-question pairs in the dataset input in this embodiment are Figure 1 The input image (remote sensing image) in the image and "How many residential buildings are there". The visual entities in the remote sensing image include buildings, cars, roads, houses, etc. "How many residential buildings are there" is one of the questions in the remote sensing visual question answering dataset.

[0094] First, the backbone network is used to encode visual features and text features, and the visual feature and text feature sequences are passed to the visual and text embedding blocks respectively, and the unimodal visual embedding and text embedding are output to obtain unimodal representation.

[0095] The backbone network is a dual-stream structure, including a text encoder and a visual encoder, which is used to extract basic features from text T and image V;

[0096] Text encoder: Specifically, the pre-trained BERT model is used to map the word sequence of text T into an embedding sequence of length L , β is the sequence number of the word in the sequence, the embedded sequence is converted to Bi-LSTM by a bidirectional long short-term memory network, and processed by a linear layer to obtain a text feature sequence , specifically expressed as follows:

[0097] ;

[0098] in, and represents the learnable parameters of the fully connected layer;

[0099] Visual encoder: The ResNet-50 model is used to encode the image V into an initial feature map with a size of 2048*7*7, which is then converted into a visual feature sequence through convolution blocks and reshaping operations. , γ is the number of layers of the neural network; the specific expression is as follows:

[0100] ;

[0101] Among them, CNN represents the ResNet-50 model, Conv represents the convolutional layer, BN represents the batch normalization layer, and ReLU represents the rectified linear unit layer.

[0102] Given a visual question answering where N is the number of image-question pairs, are the 1st, 2nd, ..., m, ..., Nth image-question pairs in visual question answering D. Each image-question pair involves two data sources, corresponding to the visual modality and the textual modality, respectively, and is expressed as ,in Represents an image, Represents text, and represents the feature dimension, v represents vision, t represents text, and the combined input feature matrix of all image-question pairs in visual question answering is ,in , representing two characteristic modes, are the combined input feature matrices of the 1st, 2nd, ..., Nth image-question pairs in visual question answering D.

[0103] As a preferred implementation, in step S1, the visual feature and text feature sequences are passed to the visual and text embedding blocks respectively, and the specific steps of obtaining the unimodal representation are as follows:

[0104] First, the triangle position is embedded in PE and added directly to X i , and defines the encoding for the input feature X i Query Q i , key K i Sum value Vi vector, a single attention head is obtained through the attention mechanism Attention map of the attention weights of a single attention head The query is , key is , the value is Vector, h is the serial number of the attention head, which is used to calculate the weighted sum of the values ​​of a single attention head:

[0105] ;

[0106] Where k is the number of heads in the self-attention mechanism, is the weighted sum of the values;

[0107] Then, through the multi-head attention mechanism, the outputs of multiple attention heads are cascaded and linearly transformed to obtain the final output:

[0108] ;

[0109] in, Indicates cascade operation, is a weight matrix, d i is the dimension of the input, A is the total number of attention heads, k is the dimension of each head, , ,……, It is the 1st, 2nd, ..., Ath attention head;

[0110] Finally, the data is output through a feedforward neural network, and residual connections and layer normalization are added to obtain .

[0111] Among them, the feedforward neural network is a basic neural network structure, which is characterized by the fact that information can only flow from the input layer to the output layer without a feedback loop.

[0112] Residual connections help alleviate the problem of gradient vanishing and information transfer by adding cross-layer connections in the network, allowing information to be directly transferred between layers. Specifically, residual connections add the output of the previous layer to the input of the current layer to form a direct connection across layers. This makes it easier for the network to learn the residual (the difference between the current layer and the previous layer) rather than directly learning the mapping function itself.

[0113] Step S2: extract multimodal features:

[0114] The visual embedding and text embedding obtained in step S1 are input into the cross-modal context fusion module, and high-order cross-modal features are generated using cross-modal local constrained attention. The features are then input into the attention selection block to obtain multi-modal features.

[0115] In step S2, the single modal representation Input into the cross-modal context fusion module, where , represents two feature modalities, and uses the cross-modal local constrained attention module to generate high-order cross-modal features , and then the cross-modal features Input into an attention selection block to obtain multimodal feature representation .

[0116] The cross-modal local constrained attention module includes a cross-modal local constrained transformer block and a cross-modal local constrained attention layer. After the unimodal transformer, after obtaining the global temporal dependency of the unimodal features, the cross-modal local constrained transformer block is applied to obtain local context information, focusing on the interaction between different modalities, wherein each cross-modal local constrained transformer block is combined with the cross-modal local constrained attention layer;

[0117] The cross-modal local constrained attention layer integrates information from different data sources and calculates the correlation or weight between text features and visual features to determine the importance of text or visual features in the conversion process.

[0118] The query, key and value of the visual feature are defined as , , , For visual input, , , are the weight matrices for visual feature queries, keys, and values ​​respectively; the query, keys, and values ​​for text features are defined as , , , For text input, , , are the weight matrices of text feature query, key, and value respectively.

[0119] As a preferred implementation, weights are assigned to image-question pairs with different question categories and relative positions in the context through cross-modal local constrained attention, including question category weights QC and relative position weights RP. Specifically:

[0120] Corresponding weights are assigned to question categories to focus on questions of the same category as the question currently being asked and to model questions of different categories. The question category weights are expressed as follows:

[0121] ;

[0122] in and are image-question pairs and is the question category, α is the parameter value of the question category weight; the question category is the four types of questions in the remote sensing VQA dataset, existence / non-existence, urban / rural, comparison and counting questions. and If they are the same, the corresponding question has a higher weight of α, and if they are different, it is 1-α. In order to make the weight of the same category of questions higher, α is a number between 0.5 and 1;

[0123] For the relative position weight RP, we calculated and The relative position weight is expressed as follows:

[0124] ;

[0125] Where N is the actual number of visual question-answer pairs, c and d are the image-question pairs and The relative position weight RP is then input into the sigmoid function, and finally it is element-wise multiplied with the problem category weight QC to obtain:

[0126] .

[0127] After the relative position weight RP is processed by the sigmoid function, it will be mapped to a value ranging from 0 to 1, and then multiplied element by element with the question category weight ranging from 0 to 1 to emphasize or weaken the influence of the local image-question pair, combining the local context.

[0128] Weighted sum of weight mapping of visual features to text features for:

[0129]

[0130] in is the weight matrix;

[0131] Weighted sum of text features and weight mapping of visual features for:

[0132] ;

[0133] in is the weight matrix.

[0134] As a preferred implementation, in step S2, the attention selection block fuses the features of the two modalities through a model-level fusion strategy, assigns different importance to different modalities, and combines the internal and cross-modal context information of the two modalities to obtain a multimodal feature representation. Specifically:

[0135] Extracting image-question pairs Visual features , text features , cross-modal features ,in is the mth image-question pair in visual question answering; then, by equalizing the feature dimensions of all inputs and concatenating them together, the different contributions of different modalities to focusing on the important modalities are considered.

[0136] At a given time, for the input features , where K is the number of modes, Represents the input features of different modes, and the input features Perform an affine transformation, Represents the modal number, and performs nonlinear mapping through the ReLU activation function to generate an activation value :

[0137] ;

[0138] Affine transformation is to linearly combine the input features through the weight matrix and add the bias; calculate the attention score of each modality :

[0139] ;

[0140] Attention scores for each modality , where K=3, the final multimodal features The generation method is as follows:

[0141] ;

[0142] in, is the attention score of each modality.

[0143] Step S3: Extract knowledge to enhance entity association representation:

[0144] The knowledge-enhanced entity association module identifies visual and textual entities in image-question pairs, and adopts a knowledge relevance reasoning strategy to learn knowledge-enhanced entity representations, measure the semantic relevance distance of each pair of entities, and select the top k consistent or inconsistent entity pairs, which are further fused through a signed attention mechanism to obtain consistent or inconsistent knowledge-enhanced entity association representations.

[0145] The specific process of the knowledge enhancement entity association module is combined with Figure 5 For convenience, Figure 5 X and Y in the figure refer to the image V and text T of the present invention, and the entity pair set E used in the present invention XY Including E TT 、E TV 、E VV , as follows:

[0146] Step S31, entity recognition and linking: Identify visual and textual entities from images and texts, and then link them together to generate a cross-modal entity pair set (E TV ) and two sets of internal modal entity pairs (E TT and E VV ).

[0147] Step S32, semantic relevance distance measurement: measure the semantic relevance distance of each entity pair in the three entity pair sets through the knowledge relevance reasoning strategy, and construct a knowledge-enhanced entity representation.

[0148] In step S32, define is a set of text-text entity pairs, is a set of image-image entity pairs, is a set of text-image entity pairs, for , or Any set of entity pairs ,in and are the two entities in an entity pair, measured by To measure the semantic relevance distance between two entities on the pre-trained knowledge graph; measurement method Semantic relevance is modeled using feature distance in vector space and graph distance on knowledge graph topology.

[0149] in The calculation method is as follows:

[0150] (1) First find the connection in the knowledge graph and The shortest semantically relevant path , which is expressed as follows:

[0151] ;

[0152] Where m represents The number of entities in Is the path All entities in;

[0153] (2) After obtaining the shortest semantically relevant path Then, the entity is Knowledge-enhanced entity representation To refine:

[0154] ;

[0155] in Is an entity in the path The feature embedding of is the weight coefficient, ρ is the entity number; is a path-aware representation that incorporates the contextual knowledge of entities, where weights decrease exponentially to weaken their contribution as the graph distance increases.

[0156] Symmetrically, for the entity Representation The details are as follows:

[0157] ;

[0158] (3) Semantic relevance distance is calculated as and The Euclidean distance between:

[0159] ;

[0160] Features to be connected As a pair Knowledge-enhanced entity representation.

[0161] Step S33, modeling knowledge-enhanced entity associations: select the top k semantically consistent and inconsistent entity pairs (i.e., the most semantically relevant and least semantically relevant entity pairs) from each set, and apply the multimodal features from the cross-modal context fusion module The positive or negative weighted attention of each entity is used to obtain the respective knowledge and enhance the entity association.

[0162] In step S33, the top k most relevant or irrelevant entity pairs are selected, and the high-order knowledge-enhanced entity associations in each entity pair set are modeled by applying positive or negative signed connection attention with multimodal features to obtain high-order entity relevance.

[0163] That is, in E XY The two subsets constructed in are the associated subset E XY, re and the irrelevant subset E XY, ir , the knowledge-enhanced entity association module uses positive attention to obtain consistent representation , using negative attention to obtain inconsistent representations ,Will and Connect them to form a set of entity pairs E XY Knowledge-enhanced entity association representation ,like Figure 5 shown.

[0164] by This process is described in detail using an example to learn knowledge-enhanced entity associations between image and text modalities, as follows:

[0165] exist Two subsets are constructed in , which contain the top k most related entity pairs or the top k least related entity pairs respectively. This is done by The entity pairs in the The k entity pairs with the smallest distance are selected as the associated subset. , the k entity pairs with the largest distance are selected as irrelevant subsets .

[0166] To explain more clearly, express The semantic relevance distance of entity pairs in , Indicates the sequence number of the entity pair, using represents the corresponding knowledge-enhanced entity representation; express The semantic relevance distance of entity pairs in , Represents the corresponding knowledge-enhanced entity representation.

[0167] In order to capture high-order consistent and inconsistent entity correlations simultaneously, the knowledge-enhanced entity association module further fuses multimodal features using a signed attention mechanism. , the signed attention mechanism is a variant of the attention mechanism that processes attention weights with positive and negative signs to capture the consistency and inconsistency between entities. The positive sign represents consistency, indicating that the entities have similar linguistic features or relations; the negative sign represents inconsistency, indicating that there are dissimilar or opposite semantic features or relations between entities. Specifically, the knowledge-enhanced entity association module adopts positive attention to capture the consistent association between image and question; it combines multimodal features As a query, semantically related knowledge is added to entity representation As keys and values, the consistent association is calculated as follows:

[0168] ;

[0169] ;

[0170] in yes The dimension of Represents the positive attention coefficient. A larger value indicates that the image and the entity in the question are more positively semantically related. Reweighting coefficients to incorporate semantically related distances into consistent representations Related entity pairs with shorter semantic correlation distances contribute more significantly to the learning of consistent associations.

[0171] At the same time, the knowledge-enhanced entity association module uses negative attention to obtain inconsistent representations. , as shown below:

[0172] ;

[0173] ;

[0174] Then it will be expressed and Connect to form a set of entity pairs Knowledge-enhanced entity association representation .

[0175] Similarly, the knowledge-enhanced entity association module uses the same mechanism to obtain and Relationship and , these representations are used for the final answer generation.

[0176] Step S4: Generate visual question answering results:

[0177] The multimodal features generated in step S2 are connected with the knowledge-enhanced entity association representation generated in step S3 to form a representation of the generated answer.

[0178] In summary, (1) the present invention introduces cross-modal local constrained attention into the cross-modal context fusion module and injects high-order knowledge according to the question category. First, different degrees of attention are given according to the question category. Secondly, a fixed window is applied, and the relative position weight of the context information in the window is calculated to solve the problem that the fixed window method treats the context information in the window equally. The question category is weighted and multiplied by the relative position to obtain the representation of enhanced high-order features.

[0179] (2) The present invention designs a knowledge-enhanced entity association module with a knowledge relevance reasoning strategy and a signed attention mechanism to obtain high-order knowledge-enhanced entity relevance. The knowledge relevance reasoning strategy can find the shortest semantic relevance distance between each pair of entities in the external knowledge graph and absorb all complementary contextual knowledge of the linked entities in the path to learn knowledge-enhanced entity representations. The signed attention mechanism can further fuse entity pairs to obtain both consistent and inconsistent knowledge-enhanced entity relevance.

[0180] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A marine remote sensing visual question answering method based on high-order knowledge enhancement, characterized in that: The following steps are involved: Step S1: Extracting unimodal features: Input image-question pair, first use backbone network to encode visual features and text features, pass the visual feature and text feature sequences to visual and text embedding blocks respectively, and output unimodal visual embedding and text embedding to obtain unimodal representation; Step S2: extract multimodal features: The visual embedding and text embedding obtained in step S1 are input into the cross-modal context fusion module, and high-order cross-modal features are generated using cross-modal local constrained attention. The high-order cross-modal features are input into the attention selection block to obtain multi-modal features. In step S2, the unimodal representation X′ i Input into the cross-modal context fusion module, where i∈{v, t} represents two feature modalities, and the cross-modal local constrained attention module is used to generate high-order cross-modal features I′ i , and then the cross-modal feature I′ i Input into an attention selection block to obtain a multimodal feature representation X c ; The cross-modal local constrained attention module includes a cross-modal local constrained transformer block and a cross-modal local constrained attention layer. After the unimodal transformer, after obtaining the global temporal dependency of the unimodal features, the cross-modal local constrained transformer block is applied to obtain local context information, focusing on the interaction between different modalities, wherein each cross-modal local constrained transformer block is combined with the cross-modal local constrained attention layer; The cross-modal local constrained attention layer integrates information from different data sources and calculates the correlation or weight between text features and visual features to determine the importance of text or visual features in the conversion process; Step S3: Extract knowledge to enhance entity association representation: The knowledge-enhanced entity association module identifies visual and textual entities in image-question pairs and adopts a knowledge relevance reasoning strategy to learn knowledge-enhanced entity representations, measure the semantic relevance distance of each pair of entities, and select the top k consistent or inconsistent entity pairs, which are further fused through a signed attention mechanism to obtain consistent or inconsistent knowledge-enhanced entity association representations; Step S4: Generate visual question answering results: The multimodal features generated in step S2 are connected with the knowledge-enhanced entity association representation generated in step S3 to form a representation of the generated answer.

2. The marine remote sensing visual question answering method based on high-order knowledge enhancement according to claim 1 is characterized in that: In step 1, the backbone network is a dual-stream structure, including a text encoder and a visual encoder, which are used to extract basic features from the text T and the image V to obtain text feature sequence and visual feature sequence layers respectively.

3. The marine remote sensing visual question answering method based on high-order knowledge enhancement according to claim 1 is characterized in that: Given a visual question answering D = {u1, u2, ... m …,u N }, where N is the number of image-question pairs, u1, u2, ...u m …,u N are the 1st, 2nd, ..., m, ..., Nth image-question pairs in visual question answering D. Each image-question pair involves two data sources, corresponding to the visual modality and the textual modality, respectively, and is expressed as in Represents an image, Indicates text, d v and d t represents the feature dimension, v represents vision, t represents text, and the combined input feature matrix of all image-question pairs in visual question answering is where i∈{v,t} represents two eigenmodes, are the combined input feature matrices of the 1st, 2nd, ..., Nth image-question pairs in visual question answering D.

4. The method for marine remote sensing visual question answering based on high-order knowledge enhancement according to claim 3 is characterized in that: In step S1, the visual feature and text feature sequences are passed to the visual and text embedding blocks respectively. The specific steps to obtain the unimodal representation are as follows: First, the triangle position is embedded in PE and added directly to X i , and defines the encoding for the input feature X i Query Q i , key K i Sum value V i vector, a single attention head α is obtained through the attention mechanism (h) The attention graph of the attention weights, the query of a single attention head is Key is Value is Vector, h is the serial number of the attention head, which is used to calculate the weighted sum of the values ​​of a single attention head: Where k is the number of heads in the self-attention mechanism, is the weighted sum of the values; Next, the outputs of multiple attention heads are concatenated and linearly transformed to obtain the final output: in, represents the cascade operation, W is a weight matrix, A is the total number of attention heads, It is the 1st, 2nd, ..., Ath attention head; Finally, add residual connections and layer normalization to get X′ i .

5. The marine remote sensing visual question answering method based on high-order knowledge enhancement according to claim 1 is characterized in that: In step S2, the query, key and value of the visual feature are defined as X′ v For visual input, are the weight matrices for visual feature queries, keys, and values ​​respectively; the query, keys, and values ​​for text features are defined as X′ t For text input, are the weight matrices of text feature query, key, and value respectively.

6. The marine remote sensing visual question answering method based on high-order knowledge enhancement according to claim 5 is characterized in that: Through cross-modal local constrained attention, weights are assigned to image-question pairs with different question categories and relative positions in the context, including question category weight QC and relative position weight RP. Specifically: Corresponding weights are assigned to question categories to focus on questions of the same category as the question currently being asked and to model questions of different categories. The question category weights are expressed as follows: where c m and c n are the image-question pairs u m and u n is the question category, α is the parameter value of the question category weight; For the relative position weight RP, u is calculated c and u d The relative position weight is expressed as follows: Where N is the actual number of visual question-answer pairs, c and d are the image-question pairs u and c and u d The relative position weight RP is input into the sigmoid function, and finally it is element-wise multiplied with the question category weight QC to obtain the cross-modal local constrained attention LCA; Weighted sum of visual features to text features weight mapping O vt (Q, K) is: O vt (Q,K)=(W vt ⊙LCA)·V t Where W vt =softmax(Q v K t ) is the weight matrix; The weighted sum of the weight mapping of text features to visual features is O tv (Q, K) is: O tv (Q,K)=(W tv ⊙LCA)·V v ; Where W tv =softmax(Q t K v ) is the weight matrix.

7. The method for marine remote sensing visual question answering based on high-order knowledge enhancement according to claim 1, characterized in that: In step S2, the attention selection block fuses the features of the two modalities through a model-level fusion strategy, assigns different importance to different modalities, and integrates the internal and cross-modal context information of the two modalities to obtain a multimodal feature representation X c ; Specifically: Extract image-problem pair u m Visual features Text Features Cross-modal features where u m is the mth image-question pair in visual question answering; then, by equalizing the feature dimensions of all inputs and concatenating them together; At a given time, for the input feature H = [H 1 , H 2 , ..., H K ], where K is the number of modes, H 1 , H 2 , ..., H K Represents the input features of different modes, and the input feature H θ Perform affine transformation, θ∈{1, 2…, K} represents the modal number, and perform nonlinear mapping through the ReLU activation function to generate the activation value a θ : a θ =ReLU(W T A θ +b); Affine transformation is to linearly combine the input features through the weight matrix and add a bias; Calculate the attention score α for each modality θ : The attention score α for each modality θ ∈R 1×K , where K = 3, the final multimodal feature X′ c The generation method is as follows: X′ c =concat([α1H 1 , …, α K H K ]); Among them, α1, ...α K is the attention score of each modality.

8. The marine remote sensing visual question answering method based on high-order knowledge enhancement according to claim 1 is characterized in that: In step S3, the specific process of the knowledge enhancement entity association module is as follows: Step S31, entity recognition and linking: identifying visual and textual entities from images and texts, and then linking them together to generate a cross-modal entity pair set and two intra-modal entity pair sets; Step S32, semantic relevance distance measurement: measuring the semantic relevance distance of each entity pair in the three entity pair sets by using a knowledge relevance reasoning strategy, and constructing a knowledge-enhanced entity representation; Step S33, modeling knowledge-enhanced entity association: select the top k semantically consistent and inconsistent entities from each set. By applying the multimodal features X′ from the cross-modal context fusion module c The positive or negative weighted attention of each entity is used to obtain the respective knowledge and enhance the entity association.

9. The marine remote sensing visual question answering method based on high-order knowledge enhancement according to claim 8 is characterized in that: In step S32, define E TT is the set of text-text entity pairs, E VV is the set of image-image entity pairs, E TV is a set of text-image entity pairs, for TT 、E VV or E TV Any set of entity pairs (E u , E v ), where E u and E v are two entities in an entity pair, measured by the metric D s (E u , E v ) to measure the semantic relevance distance between two entities on the pre-trained knowledge graph; the measurement method D s (E u , E v ) Modeling semantic relevance using feature distance in vector space and graph distance on knowledge graph topology; Where D s (E u , E v ) is calculated as follows: (1) First, find the connection E in the knowledge graph u and E v The shortest semantically relevant path π; (2) After obtaining the shortest semantically relevant path π, the entity E is queried in the following way. u Knowledge-enhanced entity representation h u To refine: in Is an entity in the path The feature embedding of , ∈ is the weight coefficient, ρ is the entity serial number; Symmetrically, for entity E v The expression h v The details are as follows: (3) Semantic relevance distance D s (E u , E v ) is calculated as h u and h v The Euclidean distance between: D s (E u ,E v )=||h u -h v ||2; The connected features [h u ;h v ] as a response to (E u , E v )’s knowledge-enhanced entity representation.

10. The marine remote sensing visual question answering method based on high-order knowledge enhancement according to claim 8, characterized in that: In step S33, the top k most relevant or irrelevant entity pairs are selected, and the high-order knowledge-enhanced entity associations in each entity pair set are modeled by applying positive or negative signed connection attention with multimodal features to obtain high-order entity relevance; TV This process is described in detail as an example to learn the knowledge-enhanced entity association between image and text modalities, as follows: TV Two subsets are constructed in E, which contain the top k most related entity pairs or the top k least related entity pairs respectively. This is achieved by replacing E TV The entity pairs in the s (E u , E v ) are sorted, and the k entity pairs with the smallest distance are selected as the associated subset E TV，re , the k entity pairs with the largest distance are selected as the irrelevant subset E TV，ir ;use Indicates E TV，re The semantic correlation distance of the entity pair in , σ represents the sequence number of the entity pair, and represents the corresponding knowledge-enhanced entity representation; Indicates E TV，re The semantic relevance distance of entity pairs in , represents the corresponding knowledge-enhanced entity representation; The knowledge-enhanced entity association module further integrates the multimodal features X′ using the signed attention mechanism. c , the positive sign indicates consistency, indicating that the entities have similar linguistic features or relations; the negative sign indicates inconsistency, indicating that there are dissimilar or opposite semantic features or relations between entities; specifically, the knowledge-enhanced entity association module uses positive attention to capture the consistent association between image and question; it converts the multimodal feature X′ c As a query, semantically related knowledge is added to entity representation As keys and values, the consistent association is calculated as follows: Among them 2d e yes The dimension of Represents the positive attention coefficient, using Reweighting coefficients to incorporate semantically related distances into consistent representations At the same time, the knowledge-enhanced entity association module uses negative attention to obtain inconsistent representations. As shown below: Then it will be expressed and Connect them to form a set of entity pairs E TV Knowledge-enhanced entity association representation Similarly, the knowledge-enhanced entity association module uses the same mechanism to obtain E TT and E VV The associated representation and These representations are used for the final answer generation.

Images