A visual question answering method and system based on a multi-level visual feature enhancement network

Abstract

The application provides a visual question answering method and system based on a multi-level visual feature enhancement network, which can enhance the relationship between local objects and local objects and the relationship between regional objects and global concepts, thereby jointly learning the visual semantic relationship of multiple spatial contexts. A separation visual feature module based on a graph attention network is used to capture pixel-level visual features and object-level regional features; a joint visual feature representation based on a graph attention network is used to jointly represent the pixel-level features and the object-level features, simultaneously learn the semantic relationship between different levels, better associate with the question text, and thus provide more rich visual feature representation. The application solves the problem that the traditional visual feature representation loses the context relationship between the regional features and the global features, so that the global semantic cannot be fully utilized, and the visual features are lost.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a visual question answering method and system based on a multi-level visual feature enhancement network. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Visual Question Answering (VQA) is a multimodal task that involves interacting with a given image and a corresponding natural language question to arrive at the final answer.

[0004] Most Visual Question Answering (VQA) models can be broadly divided into four modules: question encoding, image encoding, cross-modal fusion, and question prediction. The question encoding module typically uses models such as RNNs and LSTMs to embed the question into a vector. The image encoding module first extracts image features using a Faster R-CNN model, then adds or concatenates the question encoding with the image features for joint encoding and relationship modeling to learn the relationship between text and image, resulting in joint features. The cross-modal fusion module fuses the question encoding and joint features, and finally inputs this fusion into the question prediction module for answer prediction.

[0005] However, traditional visual question answering models typically employ top-down image feature representation based on regions, which causes regional features to lose contextual connection with global features. This results in the global semantic features of visual features not being fully utilized, thus affecting the accuracy of visual question answering results. Summary of the Invention

[0006] To address at least one of the technical problems in the background art, the present invention provides a visual question answering method and system based on a multi-level visual feature enhancement network. By enhancing the connections between regional features of an image and between regional features and global features, a more accurate visual feature representation is obtained, which is better associated with the question text, thereby improving the accuracy of the visual question answering results.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of the present invention provides a visual question answering method based on a multi-level visual feature enhancement network.

[0009] A visual question answering method based on a multi-level visual feature enhancement network includes:

[0010] Obtain the questions to be answered and the visual images;

[0011] The question to be answered is represented by features to obtain a text feature representation vector; the visual image is represented by features to obtain a visual feature representation vector.

[0012] Update the text feature representation vector and the visual feature representation vector;

[0013] The updated text feature representation vector and the visual feature representation vector are fused to obtain the fused feature vector;

[0014] The fused feature vector is first subjected to a linear transformation, and then the Sigmoid function is used to obtain the answer prediction result;

[0015] Among them, a multi-level visual feature enhancement network is used to represent the features of visual images.

[0016] Furthermore, visual features include two levels of image features: pixel-level visual features and object-level visual features; pixel-level visual features are global visual features, and object-level visual features are regional visual features.

[0017] Furthermore, the multi-level visual feature enhancement network consists of a graph attention-based separate visual feature representation network and a graph attention-based joint visual feature representation network.

[0018] Furthermore, the separate visual feature representation network is used to learn global visual features and regional visual features separately; the joint visual feature representation network is used to capture the semantic relationship between global visual features and regional visual features, and combines shallow detail features with deep semantic features to generate a complete visual feature representation vector.

[0019] Furthermore, the separated visual feature representation network designs two independent semantic relationship enhancement networks for two-level image features: an attention pixel relationship enhancement network and an attention object relationship enhancement network, which are used to learn enhanced global visual features and enhanced regional visual features.

[0020] Furthermore, the joint visual feature representation network includes a joint feature representation network and a joint feature fusion network.

[0021] Furthermore, the joint feature representation network incorporates the joint feature representation into the graph attention network to learn all semantic relationships between object-level and pixel-level objects; the joint feature fusion network employs a gating fusion mechanism to integrate information and obtain the final visual feature representation.

[0022] A second aspect of the present invention provides a visual question answering system based on a multi-level visual feature enhancement network.

[0023] A visual question answering system based on a multi-level visual feature enhancement network includes:

[0024] The question-and-answer acquisition module retrieves the questions to be answered and the visual images.

[0025] The feature representation module performs feature representation on the question to be answered, obtaining a text feature representation vector; and performs feature representation on the visual image, obtaining a visual feature representation vector.

[0026] The update module updates the text feature representation vector and the visual feature representation vector;

[0027] The fusion module fuses the updated text feature representation vector and the visual feature representation vector to obtain a fused feature vector;

[0028] The prediction module first performs a linear transformation on the fused feature vector, and then uses the Sigmoid function to obtain the answer prediction result;

[0029] Among them, a multi-level visual feature enhancement network is used to represent the features of visual images.

[0030] A third aspect of the present invention provides a computer-readable storage medium.

[0031] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual question answering method based on a multi-level visual feature enhancement network as described above.

[0032] A fourth aspect of the present invention provides a computer device.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the visual question answering method based on a multi-level visual feature enhancement network as described above.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] This invention proposes a multi-level visual feature enhancement method that strengthens the relationships between local objects and between regional objects and global concepts, thereby jointly learning the visual semantic relationships of multiple spatial contexts. A separate visual feature module based on a graph attention network is used to capture pixel-level visual features and object-level regional features; the joint visual feature representation based on a graph attention network jointly represents pixel-level and object-level features, while simultaneously learning the semantic relationships between different levels, better associating with the question text, and thus providing a richer visual feature representation. This solves the problem that traditional visual feature representations cause regional features and global features to lose their contextual connection, resulting in insufficient utilization of global semantics and loss of visual features.

[0036] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0037] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0038] Figure 1 This is an overall framework diagram of the method in the embodiment of the present invention.

[0039] Figure 2 This is the collaborative attention network structure of an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of an ablation experiment on a model using the VQA2.0 dataset from an embodiment of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] Example 1

[0043] Reference Figure 1 This embodiment presents a visual question answering method based on a multi-level visual feature enhancement network, which specifically includes the following steps:

[0044] S1. Obtain the questions to be answered and the visual images;

[0045] S2. Perform feature representation on the question to be answered to obtain the text feature representation vector; perform feature representation on the visual image to obtain the visual feature representation vector;

[0046] S3. Update the text feature representation vector and the visual feature representation vector;

[0047] S4. Fuse the updated text feature representation vector and the visual feature representation vector to obtain the fused feature vector;

[0048] S5. First, perform a linear transformation on the fused feature vector, and then use the Sigmoid function to obtain the answer prediction result;

[0049] In step S2, a multi-level visual feature enhancement network is used to represent the visual image features.

[0050] The multi-level visual feature enhancement network consists of a separate visual feature representation network and a joint visual feature representation network. The separate visual feature representation network is used to learn global visual features and regional visual features separately. The joint visual feature representation network is used to capture the semantic relationship between global visual features and regional visual features, and combines shallow detail features with deep semantic features to generate a complete visual feature representation vector. Shallow detail features refer to the captured local visual features, and deep semantic features refer to the relationship between local visual features and global features.

[0051] S2.1: For the textual feature representation of the problem, this invention uses the Glove model and the GRU model for feature representation.

[0052] First, the question text is preprocessed by segmenting the question into words based on spaces and punctuation marks. Second, the number of words in the question is padded or truncated to the same length N. Then, the GloVe model is used for word embedding. Finally, the word embeddings are input into the GRU network to obtain language feature representations.

[0053] S2.2: For visual feature representation, this invention uses a multi-level visual feature enhancement network to represent the original image as two levels of image features: pixel-level visual features and object-level visual features. Pixel-level visual features are global visual features, and object-level visual features are regional visual features.

[0054] The visual information processing module mainly consists of two modules: a separate visual feature representation module based on graph attention networks, which aims to learn object-level semantic relationships; and a joint visual feature representation module based on graph attention networks, which aims to capture the relationship between objects and the global context.

[0055] (1) Two-level image feature representation

[0056] Set the original image I, and extract global features G and region-based local features R respectively.

[0057] Global features: Extract global features from the last fully connected layer of ResNet152

[38] pre-trained on ImageNet

[37] . Where d0 represents the size of each pixel; g i Representation: Feature representation of the i-th node; Represents the characteristics of a node.

[0058] Local features: Faster R-CNN, pre-trained on the ImageNet

[37] dataset, with ResNet-101 as the main backbone network, extracts local features of the target in the detected image as follows: Where K represents the detected target object number; R represents the feature of a detected target; ri Representation: Feature representation of the detected target i.

[0059] Add a fully connected layer after them, embedding it into the shared potential space.

[0060] V G =W g G+b g V R =W r R+b r (1)

[0061] Among them W g W r Let b represent the weight matrix. g b r G represents the bias vector; G represents the global feature.

[0062] Finally, we obtain two image feature representations: Global visual features: and local visual features: Where D e Indicates the embedding dimension.

[0063] (2) Separate visual representation module based on graph attention network

[0064] To address the two-level image features, two independent semantic relationship enhancement models were designed to learn enhanced global image features (pixel relationships) and enhanced local image features (object relationships). Specifically, these models consist of an attention-based pixel relationship enhancement network and an attention-based object relationship enhancement network.

[0065] (2-1) Pay attention to pixel relationship enhancement network

[0066] For pixel relationship enhancement, this invention employs a graph attention network to capture the relationships between pixels one by one.

[0067] First, use global visual features V G Constructing the global visual graph G G =(V G E G Set edge set E) G For the affinity matrix, calculate each pair of features and The affinity between them. A higher affinity score indicates that the image region has a higher correlation.

[0068]

[0069] in, Represents node i; : represents node j; T: represents transpose.

[0070] Secondly, graph attention network modules are used to represent global visual semantic relationship enhancement features.

[0071]

[0072] Wherein, GAT represents the graph attention network module; G G : Represents the global visual graph.

[0073] This module primarily determines the degree to which each pixel is influenced by other pixels. This is manifested in the corresponding pixel having a higher attention value in the image, thereby promoting pixel-by-pixel learning.

[0074] (2-2) Pay attention to object relationship enhancement networks

[0075] To enhance object relationships, this invention employs graph attention networks to capture the relationships between regional objects.

[0076] First, construct a fully connected graph G. R =(V R E R ),like Figure 1 As shown, where V R E represents the characteristics of the object region. R Let be the edge set, and represent the affinity matrix. The affinity between each pair of features is calculated as shown in formula (2).

[0077]

[0078] in, Represents node i; Let j represent node j; T represents transpose.

[0079] Next, a graph attention network is used to process the object graph, which contains object features and their relationships. The final output is a region representation feature with enhanced semantic relationships between objects, as shown below:

[0080]

[0081] This module primarily determines the degree of mutual influence between objects, which manifests as a higher attention value for the corresponding object in the image, thereby promoting learning between objects.

[0082] (3) Joint visual feature representation module based on graph attention network

[0083] This module primarily demonstrates the representation of joint visual features, employing a multi-head image attention mechanism to integrate information between objects and pixels. The fusion process helps integrate multi-head input features and filter out useless information, such as... Figure 2As shown in the diagram. Specifically, it is divided into two parts: a joint feature representation module and a joint feature fusion module.

[0084] (3-1) Joint Feature Representation Module

[0085] For joint feature representations, the method involves feeding the joint feature representations into a graph attention network to learn all semantic relationships between object-level and pixel-level objects.

[0086] First, enhance pixel-level and object-level features. Linked into a joint vector V U ,in It is represented as a unified joint feature map G. U =(V U E U ).

[0087]

[0088] in, This represents the (n+k)th node; Represents node i; Represents node j; E U Represents an edge set.

[0089] Then, the joint graph attention representation, where the input to the graph attention model is the joint features mentioned above, allows this structure to help objects or pixels learn attention values ​​based on all objects and pixels.

[0090] To stabilize the self-attention learning process, this invention uses a multi-head attention mechanism, such as... Figure 1 As shown, G U The input is fed into k distinct GATs, and the output is represented as follows: in The representation is as follows:

[0091]

[0092] Among them, GAT K Let represent the graph attention neural network representing the k-th joint visual feature, and Mean denote average pooling. Representation: The feature vector of node k.

[0093] This module learns the semantic relationships between all independent elements, whether global or regional objects, through joint attention representation.

[0094] (3-2) Joint Feature Fusion

[0095] For joint feature fusion, this invention employs a gated fusion mechanism to integrate information and obtain the final representation of visual features.

[0096] The multi-head attention features obtained above are represented as V. C The image is then fused with a gated fusion layer to filter out more useless information and obtain the final image feature representation. The gated fusion layer uses two vectors... The input is the feature of the fused representation.

[0097]

[0098] Where W and U represent the fully connected layer parameters, σ represents the Sigmoid function; W1, W2, U1, and U2 represent the fully connected layer parameters; Two eigenvectors i and j; This represents fusion feature 1 and fusion feature 2.

[0099] Due to the different values ​​of K, we set different fusion mechanisms.

[0100] When k=1, feature fusion is not required, and the final image feature representation is as follows:

[0101] Fusion characteristics of two GAT graphs with k=2 and Therefore, the final image representation uses a gated fusion feature.

[0102]

[0103] k = 4 Four different GATs. The fusion process requires three gated fusion layers, where F() represents the fusion function. This represents the fusion feature of two GAT graphs.

[0104]

[0105] F1, F2, and F3 represent three gated fusion layers.

[0106] In step S3, the extracted text vectors and visual vectors are fed into the collaborative attention network. Through the collaborative attention network, the text vectors and visual vectors are self-attention modeled, and the visual attention is guided by a modular combination of two basic attention units. Finally, new visual and text feature vectors with attention are obtained.

[0107] The basic component of the collaborative attention network is the modular collaborative attention (MCA) layer, which is composed of two basic attention units: the self-attention unit (SA) and the guided attention unit (GA). The SA unit consists of a multi-head attention layer and a measurement feedforward layer, used to learn the relationships between samples within the same modality. The GA unit has a similar structure to the SA unit, using one modality to guide another, and is used to represent the feature relationships between different modalities.

[0108] In step S4, the updated text and visual features are fused using the Hadamard product to obtain a fused feature vector.

[0109] In step S5, the fused features are linearly transformed to obtain a transformed feature vector of dimension d. This d-dimensional feature vector is then converted into candidate answers of dimension N. The answer prediction result is obtained using the Sigmoid function, which is commonly used as a threshold function in neural networks, mapping variables to the range of 0 and 1. A higher threshold indicates a higher degree of matching with the answer. Binary cross-entropy (BCE) is used as the loss function to train the classification problem and to evaluate the goodness of the binary model's prediction results. The loss function is shown in the formula:

[0110]

[0111] Where N is the number of types in the multi-class classification, a′ i Let a be the predicted value for the i-th class. i Let be the label value of the i-th class.

[0112] Evaluation of experimental results

[0113] This embodiment evaluates the model on the VQA2.0 dataset.

[0114] (1) Dataset

[0115] The VQA 2.0 dataset is a commonly used benchmark dataset for VQA tasks. It consists of natural images from MSCOCO, with corresponding manually annotated questions and answers. Each image corresponds to 3 questions, and each question corresponds to 10 answers. The dataset is divided into three parts: a training set containing 80K images and 444K question-answer pairs; a validation set containing 40K images and 2^14 question-answer pairs; and a test set containing 80K images and 448K questions. The test set includes two subsets, test-dev and test-standard, for online evaluation of model performance.

[0116] (2) Evaluation of experimental results

[0117] Table 1 shows the online evaluation results on test-dev and test-std. As can be seen from Table 1, the method improves the overall accuracy of early models such as BUTD and MFH by 5.9% on the VQA2.0 dataset; for more recent models such as DSACA, MEDAN, and MCAN, the proposed method shows improvements of 0.47%–1.6%, 0.11%–1.81%, 0.07%–9.32%, and 0.12%–1.14% respectively on the dataset. This is because the proposed method considers the representation of visual features at multiple levels during the modeling process. By learning the complementary relationship between global and local features, the relationship between regional objects and global objects in the visual representation is strengthened, while also learning the interrelationships between individual objects. This method ensures the completeness and accuracy of information extraction through multi-level visual feature enhancement. In particular, because the BAN+Counter method has a dedicated object counting module (i.e., numeric type), its counting accuracy is better.

[0118] Table 1 compares the accuracy of our method with current leading visual question answering models on the VQA2.0 dataset.

[0119]

[0120] (3) Ablation test

[0121] To analyze the contribution and role of each part in the model, extensive ablation experiments were conducted on the proposed model on the VQA2.0 dataset, demonstrating the effectiveness of each module. For the visual feature representation part, there are two paths: global visual feature representation and local visual feature representation. Therefore, the ablation experiments were categorized as follows: 1) using only the global feature representation path; 2) using only the local visual feature representation path; 3) using both paths simultaneously for visual feature representation. The results are shown in Table 2, where "Regional" indicates global visual feature representation; "Global" indicates local visual feature representation; 'SRR' indicates separate visual feature representation; and "JRR" indicates joint visual feature representation.

[0122] As shown in rows 1-4 of Table 2, image feature representation is performed using only a single path, i.e., only local or global visual feature representation. Experimental results show that the use of Separate Visual Feature Representation (SRR) significantly improves performance, thus proving the effectiveness of SRR, which aims to learn pixel-level global relationships or object-level region relationships. Rows 2-7 show image feature representation using two paths. The two-level visual feature representation, consisting of the separate visual feature representation module and the joint visual feature representation, significantly improves the performance of the visual question answering model. This verifies that these two modules can learn multi-level image semantic information, thereby extracting more complete visual feature representations and ultimately improving the overall performance of visual question answering.

[0123] Table 2 Model ablation experiment

[0124]

[0125] Alternative approach: Other reasonable methods can be used, such as feature fusion algorithms based on other deep learning or feature fusion algorithms based on cross-modal similarity, to perform the feature fusion process.

[0126] Example 2

[0127] This embodiment provides a visual question answering system based on a multi-level visual feature enhancement network, including:

[0128] The question-and-answer acquisition module retrieves the questions to be answered and the visual images.

[0129] The feature representation module performs feature representation on the question to be answered, obtaining a text feature representation vector; and performs feature representation on the visual image, obtaining a visual feature representation vector.

[0130] The update module updates the text feature representation vector and the visual feature representation vector;

[0131] The fusion module fuses the updated text feature representation vector and the visual feature representation vector to obtain a fused feature vector;

[0132] The prediction module first performs a linear transformation on the fused feature vector, and then uses the Sigmoid function to obtain the answer prediction result;

[0133] Among them, a multi-level visual feature enhancement network is used to represent the features of visual images.

[0134] Example 3

[0135] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the visual question answering method based on a multi-level visual feature enhancement network as described in Embodiment 1 above.

[0136] Example 4

[0137] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the visual question answering method based on a multi-level visual feature enhancement network as described in Embodiment 1 above.

[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual question answering method based on a multi-level visual feature enhancement network, characterized in that, The method comprises the following steps: acquiring a question to be answered and a visual image; performing feature representation on the question to be answered to obtain a text feature representation vector; performing feature representation on the visual image to obtain a visual feature representation vector; updating the text feature representation vector and the visual feature representation vector; fusing the updated text feature representation vector and the visual feature representation vector to obtain a fused feature vector; performing linear transformation on the fused feature vector and then using a Sigmoid function to obtain an answer prediction result; wherein a multi-level visual feature enhancement network is used to perform feature representation on the visual image; the multi-level visual feature enhancement network comprises a graph attention-based separated visual feature representation network and a graph attention-based joint visual feature representation network; the separated visual feature representation network is used to learn global visual features and regional visual features respectively; the joint visual feature representation network is used to capture semantic relationships between the global visual features and the regional visual features, and combine shallow detail features with deep semantic features to generate a complete visual feature representation vector; the separated visual feature representation network is designed with two independent semantic relationship enhancement networks, namely an attention pixel relationship enhancement network and an attention object relationship enhancement network, for learning enhanced global visual features and enhanced regional visual features; the joint visual feature representation network comprises a joint feature representation network and a joint feature fusion network; the joint feature representation network is placed in a graph attention network to learn all semantic relationships between object-level and pixel-level objects; the joint feature fusion network uses a gating fusion mechanism to integrate information to obtain a final visual feature representation. 2.The visual question answering method based on the multi-level visual feature enhancement network of claim 1, wherein, The visual features include two levels of image features, namely pixel-level visual features and object-level visual features; the pixel-level visual features are global visual features, and the object-level visual features are regional visual features.

3. A visual question answering system based on a multi-level visual feature enhancement network, characterized in that, The method comprises the following steps: a question and answer acquisition module acquires a question to be answered and a visual image; a feature representation module performs feature representation on the question to be answered to obtain a text feature representation vector; a visual image is subjected to feature representation to obtain a visual feature representation vector; an updating module updates the text feature representation vector and the visual feature representation vector; a fusion module fuses the updated text feature representation vector and the visual feature representation vector to obtain a fused feature vector; a prediction module performs linear transformation on the fused feature vector and then uses a Sigmoid function to obtain an answer prediction result; wherein a multi-level visual feature enhancement network is used to perform feature representation on the visual image; the multi-level visual feature enhancement network comprises a graph attention-based separated visual feature representation network and a graph attention-based joint visual feature representation network; the separated visual feature representation network is used to learn global visual features and regional visual features respectively; the joint visual feature representation network is used to capture semantic relationships between the global visual features and the regional visual features, and combine shallow detail features with deep semantic features to generate a complete visual feature representation vector; The separate semantic relation enhancement networks are designed for two levels of image features, including an attention pixel relation enhancement network and an attention object relation enhancement network, for learning enhanced global visual features and enhanced regional visual features. The joint visual feature representation network comprises a joint feature representation network and a joint feature fusion network. The joint feature representation network is placed in a graph attention network to learn all semantic relations between object levels and pixel levels of objects, and the joint feature fusion network adopts a gating fusion mechanism to integrate information and obtain a final visual feature representation.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the steps in the method of any one of claims 1-2.

5. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps in the method of any one of claims 1-2.

Images