A visual question answering task processing method, system, electronic device, and storage medium

Abstract

This application discloses a visual question-answering task processing method, system, electronic device, and storage medium, belonging to the technical field of artificial intelligence technology, for achieving efficient visual question-answering task processing. The aforementioned visual question-answering task processing method includes: fusing image detection features, question text features, and classification fields to obtain comprehensive features; inputting the comprehensive features into a comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights; initializing a heterogeneous graph using the image feature segments, text feature segments, and image-text weights; constructing a feature corrector based on the attention relationships between the image and text features contained in the heterogeneous graph; correcting the image feature segments and text feature segments using the feature corrector; determining a prediction vector based on the position of the classification field in the correction result; and determining the answer corresponding to the visual question-answering task based on the prediction vector. This application can improve the accuracy and efficiency of visual question-answering task processing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a visual question answering task processing method, system, electronic device, and storage medium. Background Technology

[0002] Visual Question Answering (VQA) is an important branch of research in the field of multimodal computing. Its goal is to predict the answer to an input question using visual information. For example, by inputting an image and text into a VQA model, the model can understand the content of the question and answer based on the image information.

[0003] In related technologies, the LXMERT model is commonly used to handle visual question answering tasks. This method concatenates image features and text features to fuse information from different modalities. However, some information in the text cannot be found in the image, and vice versa. Directly concatenating image features and text features would require a large amount of useless computation, resulting in low accuracy and efficiency in visual question answering tasks.

[0004] Therefore, how to improve the accuracy and efficiency of visual question answering tasks is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a visual question answering task processing method, a visual question answering task processing system, an electronic device, and a storage medium, which can improve the accuracy and efficiency of visual question answering task processing.

[0006] To address the aforementioned technical problems, this application provides a visual question-answering task processing method, comprising:

[0007] Receive a visual question-answering task and determine the target image and question text based on the visual question-answering task;

[0008] Image detection features are extracted from the target image, and question text features are extracted from the question text.

[0009] The image detection features, the question text features, and the classification field are fused to obtain comprehensive features, and the comprehensive features are input into a comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights; wherein, the image-text weights include image-text attention weights and text-image attention weights;

[0010] A heterogeneous graph corresponding to the visual question answering task is generated, and the heterogeneous graph is initialized using the image feature segments, the text feature segments, and the image-text weights. A feature modifier is constructed based on the attention relationship between the image and text features contained in the heterogeneous graph. The heterogeneous graph includes visual nodes and text nodes.

[0011] The image feature segment and the text feature segment are corrected using the feature corrector, and a prediction vector is determined based on the position of the classification field in the correction result. The answer corresponding to the visual question answering task is then determined based on the prediction vector.

[0012] Optionally, initializing the heterogeneous graph using the image feature segments, the text feature segments, and the image-text weights includes:

[0013] Each image feature in the image feature segment is sequentially stored into the corresponding visual node in order to initialize the visual node of the heterogeneous graph;

[0014] Each text feature in the text feature segment is sequentially stored into the corresponding text node in order to initialize the text nodes of the heterogeneous graph;

[0015] Multiply the matrix corresponding to the graph weights with the prior filtering matrix to obtain the edge initialization matrix, so as to initialize the edges of the heterogeneous graph.

[0016] Optionally, before multiplying the matrix corresponding to the image and text weights with the prior filtering matrix, the method further includes:

[0017] Construct an N×M zero matrix; where N is the number of visual nodes, M is the number of text nodes, the horizontal axis of the zero matrix represents image features, and the vertical axis of the zero matrix represents image features.

[0018] The maximum value of the image-text attention weight is located in the text direction to obtain the text feature with the highest correlation to each image feature, and the coordinates of the image feature on the target image are set as the virtual coordinates of the text feature with the highest correlation.

[0019] The virtual coordinates of the text features are matched with the coordinates of all image features on the target image, and the elements corresponding to the matched coordinates in the zero matrix are set to 1 to obtain the prior filtering matrix.

[0020] Optionally, the feature modifier includes multiple cascaded modification interaction layers, each of which includes a first graph neural update unit and a second graph neural update unit. The first graph neural update unit is used to implement graph-text feature aggregation, and the second graph neural update unit is used to implement text-graph feature aggregation.

[0021] Optionally, the process of the first graph neural update unit realizing graph-text feature aggregation includes:

[0022] Construct an attention matrix, and calculate the first mapping weight of the visual node to the text node based on the attention matrix;

[0023] The first mapping weights are normalized.

[0024] The image features corresponding to the visual node are updated based on the image-text orientation attention weights between the visual node and the text node and the first mapping weights, so as to achieve image-text feature aggregation.

[0025] Optionally, the process of text-image feature aggregation implemented by the second graph neural update unit includes:

[0026] Construct an attention matrix, and calculate the second mapping weights of text nodes to visual nodes based on the attention matrix;

[0027] The second mapping weights are normalized.

[0028] The text features corresponding to the text node are updated based on the text-image attention weights between the visual node and the text node and the second mapping weights, so as to achieve text-image feature aggregation.

[0029] Optionally, a prediction vector is determined based on the position of the classification field in the correction result, and the answer corresponding to the visual question answering task is output based on the prediction vector, including:

[0030] The prediction vector is obtained by extracting the content corresponding to the position of the classification field in the correction result.

[0031] The predicted vectors are classified, and the answer corresponding to the visual question answering task is determined in the answer space based on the classification results.

[0032] This application also provides a visual question-answering task processing system, the system comprising:

[0033] The task receiving module is used to receive visual question answering tasks and determine the target image and question text based on the visual question answering tasks.

[0034] The feature extraction module is used to extract image detection features from the target image and question text features from the question text.

[0035] The encoding module is used to fuse the image detection features, the question text features, and the classification field to obtain comprehensive features, and input the comprehensive features into the comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights; wherein, the image-text weights include image-text attention weights and text-image attention weights;

[0036] A modifier construction module is used to generate a heterogeneous graph corresponding to the visual question answering task, initialize the heterogeneous graph using the image feature segments, the text feature segments, and the image-text weights, and construct a feature modifier based on the attention relationship between the image-text features contained in the heterogeneous graph; wherein, the heterogeneous graph includes visual nodes and text nodes;

[0037] The answer determination module is used to correct the image feature segments and the text feature segments using the feature corrector, and to determine the prediction vector based on the position of the classification field in the correction result, and to determine the answer corresponding to the visual question answering task based on the prediction vector.

[0038] This application also provides a storage medium on which a computer program is stored, wherein the computer program, when executed, implements the steps of the above-described visual question-answering task processing method.

[0039] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor invokes the computer program in the memory to implement the steps of the above-described visual question-answering task processing method.

[0040] This application provides a visual question-answering task processing method, comprising: receiving a visual question-answering task and determining a target image and question text according to the visual question-answering task; extracting image detection features from the target image and question text features from the question text; fusing the image detection features, the question text features, and a classification field to obtain comprehensive features, and inputting the comprehensive features into a comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights; wherein the image-text weights include image-text attention weights and text-image attention weights; generating a heterogeneous graph corresponding to the visual question-answering task, initializing the heterogeneous graph using the image feature segments, the text feature segments, and the image-text weights, and constructing a feature corrector according to the attention relationship between the image and text features contained in the heterogeneous graph; wherein the heterogeneous graph includes visual nodes and text nodes; correcting the image feature segments and text feature segments using the feature corrector, determining a prediction vector according to the position of the classification field in the correction result, and determining the answer corresponding to the visual question-answering task according to the prediction vector.

[0041] This application extracts corresponding image detection features and question text features after receiving a visual question answering task, and fuses these features with a classification field to obtain comprehensive features. This application uses a comprehensive feature transformer to transform these comprehensive features, obtaining image feature segments, text feature segments, and image-text weights. After generating a heterogeneous graph corresponding to the visual question answering task, the heterogeneous graph is initialized using the image feature segments, text feature segments, and image-text weights. A feature modifier is constructed using the attention relationships between the image and text features contained in the heterogeneous graph. Since images and text belong to different modalities, the nodes storing image features and the nodes storing text features in the heterogeneous graph are heterogeneous. The feature modifier constructed using the heterogeneous graph can retain effective information in both image and text feature segments, improving the accuracy and efficiency of visual question answering task processing. This application also provides a visual question answering task processing system, a storage medium, and an electronic device, which have the above-mentioned beneficial effects, and will not be elaborated further here. Attached Figure Description

[0042] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a visual question-answering task processing method provided in an embodiment of this application;

[0044] Figure 2 A flowchart illustrating a visual question-answering task processing scheme based on prior heterogeneous interaction provided in an embodiment of this application;

[0045] Figure 3 A schematic diagram illustrating the principle of a comprehensive feature encoder provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the initialization of a text-image heterogeneous graph structure provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of a feature modifier provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a visual question-answering task processing system provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Please see below. Figure 1 , Figure 1 This is a flowchart of a visual question-answering task processing method provided in an embodiment of this application.

[0051] Specific steps may include:

[0052] S101: Receive a visual question-answering task and determine the target image and question text based on the visual question-answering task;

[0053] This embodiment can be applied to electronic devices with visual question-answering capabilities. The visual question-answering task can be a task issued by a user or a task transmitted by other devices. Based on the question-answering task, the target image and the question text can be determined. The question text is a question about certain information in the target image. For example, if the target image is an image containing multiple bicycles, the question text could be "How many bicycles are there in the picture?"

[0054] S102: Extract image detection features from the target image, and extract question text features from the question text;

[0055] In this step, convolutional neural networks can be used to extract image detection features from the target image, and text encoders (such as BERT or RoBERTa) can be used to extract features from the question text to obtain question text features.

[0056] S103: The image detection features, the question text features, and the classification field are fused to obtain comprehensive features, and the comprehensive features are input into the comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights;

[0057] This step involves encoding both image detection features and question text features. The image and text feature encoding results are then concatenated and fused to obtain an image-text fused feature. This image-text fused feature is then fused with a classification field to obtain a comprehensive feature. In this embodiment, a fixed vector can be initialized to represent a classification field (CLS), and the classification field is appended to the fused feature to obtain the comprehensive feature.

[0058] As another feasible implementation, this embodiment can perform image feature encoding on the image detection features, fuse the question text features with the classification field to obtain the comprehensive text features, perform text feature encoding on the comprehensive text features, and fuse the image feature encoding results and the text feature encoding results to obtain the comprehensive features.

[0059] After obtaining the comprehensive features, these features can be input into the comprehensive feature encoder. The output features of the comprehensive feature encoder include image feature segments, text feature segments, and image-text weights. The image-text weights include image-text attention weights and text-image attention weights. Image-text attention weights are the attention weights of image features on text features, and text-image attention weights are the attention weights of text features on image features.

[0060] S104: Generate a heterogeneous graph corresponding to the visual question answering task, initialize the heterogeneous graph using the image feature segments, the text feature segments, and the image-text weights, and construct a feature modifier based on the attention relationship between the image and text features contained in the heterogeneous graph; wherein, the heterogeneous graph includes visual nodes and text nodes;

[0061] This embodiment can generate a heterogeneous graph (i.e., a graph-text heterogeneous graph structure) that includes a visual space and a text space. The visual space contains multiple visual nodes, and the text space contains multiple text nodes. The process of initializing the above heterogeneous graph includes node initialization and edge initialization. Node initialization includes visual node initialization and text node initialization.

[0062] The visual node initialization process is as follows: each image feature in the image feature segment is sequentially stored into the corresponding visual node to initialize the visual nodes of the heterogeneous graph. The text node initialization process is as follows: each text feature in the text feature segment is sequentially stored into the corresponding text node to initialize the text nodes of the heterogeneous graph. The edge initialization process is as follows: the matrix corresponding to the graph-text weights is multiplied by the prior filtering matrix to obtain the edge initialization matrix to initialize the edges of the heterogeneous graph.

[0063] Specifically, during edge initialization, the image-text attention weights and text-image attention weights can be multiplied by the prior filtering matrix to obtain the edge initialization matrix. The prior filtering matrix is ​​generated by: constructing an N×M zero matrix; where N is the number of visual nodes and M is the number of text nodes, the horizontal axis of the zero matrix represents image features, and the vertical axis of the zero matrix also represents image features; locating the maximum value of the image-text attention weights in the text direction to obtain the text feature with the highest correlation to each image feature, and setting the coordinates of the image feature on the target image as the virtual coordinates of the text feature with the highest correlation; matching the virtual coordinates of the text feature with the coordinates of all image features on the target image, and setting the corresponding element of the matched coordinate on the zero matrix to 1 to obtain the prior filtering matrix.

[0064] The initialized heterogeneous graph stores the attention relationship between image features and text features, and a feature modifier can be constructed based on this attention relationship.

[0065] S105: The image feature segment and the text feature segment are corrected using the feature corrector, and a prediction vector is determined based on the position of the classification field in the correction result. The answer corresponding to the visual question answering task is determined based on the prediction vector.

[0066] In this process, after obtaining the feature modifier, both image and text feature segments can be modified bilaterally. Then, based on the position of the classification field in the modified result, a prediction vector is determined. Finally, the answer to the visual question-answering task is determined by combining the prediction vector. Specifically, in this embodiment, the content corresponding to the position of the classification field in the modified result can be extracted to obtain the prediction vector; the prediction vector is classified, and the answer to the visual question-answering task is determined in the answer space based on the classification result.

[0067] This embodiment extracts corresponding image detection features and question text features after receiving a visual question answering task. The image detection features, question text features, and classification fields are then fused to obtain comprehensive features. This embodiment uses a comprehensive feature transformer to transform the comprehensive features, obtaining image feature segments, text feature segments, and image-text weights. After generating a heterogeneous graph corresponding to the visual question answering task, the heterogeneous graph is initialized using the image feature segments, text feature segments, and image-text weights. A feature modifier is constructed using the attention relationships between the image and text features contained in the heterogeneous graph. Since images and text belong to different modalities, the nodes storing image features and the nodes storing text features in the heterogeneous graph are heterogeneous. The feature modifier constructed using the heterogeneous graph can retain effective information in both image and text feature segments, improving the accuracy and efficiency of visual question answering task processing.

[0068] As for Figure 1 In a further description of the corresponding embodiment, the feature corrector may include multiple cascaded correction interaction layers. Each correction interaction layer includes a first graph neural update unit and a second graph neural update unit. The first graph neural update unit is used to implement image-text feature aggregation, and the second graph neural update unit is used to implement text-image feature aggregation. Image-text feature aggregation refers to aggregating image features to text nodes, and text-image feature aggregation refers to aggregating text features to visual nodes.

[0069] Specifically, the process of the first graph neural update unit to achieve graph-text feature aggregation includes: constructing an attention matrix; calculating the first mapping weight of the visual node to the text node based on the attention matrix; normalizing the first mapping weight; and updating the image features corresponding to the visual node based on the graph-text attention weight between the visual node and the text node and the first mapping weight, so as to achieve graph-text feature aggregation.

[0070] Specifically, the process of text-image feature aggregation implemented by the second graph neural update unit includes: constructing an attention matrix; calculating the second mapping weight of the text node to the visual node based on the attention matrix; normalizing the second mapping weight; and updating the text features corresponding to the text node based on the text-image attention weight between the visual node and the text node and the second mapping weight, so as to realize text-image feature aggregation.

[0071] The process described in the above embodiments is illustrated below through examples in practical applications.

[0072] Visual question answering (VQA) is a type of image and text understanding task, and multimodal research has become one of the most popular research directions in the field of artificial intelligence. Because multimodal learning often involves various features such as vision, speech, and text, it is more similar to everyday scenarios and therefore has better application prospects, making it one of the mainstream research directions in artificial intelligence. Currently, multimodal research mainly focuses on content understanding (artificial intelligence theoretical research can be broadly divided into content understanding and content generation), and VQA is a fundamental task of content understanding. The accuracy of VQA is reflected in the ability of artificial intelligence to understand the content of images and text. Since visual question answering (VQA) is a relatively classic task, many methods have been developed to solve it. Among these, the most classic and accurate methods are based on the transformer architecture, such as VLBERT and LXMERT models.

[0073] The typical process for visual question answering tasks includes: image feature extraction using a convolutional neural network; and input question feature extraction using an existing text encoder (such as BERT or RoBERTa). The extracted features are then encoded separately. The two types of features are then concatenated and fused to obtain a fused image-text feature. A fixed vector is initialized to represent a classification field [CLS], which is appended to the fused feature to obtain a comprehensive feature. This is further encoded, and the positions corresponding to the [CLS] feature are re-extracted to represent the prediction vector. Finally, the prediction vector is classified, and the answer with the highest score in the answer space is output.

[0074] The conventional approach to visual question answering tasks described above fuses information from different modalities by concatenating visual and textual features. However, this process has a problem: not all features need to be fused; that is, some information in the text cannot be found in the image, and vice versa. Therefore, simply concatenating the two and training on a massive dataset is not a straightforward solution.

[0075] To address the shortcomings of conventional techniques, this application provides a visual question-answering task processing scheme based on prior heterogeneous interaction. This scheme designs a feature modifier suitable for visual question answering and ensures that the modified features contain more "valuable information." The graph structure consists of nodes and edges. The strength of the relationship between connected nodes can be adjusted according to the size of the edges, which is physically consistent with whether there is a correlation between different features of images and text. Since images and text belong to different modalities, the graph nodes storing image features and the graph nodes storing text features are heterogeneous.

[0076] Please see Figure 2 , Figure 2 The flowchart of a visual question-answering task processing scheme based on prior heterogeneous interaction provided in this application embodiment is as follows: The target image is processed by an image feature extraction network to obtain image detection features; the question text is processed by a word frequency dictionary text feature extraction and concatenated with the initial vector of the classification field CLS to obtain comprehensive text features. Image feature encoding is performed on the image change features, and text feature encoding is performed on the comprehensive text features. The image feature encoding results and text feature encoding results are fused to obtain comprehensive features. Comprehensive feature encoding is performed on the comprehensive features to obtain image feature segments, text feature segments, and image-text weights. A feature corrector is used to correct the image feature segments and text feature segments to obtain corrected image feature segments and corrected text feature segments. The prediction vector is determined based on the position of the classification field in the corrected text feature segments and prediction is performed in the answer space.

[0077] In the above process, the comprehensive feature encoding is implemented by a comprehensive feature encoder, which includes several cascaded cross-attention layers. Each cross-attention layer includes a self-attention sublayer and a cross-attention sublayer, which are respectively composed of self / cross-attention mechanisms, random erasure, layer normalization, and summation. That is, the self-attention sublayer can implement the self-attention mechanism, random erasure, layer normalization, and summation; the cross-attention sublayer can implement the cross-attention mechanism, random erasure, layer normalization, and summation. Random erasure refers to randomly erasing a portion of the feature values ​​according to a certain proportion to prevent overfitting; layer normalization is used to normalize between layers; summation is used to add the model output to the original features. The formula for the self-attention mechanism is as follows:

[0078]

[0079]

[0080] The formula for the cross-attention mechanism is as follows:

[0081]

[0082] In the above formula, f and g represent two different input features of size [N, d], where N represents the number of features, d represents the feature dimension, and W... q W k W v Let T and g be matrices of size [d, d], used to map the input features to a specified space. T represents the transpose matrix, size(f) represents the dimension of f, and size(g) represents the dimension of g. The cross-attention mechanism is mainly used to compute the representation of feature f on g, thereby achieving attention of g on f.

[0083] This embodiment adds an output interface to the existing integrated feature encoder. This output interface is used to output the text-to-image attention weights and text-to-image attention weights of the last cross-attention mechanism.

[0084] Attention weight W for text-to-image attention weights gf for:

[0085]

[0086] Image and text focus attention weight W fg for:

[0087]

[0088] In the above formula, q represents query and k represents key. Query and key are the inputs to the self-attention mechanism.

[0089] Please see Figure 3, Figure 3 This is a schematic diagram illustrating the principle of a comprehensive feature encoder provided in this application embodiment. The comprehensive features input to the comprehensive feature encoder include image feature segments and text feature segments. After processing by a self-attention sub-layer (self-attention, random erasure, layer normalization, and addition) and a cross-attention sub-layer (cross-attention, random erasure, layer normalization, and addition), image feature segments, text feature segments, image-to-text attention weights, and text-to-image attention weights are obtained.

[0090] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the initialization of a heterogeneous graph structure provided in an embodiment of this application. Graph structures are one of the fundamental structures in computer science, consisting of nodes and edges. The structure and initialization method of the heterogeneous graph designed in this solution are described below. Figure 4 middle For visual nodes, For text nodes,

[0091] This embodiment addresses the problem of low effectiveness of image and text features in the field of visual question answering by proposing a solution using the idea of ​​heterogeneous graph structure aggregation. For example... Figure 4 As shown, a heterogeneous graph includes nodes of two types and edges between different nodes. For nodes, the two types of properties refer to the source of the feature space: visual space or text space; for edges, there are only edges between nodes in different spaces, not between two nodes in the same space. All edges are directed, meaning there are two edges between any two nodes.

[0092] After constructing the heterogeneous graph, node initialization and edge initialization can be performed. During node initialization, image feature segments of size [N, d] can be sequentially stored into N visual nodes; text feature segments of size [M, d] can be sequentially stored into M text nodes. N is the number of features in the image feature segment, M is the number of features in the text feature segment, and d is the feature dimension.

[0093] For the edge initialization process, the edge initialization matrix is ​​first calculated, as shown in the figure. The edge initialization matrix is ​​obtained by multiplying the graph-text matrix weights by the prior filtering matrix. The graph-text weight matrix is ​​the corresponding output of the previous module; the prior filtering matrix represents the prior correlation between the graph and the text. This matrix is ​​a binary matrix of size [N,M], consisting of 0s and 1s. The number in the i-th row and j-th column of the matrix indicates whether there is a correlation between the i-th graph node and the j-th text node. The specific generation method for the prior filtering matrix is ​​as follows: First, construct a zero matrix of size [N, M]. For the image-text attention weights, locate the maximum value in the text direction to find the text feature most likely associated with each image feature. Use the coordinates of this image feature on the original input image as the virtual coordinates of the text feature (if a text feature is the attention maximum of multiple image features, then merge the coordinates of these image features into a list as the virtual coordinates of the text feature). For each text feature, compare the virtual coordinates (or the list of coordinates) with the coordinates of all image features, and set the corresponding positions on the zero matrix to 1 for spatially overlapping features, thus obtaining the prior filtering matrix. Multiplying the image-text weight matrix with the prior filtering matrix initializes the edges of the heterogeneous graph structure.

[0094] Please see Figure 5 , Figure 5 This is a schematic diagram of a feature modifier provided in an embodiment of this application. The feature modifier can be obtained based on a text-image heterogeneous graph. The feature modifier includes several cascaded modification interaction layers. Each cascaded interaction layer includes two graph neural update units (triangles in the figure), one graph neural update unit is used for text-image feature aggregation, and the other graph neural update unit is used for text-image feature aggregation.

[0095] Taking the process of visual nodes to text nodes as an example, the operation process of the graph neural update unit is as follows:

[0096] Step 1: Construct four attention matrices Wc, Wv, Wb, and Wn, each with a size of [d, d].

[0097] Here, the input vector of the attention matrix is ​​q; Wq is the result of matrix operation, used to represent the mapping process of vector q.

[0098] Step 2: Calculate the mapping weight Z from visual node I to text node T. ti .

[0099] z ti =ReLU(W c [W v f t W b f i ]);

[0100] Among them, f t f represents the feature vector stored in text node T. i Z represents the feature vector stored in visual node I, and the calculated Z ti This represents the mapping weight of visual node I to text node T.

[0101] Step 3: Map the weights Z ti Normalization is performed.

[0102] The normalization formula is: α ti This represents the normalized mapping weights, and exp(*) represents the exponential operator.

[0103] Step 4: Calculate the edge matrix W between visual node I and text node T. ti (Image-text attention weights) are incorporated to update node features, as shown in the following formula:

[0104] Where w ti Represents the edge matrix W ti The corresponding boundary value in f. t Let σ be the original feature of the node, σ be the hyperparameter, and Nt be the number of nodes.

[0105] Step 5: Reweight all text nodes after feature updates. Specifically, construct a matrix W of size [d, d] ti Multiply the resulting feature and map it.

[0106] For the reverse node update, similar to steps 1-5, simply swap the visual node I with the text node T. Finally, the feature segment corresponding to [CLS] at the text feature location can be extracted as a prediction vector and classified in the answer space to obtain the final output.

[0107] This embodiment targets visual question answering tasks, employing a graph neural network structure to design a novel visual question answering system. By designing a reasonable heterogeneous graph structure and initialization method, it provides suitable interfaces and logic for the system. Following the comprehensive feature encoding module, this embodiment constructs a feature modifier using the attention relationship between graph and text features, and uses it to modify the bilateral features. The corresponding position of the classification field CLS of the modified text feature segment is extracted for subsequent answer prediction.

[0108] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a visual question-answering task processing system provided in an embodiment of this application. The system may include:

[0109] The task receiving module 601 is used to receive visual question answering tasks and determine the target image and question text according to the visual question answering tasks;

[0110] Feature extraction module 602 is used to extract image detection features from the target image and extract question text features from the question text;

[0111] The encoding module 603 is used to fuse the image detection features, the question text features, and the classification field to obtain comprehensive features, and input the comprehensive features into the comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights; wherein, the image-text weights include image-text attention weights and text-image attention weights;

[0112] The modifier construction module 604 is used to generate a heterogeneous graph corresponding to the visual question answering task, initialize the heterogeneous graph using the image feature segments, the text feature segments, and the image-text weights, and construct a feature modifier based on the attention relationship between the image-text features contained in the heterogeneous graph; wherein, the heterogeneous graph includes visual nodes and text nodes;

[0113] The answer determination module 605 is used to correct the image feature segment and the text feature segment using the feature corrector, and to determine the prediction vector based on the position of the classification field in the correction result, and to determine the answer corresponding to the visual question answering task based on the prediction vector.

[0114] This embodiment extracts corresponding image detection features and question text features after receiving a visual question answering task. The image detection features, question text features, and classification fields are then fused to obtain comprehensive features. This embodiment uses a comprehensive feature transformer to transform the comprehensive features, obtaining image feature segments, text feature segments, and image-text weights. After generating a heterogeneous graph corresponding to the visual question answering task, the heterogeneous graph is initialized using the image feature segments, text feature segments, and image-text weights. A feature modifier is constructed using the attention relationships between the image and text features contained in the heterogeneous graph. Since images and text belong to different modalities, the nodes storing image features and the nodes storing text features in the heterogeneous graph are heterogeneous. The feature modifier constructed using the heterogeneous graph can retain effective information in both image and text feature segments, improving the accuracy and efficiency of visual question answering task processing.

[0115] Furthermore, the process by which the corrector construction module 604 initializes the heterogeneous graph using the image feature segment, the text feature segment, and the image-text weights includes: storing each image feature in the image feature segment sequentially into the corresponding visual node to initialize the visual node of the heterogeneous graph; storing each text feature in the text feature segment sequentially into the corresponding text node to initialize the text node of the heterogeneous graph; and multiplying the matrix corresponding to the image-text weights with the prior filtering matrix to obtain the edge initialization matrix to initialize the edges of the heterogeneous graph.

[0116] Furthermore, it also includes:

[0117] The prior filtering matrix construction module is used to construct an N×M zero matrix before multiplying the matrix corresponding to the image-text weights with the prior filtering matrix; where N is the number of visual nodes, M is the number of text nodes, the horizontal axis of the zero matrix represents the image features, and the vertical axis of the zero matrix represents the image features; it is also used to locate the maximum value of the image-text attention weights in the text direction to obtain the text feature with the highest correlation to each image feature, and set the coordinates of the image feature on the target image as the virtual coordinates of the text feature with the highest correlation; it is also used to match the virtual coordinates of the text feature with the coordinates of all image features on the target image, and set the element corresponding to the matched coordinate on the zero matrix to 1 to obtain the prior filtering matrix.

[0118] Furthermore, the feature modifier includes multiple cascaded modification interaction layers, each of which includes a first graph neural update unit and a second graph neural update unit. The first graph neural update unit is used to realize graph-text feature aggregation, and the second graph neural update unit is used to realize text-graph feature aggregation.

[0119] Furthermore, the process of the first graph neural update unit to realize graph-text feature aggregation includes: constructing an attention matrix, calculating the first mapping weight of the visual node to the text node based on the attention matrix; normalizing the first mapping weight; and updating the image features corresponding to the visual node based on the graph-text attention weight between the visual node and the text node and the first mapping weight, so as to realize graph-text feature aggregation.

[0120] Furthermore, the process of text-image feature aggregation implemented by the second graph neural update unit includes: constructing an attention matrix; calculating the second mapping weight of the text node to the visual node based on the attention matrix; normalizing the second mapping weight; and updating the text features corresponding to the text node based on the text-image attention weight between the visual node and the text node and the second mapping weight, so as to realize text-image feature aggregation.

[0121] Furthermore, the answer determination module 605 is used to extract the content corresponding to the position of the classification field in the correction result into the prediction vector; it is also used to classify the prediction vector and determine the answer corresponding to the visual question answering task in the answer space based on the classification result.

[0122] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and they will not be repeated here.

[0123] This application also provides a storage medium on which a computer program is stored, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0124] This application also provides an electronic device that may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the electronic device may also include various network interfaces, power supplies, and other components.

[0125] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0126] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A visual question-answering task processing method, characterized in that, include: Receive a visual question-answering task and determine the target image and question text based on the visual question-answering task; Image detection features are extracted from the target image, and question text features are extracted from the question text. The image detection features are encoded using image feature encoding, the question text features are encoded using text feature encoding, the image feature encoding results and the text feature encoding results are concatenated and fused to obtain image-text fusion features, and the image-text fusion features are fused with the classification field to obtain comprehensive features; The classification field is represented by an initialized fixed vector; the comprehensive features are input into the comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights; wherein, the image-text weights include image-text attention weights and text-image attention weights; the image-text attention weights are the attention weights of image features on text features, and the text-image attention weights are the attention weights of text features on image features; Generate a heterogeneous graph corresponding to the visual question-answering task, the heterogeneous graph including visual nodes and text nodes; sequentially store each image feature in the image feature segment into the corresponding visual node to initialize the visual nodes of the heterogeneous graph; sequentially store each text feature in the text feature segment into the corresponding text node to initialize the text nodes of the heterogeneous graph; construct an N×M zero matrix; where N is the number of visual nodes and M is the number of text nodes, the horizontal axis of the zero matrix represents the image feature, and the vertical axis of the zero matrix represents the image feature; locate the maximum value of the image-text direction attention weight in the text direction to obtain the value corresponding to each image feature. The text feature with the highest correlation is used, and the coordinates of the image feature on the target image are set as the virtual coordinates of the text feature with the highest correlation. The virtual coordinates of the text feature are matched with the coordinates of all image features on the target image, and the elements corresponding to the matched coordinates in the zero matrix are set to 1 to obtain the prior filtering matrix. The matrix corresponding to the image and text weights is multiplied by the prior filtering matrix to obtain the edge initialization matrix, so as to initialize the edges of the heterogeneous graph. The process of initializing the heterogeneous graph includes node initialization and edge initialization. Node initialization includes visual node initialization and text node initialization. A feature modifier is constructed based on the attention relationships between the image and text features contained in the initialized heterogeneous graph. The feature modifier includes multiple cascaded modification interaction layers. Each modification interaction layer includes a first graph neural update unit and a second graph neural update unit. The first graph neural update unit is used to realize image-text feature aggregation, and the second graph neural update unit is used to realize text-image feature aggregation. The feature modifier retains the effective information in the image feature segments and the text feature segments. The image feature segment and the text feature segment are corrected using the feature corrector, and a prediction vector is determined based on the position of the classification field in the correction result. The answer corresponding to the visual question answering task is then determined based on the prediction vector.

2. The visual question-answering task processing method according to claim 1, characterized in that, The process of image-text feature aggregation implemented by the first image neural update unit includes: Construct an attention matrix, and calculate the first mapping weight of the visual node to the text node based on the attention matrix; The first mapping weights are normalized. The image features corresponding to the visual node are updated based on the image-text orientation attention weights between the visual node and the text node and the first mapping weights, so as to achieve image-text feature aggregation.

3. The visual question-answering task processing method according to claim 1, characterized in that, The process of text-image feature aggregation implemented by the second graph neural update unit includes: Construct an attention matrix, and calculate the second mapping weights of text nodes to visual nodes based on the attention matrix; The second mapping weights are normalized. The text features corresponding to the text node are updated based on the text-image attention weights between the visual node and the text node and the second mapping weights, so as to achieve text-image feature aggregation.

4. The visual question-answering task processing method according to claim 1, characterized in that, The prediction vector is determined based on the position of the classification field in the correction result, and the answer corresponding to the visual question answering task is output based on the prediction vector, including: The prediction vector is obtained by extracting the content corresponding to the position of the classification field in the correction result. The predicted vectors are classified, and the answer corresponding to the visual question answering task is determined in the answer space based on the classification results.

5. A visual question-answering task processing system, characterized in that, include: The task receiving module is used to receive visual question answering tasks and determine the target image and question text based on the visual question answering tasks. The feature extraction module is used to extract image detection features from the target image and question text features from the question text. The encoding module is used to perform image feature encoding on the image detection features, perform text feature encoding on the question text features, concatenate and fuse the image feature encoding results and the text feature encoding results to obtain image-text fusion features, and fuse the image-text fusion features with the classification field to obtain comprehensive features; The classification field is represented by an initialized fixed vector; the comprehensive features are input into the comprehensive feature encoder to obtain image feature segments, text feature segments, and image-text weights; wherein, the image-text weights include image-text attention weights and text-image attention weights; the image-text attention weights are the attention weights of image features on text features, and the text-image attention weights are the attention weights of text features on image features; A corrector construction module is used to generate a heterogeneous graph corresponding to the visual question-answering task, the heterogeneous graph including visual nodes and text nodes; each image feature in the image feature segment is sequentially stored into the corresponding visual node to initialize the visual nodes of the heterogeneous graph; each text feature in the text feature segment is sequentially stored into the corresponding text node to initialize the text nodes of the heterogeneous graph; an N×M zero matrix is ​​constructed; where N is the number of visual nodes and M is the number of text nodes, the horizontal axis of the zero matrix represents the image feature, and the vertical axis of the zero matrix represents the image feature; the maximum value of the image-text direction attention weight is located in the text direction to obtain the text feature with the highest correlation to each image feature, and the coordinates of the image feature on the target image are set as the virtual coordinates of the text feature with the highest correlation; the virtual coordinates of the text feature and all image features on the target image are then compared. Coordinates on the image are matched, and the elements corresponding to the matched coordinates in the zero matrix are set to 1 to obtain a prior filtering matrix. The matrix corresponding to the image-text weights is multiplied by the prior filtering matrix to obtain an edge initialization matrix, which is used to initialize the edges of the heterogeneous graph. The process of initializing the heterogeneous graph includes node initialization and edge initialization. Node initialization includes visual node initialization and text node initialization. A feature modifier is constructed based on the attention relationship between the image and text features contained in the initialized heterogeneous graph. The feature modifier includes multiple cascaded modification interaction layers. Each modification interaction layer includes a first graph neural update unit and a second graph neural update unit. The first graph neural update unit is used to implement image-text feature aggregation, and the second graph neural update unit is used to implement text-image feature aggregation. The feature modifier retains the effective information in the image feature segments and the text feature segments. The answer determination module is used to correct the image feature segments and the text feature segments using the feature corrector, and to determine the prediction vector based on the position of the classification field in the correction result, and to determine the answer corresponding to the visual question answering task based on the prediction vector.

6. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the visual question answering task processing method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the visual question-answering task processing method as described in any one of claims 1 to 4.

Images