External knowledge visual question answering method based on multimodal information enhancement and interaction

Abstract

The present invention relates to an external knowledge visual question-answering method based on multimodal information enhancement and interaction. The method comprises: extracting visual features with semantic information from the original image; processing the original question to obtain question text embedding, and extracting each feature through the LXMERT model; calculating the similarity matrix based on the tensor Tucker fusion method to obtain an autoregressive prompt; obtaining a unimodal prompt based on a single model text question-answering module; connecting the autoregressive prompt and the unimodal prompt with the original question, and calculating again to obtain the final answer. The original question is used to generate an autoregressive prompt and a unimodal prompt is generated based on the image description and the original question to improve the richness of the question text information; the tensor Tucker fusion method is used to perform feature interaction to capture the correlation between the target and the question; the autoregressive prompt and the unimodal prompt are connected to the original question, and the recursive answer generation paradigm is used to strengthen the consistency of the two stages, so that the accuracy of the generated final answer is high.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision question answering, and in particular to an external knowledge visual question answering method based on multimodal information enhancement and interaction. Background Art

[0002] With the booming development of computer vision and natural language processing, Visual Question Answering (VQA) has attracted extensive attention from academia and industry as a cutting-edge task that integrates multimodal information. VQA requires computer systems to understand not only the content of images, but also natural language questions and provide accurate responses, thus involving complex tasks of cross-modal information understanding and reasoning. In traditional Visual Question Answering (VQA) tasks, models mainly rely on local information between images and questions. However, with the increasing complexity and diversity of questions, relying solely on local information between images and questions often cannot meet the task requirements. In order to improve the performance and robustness of visual question answering systems, more and more studies are exploring how to combine external knowledge to assist question understanding and answer reasoning, namely knowledge-based Visual Question Answering (KB-VQA), which aims to use rich external knowledge resources such as text corpora, knowledge graphs, and network resources to enrich the semantic representation of questions and images, thereby improving the system's ability to understand and answer complex questions.

[0003] Currently, the main need is to shift the attention of knowledge-based visual question answering to capturing relevant knowledge from structured sources or unstructured / semi-structured repositories. While these repositories provide high-quality knowledge through meticulous manual annotations, the information they provide is often limited to facts explicitly expressed in natural language or simple triples using first-order predicates. As a result, these knowledge bases encounter challenges in representing high-order predicates and multimodal knowledge, which are essential for human reasoning to solve complex problems. Representing and accumulating complex multimodal knowledge in VQA scenarios while retaining the interpretability advantage of traditional knowledge graphs remains an important but unexplored problem. Recent advances in the emerging multimodal knowledge graphs aim to connect visual content with textual facts and create rich knowledge graphs. Common solutions fall into two categories: parsing images and text into structured representations, building cross-modal foundations for events / entities, or aligning entities in existing knowledge graphs with relevant images. However, these multimodal knowledge graphs fundamentally represent knowledge using first-order predicates and have difficulty modeling complex high-order relations. There are also models that introduce a triple-based answer prediction within a framework of multimodal knowledge extraction and accumulation. Due to the accumulation of multimodal knowledge, the complex relationships observed from VQA samples are integrated and explainable reasoning is performed based on the acquired knowledge.

[0004] However, the above methods mainly optimize the answer prediction module, ignoring the impact of the input multimodal data on answer generation. First, visual features are usually the embedding of the target appearance and the spatial information provided by the detector in the image; during training, it is impossible to augment the data as with the original image data; most methods directly input these features into the multimodal encoder for interactive processing. The lack of diversity and complexity of visual features makes the model prone to overfitting. Second, compared with the VQA task, the questions in KB-VQA are more complex and may not have a clear correspondence with the image. Direct input into the model poses challenges to semantic understanding. Summary of the invention

[0005] Based on this, in order to solve the above technical problems, an external knowledge visual question answering method based on multimodal information enhancement and interaction is provided. By enhancing visual features, the diversity and complexity of visual data are increased, thereby increasing the richness of information and alleviating overfitting; the recursive answer generation paradigm is used to provide autoregressive prompts for questions, and the text question answering module is used to provide single model prompts. With the assistance of the two prompts, the model's ability to understand questions is improved; the proposed answer generation model based on triple representation uses enhanced multimodal features to generate accurate answers, which significantly improves the accuracy of answer generation.

[0006] A method for external knowledge visual question answering based on multimodal information enhancement and interaction, the method comprising:

[0007] Inputting the original image into the target detector, extracting each image block and the corresponding label text in the original image, and inputting the image block and the label text into the multimodal feature extractor to obtain the visual features with semantic information corresponding to each target in the original image;

[0008] The original question is segmented and characterized to obtain question text embedding, and the visual features with semantic information and the question text embedding are input into the LXMERT model to obtain comprehensive visual features, text features, and multimodal features;

[0009] The comprehensive visual features and text features are processed based on the tensor Tucker fusion method to calculate a similarity matrix, and a predicted answer to the visual question and answer is calculated based on the similarity matrix and the multimodal features as an autoregressive prompt;

[0010] Extracting an image description from the original image based on the VinVL model, and inputting the image description and the original question into a single-model text question-answering module to obtain an answer prediction as a single-modal prompt;

[0011] The autoregressive prompt and the unimodal prompt are connected to the original question, and are used as a new question and are again segmented and characterized, and the final answer is calculated based on the newly obtained similarity matrix.

[0012] In one embodiment, the step of inputting the original image into the object detector and extracting each image block and the corresponding label text in the original image includes:

[0013] Using Faster-RCNN to extract each target in the original image, obtain a bounding box and a label text corresponding to each target, and determine an image block based on the bounding box;

[0014] A corresponding relationship between the image block and the label text is established.

[0015] In one embodiment, the method further comprises:

[0016] Using visual feature descent to mask the visual features with semantic information corresponding to different targets in the original image with target probability;

[0017] The visual features with semantic information are replaced with features extracted from other feature extractors using the target probability by visual feature exchange.

[0018] In one embodiment, the word segmentation and characterization processing of the original question to obtain question text embedding includes:

[0019] Apply WordPiece to the original question to perform word segmentation processing to obtain a question sequence;

[0020] The question sequence is input into the pre-trained LXMERT model to obtain question text embedding.

[0021] In one embodiment, the tensor-based Tucker fusion method is used to process the comprehensive visual features and text features to calculate a similarity matrix, including:

[0022] Based on the tensor Tucker fusion method, tensor multiplication between the comprehensive visual features and the text features is performed, and the similarity matrix between the target and the word token in the question sequence is obtained by summing along the fourth dimension;

[0023] The method further comprises:

[0024] Using row-by-row maximum pooling on the similarity matrix to evaluate the relevance between each target in the target set and the question, and obtaining an evaluation result;

[0025] According to the evaluation result, the target in the target set that is most relevant to the question is selected as the visual representation of the question, and based on the visual representation, a head entity is determined.

[0026] In one embodiment, the method further comprises:

[0027] Using the LXMERT model, a cross-modal representation is extracted from the question sequence;

[0028] The paramodal representation is input into a feed-forward network layer to derive the embedding of the relation as a relation entity.

[0029] In one embodiment, the method further comprises:

[0030] The head entity and the relationship entity are processed, and tail entity embedding is calculated to serve as a predicted answer for visual question answering.

[0031] In one embodiment, the image description and the original question are input into a single-model text question-answering module to obtain an answer prediction as a single-modal prompt, including:

[0032] Input the image description and the original question into a single model text question answering module, and obtain a reference answer and a prediction probability corresponding to the reference answer by representing the answer prediction model through a triple;

[0033] The reference answer and the predicted probability are used as unimodal prompts.

[0034] In one embodiment, the autoregressive prompt and the unimodal prompt are connected to the original question as a new question and are subjected to word segmentation and characterization again, and a final answer is obtained by calculating based on the newly obtained similarity matrix, including:

[0035] Perform word segmentation and characterization processing on the new question to obtain a new question text embedding, and input the visual features with semantic information and the new question text embedding into the LXMERT model to obtain new comprehensive visual features, text features, and multimodal features;

[0036] The new comprehensive visual features and text features are processed based on the tensor Tucker fusion method to calculate a new similarity matrix, and the final answer to the visual question and answer is calculated based on the new similarity matrix and the new multimodal features.

[0037] The above-mentioned external knowledge visual question answering method based on multimodal information enhancement and interaction uses the original question to generate autoregressive prompts and uses unimodal prompts based on image description and original question to improve the information richness of question text; adopts tensor Tucker fusion method for feature interaction to capture the correlation between target and question; connects autoregressive prompts and unimodal prompts with the original question, and uses recursive answer generation paradigm to strengthen the consistency of the two stages, so that the final generated answer has a higher accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is an application environment diagram of an external knowledge visual question answering method based on multimodal information enhancement and interaction in one embodiment;

[0039] Figure 2 1 is a flow chart of an external knowledge visual question answering method based on multimodal information enhancement and interaction in one embodiment;

[0040] Figure 3 A network architecture of an external knowledge visual question answering method based on multimodal information enhancement and interaction in one embodiment;

[0041] Figure 4 A schematic diagram of comparative data results of an external knowledge visual question answering method MIAI based on multimodal information enhancement and interaction and other advanced visual question answering algorithms in one embodiment;

[0042] Figure 5 A schematic diagram of test data results of the effectiveness of each module in an embodiment;

[0043] Figure 6 Schematic diagram of comparative data results of different feature enhancement methods;

[0044] Figure 7 It is a schematic diagram of comparative data results of the impact of autoregressive prompts at different stages on the model;

[0045] Figure 8 A schematic diagram of the data results showing the impact of using prompts during training;

[0046] Fig. 9 This is a schematic diagram of the data results of the impact of alignment loss on the model;

[0047] Fig.10 Schematic diagram of the visual effects of different models;

[0048] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The external knowledge visual question answering method based on multimodal information enhancement and interaction provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment includes a computer device 110. The computer device 110 can input the original image into the target detector, extract each image block and the corresponding label text in the original image, and input the image block and the label text into the multimodal feature extractor to obtain the visual features with semantic information corresponding to each target in the original image; the computer device 110 can perform word segmentation and characterization on the original question to obtain the question text embedding, and input the visual features with semantic information and the question text embedding into the LXMERT model to obtain comprehensive visual features, text features, and multimodal features; the computer device 110 can be based on the tensor Tucker The fusion method processes the comprehensive visual features and text features, calculates the similarity matrix, and calculates the predicted answer of the visual question answering based on the similarity matrix and multimodal features as an autoregressive prompt; the computer device 110 can extract the image description in the original image based on the VinVL model, input the image description and the original question into the single model text question answering module to obtain the answer prediction as a single modal prompt; the computer device 110 can establish a connection between the autoregressive prompt and the single modal prompt and the original question as a new question and perform word segmentation and characterization again, and calculate the final answer based on the newly obtained similarity matrix. Among them, the computer device 110 can be, but not limited to, various personal computers, laptops, robots, unmanned aerial vehicles, tablet computers and other devices.

[0051] In one embodiment, Figure 2 As shown, a method for external knowledge visual question answering based on multimodal information enhancement and interaction is provided, comprising the following steps:

[0052] Step 202, input the original image into the target detector, extract each image block and the corresponding label text in the original image, and input the image block and the label text into the multimodal feature extractor to obtain the visual features with semantic information corresponding to each target in the original image.

[0053] In one embodiment, a method for external knowledge visual question answering based on multimodal information enhancement and interaction is provided, which may also include a process of extracting image blocks and label texts. The specific process includes: using Faster-RCNN to extract each target in the original image, obtaining a bounding box and label text corresponding to each target, and determining the image block based on the bounding box; establishing a corresponding relationship between the image block and the label text.

[0054] Specifically, the computer device can use Faster-RCNN to extract the target in image I Each target contains a visual feature vector and spatial eigenvectors The extracted spatial feature vector can be used to determine the bounding box of the target, thereby determining the image block. Among them, one image block corresponds to one label text. Then, the computer device can use the multimodal feature extractor Clip to obtain the semantic feature embedding for each target based on the image block and the label text.

[0055] In one embodiment, a method for visual question answering based on external knowledge of multimodal information enhancement and interaction can also include feature enhancement using two methods: Visual Features Drop (VFD) and Visual Features Swap (VFS). Specifically, it includes: using visual features drop to mask the visual features with semantic information corresponding to different targets in the original image with target probability; using visual features swap to replace the visual features with semantic information with features extracted from other feature extractors with target probability.

[0056] In this embodiment, the visual features with semantic information are represented as VFD masks the visual features corresponding to different targets in the same image with a certain probability: droped ={f i ∈F|Bernoulli(1-P drop )=1}; where P drop represents the probability of each target being deleted, Bernoulli (1-P drop ) is a random variable that follows a Bernoulli distribution, representing the characteristic f i The probability of being retained is (1-P drop ), the probability of being deleted is P drop The tail entity in the context of an image-question-answer example is designated as the answer, encapsulating specific aspects of the factual information of the visual object referenced in the question. VFS probabilistically replaces the features of the target with those extracted from other feature extractors. This increases the overall characteristic scale:

[0057] Step 204, the original question is segmented and characterized to obtain question text embedding, and the visual features with semantic information and question text embedding are input into the LXMERT model to obtain comprehensive visual features, text features, and multimodal features.

[0058] Specifically, in one embodiment, a method for visual question answering based on external knowledge of multimodal information enhancement and interaction may also include a process of word segmentation of the original question, and the specific process includes: applying WordPiece to the original question for word segmentation to obtain a question sequence; inputting the question sequence into a pre-trained LXMERT model to obtain question text embedding.

[0059] WordPiece is applied to the original question to obtain a question sequence consisting of D tokens. In this embodiment, the visual features with semantic information of a batch of images can be The question text embedding token is input into the pre-trained LXMERT to obtain the target visual embedding tensor and the question text embedding tensor Where bs represents the batch size, bs represents the number of objects in the image, and len q Indicates the question length.

[0060] Step 206, based on the tensor Tucker fusion method, the comprehensive visual features and text features are processed to calculate a similarity matrix, and the predicted answer of the visual question answering is calculated based on the similarity matrix and multimodal features as an autoregressive prompt.

[0061] In one embodiment, a method for visual question answering based on external knowledge based on multimodal information enhancement and interaction may also include a process for calculating a similarity matrix, the specific process including: performing tensor multiplication between comprehensive visual features and text features based on the tensor Tucker fusion method, and summing along the fourth dimension to obtain a similarity matrix between word tokens in the target and question sequence.

[0062] Specifically, it can be based on h core tensors Perform tensor multiplication between the target visual embedding tensor V and the question text embedding tensor Q, and sum along the fourth dimension to get the similarity matrix between the target and word tokens:

[0063] In one embodiment, a method for visual question answering of external knowledge based on multimodal information enhancement and interaction is provided, which may also include a process for extracting a head entity. The specific process includes: using row-by-row maximum pooling on a similarity matrix to evaluate the relevance of each target in a target set to a question, and obtaining an evaluation result; based on the evaluation result, selecting a target in the target set that has the greatest relevance to the question as a visual representation of the question, and determining a head entity based on the visual representation.

[0064] The row-wise maximum pooling is used on the similarity matrix A to evaluate each target o i The relevance to the problem is to select the single target with the largest relevance to the problem in the target set O as the most relevant visual representation of the problem: Then, based on the matrix The Gumbel-Softmax method is used to approximate the one-hot classification distribution, and hard attention is used to designate the most relevant object as the head entity. i The attention weight is calculated as follows: in, is a sample extracted from Gumbel(0,1), and τ is the "temperature" parameter in softmax. Finally, we aggregate information about the central object in the question and derive a representation of the head entity h: Among them, v i ∈V and FFN denotes a feed-forward network consisting of two fully connected layers.

[0065] In one embodiment, a method for visual question answering based on external knowledge of multimodal information enhancement and interaction may also include a process of extracting relational entities, the specific process including: using the LXMERT model to extract cross-modal representations from the question sequence; inputting the cross-modal representations into the feedforward network layer to derive the embedding of the relationship as the relational entity.

[0066] Specifically, LXMERT can be used to extract cross-modal representations from the [CLS] token, and then input these representations into the feed-forward network (FFN) layer to derive the embedding of the relation, denoted as the relation entity embedding r.

[0067] In one embodiment, the computer device may also process the head entity and the relationship entity, and calculate the tail entity embedding as a predicted answer for the visual question answering.

[0068] Step 208: extract the image description in the original image based on the VinVL model, input the image description and the original question into the single-model text question-answering module to obtain the answer prediction as a single-modal prompt.

[0069] Specifically, the computing device may use the image content description model VinVL to provide an image description S of the content in the image I.I , and then use the single-model text question-answering module CBM-large to get the answer prediction as a single-modal prompt.

[0070] In one embodiment, a computer device may input an image description and an original question into a single-model text question-answering module, represent an answer prediction model through triples, obtain a reference answer and a prediction probability corresponding to the reference answer, and use the reference answer and the prediction probability as a single-modal prompt.

[0071] The process of obtaining a unimodal prompt may be the process of obtaining an answer in the first stage. Specifically, in the first stage, only the original question and visual features are used to generate an answer to obtain a potential correct answer: in, Indicates the first stage n th The answer is P n represents the corresponding probability, TR represents the triplet representing the answer prediction model, and F denote the question and the corresponding visual features in the first stage, respectively.

[0072] Step 210, connect the autoregressive prompt and the unimodal prompt with the original question, treat them as new questions, perform word segmentation and characterization again, and calculate the final answer based on the newly obtained similarity matrix.

[0073] In one embodiment, the process of performing autoregressive prompting may be a process of obtaining an answer in the second stage, and the computer device may connect the top K predictions with high probabilities with the question text as autoregressive prompts for generating a second round of answers.

[0074] In one embodiment, a method for visual question answering based on external knowledge based on multimodal information enhancement and interaction is provided, which may also include a process of re-predicting answers based on autoregressive prompts and unimodal prompts. The specific process includes: segmenting and characterizing new questions to obtain new question text embeddings, and inputting visual features with semantic information and new question text embeddings into the LXMERT model to obtain new comprehensive visual features, text features, and multimodal features; processing the new comprehensive visual features and text features based on the tensor Tucker fusion method to calculate a new similarity matrix, and calculating the final answer to the visual question answering based on the new similarity matrix and the new multimodal features.

[0075] In one embodiment, a network architecture of a visual question answering method based on external knowledge enhanced and interactive multimodal information is provided as follows: Figure 3As shown in FIG. 1 , the external knowledge visual question answering method based on multimodal information enhancement and interaction in this embodiment can be used as an overall objective function model, and the external knowledge visual question answering method based on multimodal information enhancement and interaction can be implemented by training the objective function model. During the training process, the visual enhancement features of the image and the questions processed by the triple-based answer prediction model are initially used to obtain the first stage tail entity embedding Subsequently, the real answer is concatenated with the question, and the visual enhancement features are again subjected to the triple answer prediction model to obtain the second-stage tail entity embedding. set up and They represent the correct answer (positive sample) and the wrong answer (negative sample), respectively, and the TranE loss commonly used in the traditional knowledge graph field is adopted: in,[.] + =max(0,.), d(.) represents the cosine distance, and the triplet loss calculation method can be expressed as: L Tri =MSE(h+r,t + ), where MSE stands for mean square error; the alignment loss calculation method can be expressed as: The overall training objective is obtained by weighting the TransE loss, triplet loss, and alignment loss to train the model: L total =λ1L TranE +λ2L Tri +λ3L Align .

[0076] In the answer prediction reasoning stage, answer prediction can be regarded as a multimodal knowledge graph completion problem. Specifically, an image and a question can be given to the network, and the embedding of the head entity h and the relation r is obtained, and the h+r is calculated with each tail entity t in the lookup table T. i The distance between Select the tail entity with the smallest distance, and select the answer corresponding to the optimal tail entity t as the predicted answer.

[0077] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0078] In one embodiment, this application relies on the multimodal knowledge triple representation mode and takes advantage of the tensor's advantage in affinity representation to study a visual question-answering model that strengthens the representation, accumulation and injection of multimodal knowledge, that is, an external knowledge visual question-answering method based on multimodal information enhancement and interaction. It not only improves the diversity of visual features in visual question-answering scenarios, but also improves the utilization rate of question texts. At the same time, it strengthens the consistency of feature space of knowledge representation in the two answer generation stages, enhances the understanding of external knowledge, and improves the accuracy of answer prediction. Compared with other baseline models, multiple evaluation indicators on commonly used data sets have been improved to a certain extent.

[0079] The specific experimental process and experimental results are as follows:

[0080] OKVQA is used as the dataset for evaluating this application. OK-VQA contains more than 14,000 questions covering 10 knowledge categories. This application provides an external knowledge visual question answering method MIAI based on multimodal information enhancement and interaction. The results on the OKVQA dataset are as follows compared with other advanced visual question answering algorithms: Figure 4 As shown, it can be seen that the external knowledge visual question answering method MIAI based on multimodal information enhancement and interaction provided by this application has certain improvements compared with many methods.

[0081] like Figure 5 As shown, the two methods of visual feature drop (VFD) and visual feature swap (VFS) proposed in this application for feature enhancement are more effective than other methods and can improve visual question answering to a certain extent.

[0082] like Figure 6As shown in the figure, compared with other methods, the Visual Features Drop (VFD) and Visual Features Swap (VFS) methods increase the complexity of the training data set and solve the overfitting problem.

[0083] The accuracy data of the impact of autoregressive prompts on the model at different stages are as follows Figure 7 As shown, autoregressive hints can improve the performance of visual question answering when the stage is small.

[0084] The external knowledge visual question answering method based on multimodal information enhancement and interaction provided in this application uses unimodal prompts and autoregressive prompts when training the model, such as Figure 8 As shown, using hints during training can improve the accuracy of visual question answering.

[0085] like Fig. 9 As shown in Figure 2, using alignment loss during training can improve the accuracy of visual question answering.

[0086] In this embodiment, the prediction results of the MIA-RAG and MuKEA models based on multimodal information enhancement and interaction are visualized as follows: Fig.10 As shown, it can be seen that for some problems where the MuKEA model makes incorrect predictions, the external knowledge visual question answering method MIA-RAG based on multimodal information enhancement and interaction provided in this application can also generate correct answers.

[0087] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method of visual question answering of external knowledge based on multimodal information enhancement and interaction is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0088] Those skilled in the art will understand that Fig.11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the external knowledge visual question answering method based on multimodal information enhancement and interaction are implemented.

[0090] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the external knowledge visual question answering method based on multimodal information enhancement and interaction are implemented.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A visual question answering method based on external knowledge enhancement and interaction of multimodal information, characterized in that: The method comprises: Inputting the original image into the target detector, extracting each image block and the corresponding label text in the original image, and inputting the image block and the label text into the multimodal feature extractor to obtain the visual features with semantic information corresponding to each target in the original image; The original question is segmented and characterized to obtain question text embedding, and the visual features with semantic information and the question text embedding are input into the LXMERT model to obtain comprehensive visual features, text features, and multimodal features; The comprehensive visual features and text features are processed based on the tensor Tucker fusion method to calculate a similarity matrix, and a predicted answer to the visual question and answer is calculated based on the similarity matrix and the multimodal features as an autoregressive prompt; Extracting an image description from the original image based on the VinVL model, and inputting the image description and the original question into a single-model text question-answering module to obtain an answer prediction as a single-modal prompt; The autoregressive prompt and the unimodal prompt are connected with the original question as a new question and are segmented and characterized again, and a final answer is calculated based on a newly obtained similarity matrix, including: segmenting and characterizing the new question to obtain a new question text embedding, and inputting the visual features with semantic information and the new question text embedding into the LXMERT model to obtain new comprehensive visual features, text features, and multimodal features; processing the new comprehensive visual features and text features based on the tensor Tucker fusion method to obtain a new similarity matrix, and calculating the final answer to the visual question and answer based on the new similarity matrix and the new multimodal features.

2. The external knowledge visual question answering method based on multimodal information enhancement and interaction according to claim 1 is characterized in that: The step of inputting the original image into the target detector and extracting each image block and the corresponding label text in the original image includes: Using Faster-RCNN to extract each target in the original image, obtain a bounding box and a label text corresponding to each target, and determine an image block based on the bounding box; A corresponding relationship between the image block and the label text is established.

3. The external knowledge visual question answering method based on multimodal information enhancement and interaction according to claim 1 is characterized in that: The method further comprises: Using visual feature descent to mask the visual features with semantic information corresponding to different targets in the original image with target probability; The visual features with semantic information are replaced with features extracted from other feature extractors using the target probability by visual feature exchange.

4. The external knowledge visual question answering method based on multimodal information enhancement and interaction according to claim 1 is characterized in that: The word segmentation and characterization processing of the original question to obtain the question text embedding includes: Apply WordPiece to the original question to perform word segmentation processing to obtain a question sequence; The question sequence is input into the pre-trained LXMERT model to obtain question text embedding.

5. The external knowledge visual question answering method based on multimodal information enhancement and interaction according to claim 4 is characterized in that: The tensor-based Tucker fusion method processes the comprehensive visual features and text features to calculate a similarity matrix, including: Based on the tensor Tucker fusion method, tensor multiplication between the comprehensive visual features and the text features is performed, and the similarity matrix between the target and the word token in the question sequence is obtained by summing along the fourth dimension; The method further comprises: Using row-by-row maximum pooling on the similarity matrix to evaluate the relevance between each target in the target set and the question, and obtaining an evaluation result; According to the evaluation result, the target in the target set that is most relevant to the question is selected as the visual representation of the question, and based on the visual representation, a head entity is determined.

6. The external knowledge visual question answering method based on multimodal information enhancement and interaction according to claim 5 is characterized in that: The method further comprises: Using the LXMERT model, a cross-modal representation is extracted from the question sequence; The paramodal representation is input into a feed-forward network layer to derive the embedding of the relation as a relation entity.

7. The external knowledge visual question answering method based on multimodal information enhancement and interaction according to claim 6 is characterized in that: The method further comprises: The head entity and the relationship entity are processed, and tail entity embedding is calculated to serve as a predicted answer for visual question answering.

8. The external knowledge visual question answering method based on multimodal information enhancement and interaction according to claim 1 is characterized in that: Inputting the image description and the original question into a single-model text question-answering module to obtain an answer prediction as a single-modal prompt includes: Input the image description and the original question into a single model text question answering module, and obtain a reference answer and a prediction probability corresponding to the reference answer by representing the answer prediction model through a triple; The reference answer and the predicted probability are used as unimodal prompts.

Images