Visual question-answering method and device based on multiple modes, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the fields of financial science and technology and medical health, and discloses a multi-modality-based visual question and answer method, device, equipment and medium, and the method comprises the steps: obtaining an input image and a question, and inputting the image and the question into a visual language coding model for coding to generate a multi-modality feature; inputting the multi-modal features into an answer reasoning module to generate answers and deep reasoning features corresponding to the answers; inputting the multi-modal features into a basic principle generation module to generate guide features, and inputting the guide features into a large language model to generate a text basic principle; extracting features of the text basic principle to obtain context features, and generating a visual basic principle through an object detector according to the context features, the image and the depth reasoning features; and taking the visual basic principle, the text basic principle and the answers as final interpretable answers. And the transparency and the credibility of the visual question-answering system are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and can be applied to the fields of fintech and healthcare, particularly to a multimodal visual question answering method, apparatus, device, and medium. Background Technology

[0002] Visual Question Answering (VQA) systems (such as loan application review in fintech and medical image-assisted diagnosis in healthcare) are a cutting-edge technology in artificial intelligence, aiming to enable computers to understand image content and accurately answer related natural language questions. These systems are typically built on complex deep neural networks, especially architectures like the Transformer. While existing VQA models demonstrate significant accuracy, they generally suffer from a fundamental flaw—their decision-making process is opaque, becoming a "black box." Users cannot determine whether the model truly understands the visual and textual content or merely relies on statistical biases in the data to arrive at the correct answer. This lack of interpretability severely limits the application of VQA models in high-risk scenarios such as medical diagnosis, autonomous driving, and forensic investigation.

[0003] Existing interpretable visual question answering methods in visual question answering systems mainly fall into the following categories, but all have significant shortcomings: 1. Providing only visual explanations: For example, generating "heatmaps" (using color depth to represent areas of interest for the model) to coarsely display the distribution of the model's attention. However, these methods cannot accurately identify specific objects, and the explanations are often vague and lack detail; 2. Providing only text explanations: Generating a natural language description to explain the reasoning process. Although highly readable, these methods lack a direct correspondence with visual elements in the image, making it difficult for users to verify whether the description accurately reflects the image content; 3. Methods relying on external knowledge: For example, using external knowledge bases such as "scene graphs" (pre-built object relationship graphs) to assist in explanation. These methods are not only costly but also severely limit the model's generalization ability in open environments, making it difficult to adapt to practical application needs. Summary of the Invention

[0004] This invention provides a multimodal visual question answering method, apparatus, computer equipment, and medium to solve the technical problem that existing visual question answering systems have low transparency and reliability.

[0005] Firstly, a multimodal visual question answering method is provided, applied to a visual question answering system. The visual answering model in the system includes a visual language encoding model, an answer reasoning module, a basic principle generation module, a large-scale language model, and an object detector. The method includes: The input image and question are acquired, and the image and question are input into the visual language encoding model for encoding to generate multimodal features; The multimodal features are input into the answer reasoning module to generate an answer and the corresponding deep reasoning features; The multimodal features are input into the basic principle generation module to generate guiding features, and the guiding features are input into the large language model to generate text basic principles; The textual basic principle features are extracted to obtain contextual features, and the visual basic principle is generated by the object detector based on the contextual features, the image, and the deep reasoning features. The visual fundamentals, the textual fundamentals, and the answer are taken as the final interpretable response.

[0006] Secondly, a multimodal visual question-answering device is provided for use in a visual question-answering system. The visual answer model in the system includes a visual language encoding model, an answer reasoning module, a basic principle generation module, a large-scale language model, and an object detector. The device comprises: An encoding unit is used to acquire the input image and question, and input the image and question into the visual language encoding model for encoding to generate multimodal features; The first input generation unit is used to input the multimodal features into the answer reasoning module to generate an answer and deep reasoning features corresponding to the answer; The second input generation unit is used to input the multimodal features into the basic principle generation module to generate guiding features, and input the guiding features into the large language model to generate text basic principles; An extraction and generation unit is used to extract features of the textual basic principle to obtain contextual features, and generate a visual basic principle through the object detector based on the contextual features, the image, and the deep reasoning features. As a unit, it is used to take the visual fundamental principle, the textual fundamental principle, and the answer as the final interpretable response.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal visual question answering method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described multimodal visual question-answering method.

[0009] In the above-described scheme based on multimodal visual question answering methods, devices, computer equipment, and storage media, the input image and question can be acquired, and the image and question can be input into the visual language encoding model for encoding to generate multimodal features; the multimodal features can be input into the answer reasoning module to generate the answer and the deep reasoning features corresponding to the answer; the multimodal features can be input into the basic principle generation module to generate guiding features, and the guiding features can be input into the large language model to generate textual basic principles; the features of the textual basic principles can be extracted to obtain contextual features, and the visual basic principles can be generated through the object detector based on the contextual features, the image, and the deep reasoning features; the visual basic principles, the textual basic principles, and the answer can be used as the final interpretable answer. In this invention, images and questions are first encoded into multimodal features using a visual language encoding model. Then, based on these multimodal features, an answer reasoning module generates the answer and corresponding deep reasoning features, while a basic principle generation module and a large language model generate textual basic principles. Finally, based on the contextual features, images, and deep reasoning features extracted from the textual basic principles, a visual basic principle is generated using an object detector. The visual basic principle, textual basic principle, and answer are used as the final interpretable response, constructing a complete multimodal evidence chain. This makes the decision-making process of the visual question-answering model clear and traceable, effectively improving the transparency and credibility of the visual question-answering system. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating a multimodal visual question answering method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 3 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The multimodal visual question answering method provided in this invention can be applied to either a client or a server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. Currently, in the fintech and healthcare fields, existing visual question answering systems have low transparency and credibility. To address these issues, this invention proposes a multimodal visual question answering method. This method first encodes images and questions into multimodal features using a visual language encoding model. Then, based on these multimodal features, an answer reasoning module generates the answer and corresponding deep reasoning features, while a basic principle generation module and a large language model generate textual basic principles. Finally, based on the contextual features, images, and deep reasoning features extracted from the textual basic principles, an object detector generates visual basic principles. The visual basic principles, textual basic principles, and the answer are used as the final interpretable response, constructing a complete multimodal evidence chain. This makes the decision-making process of the visual question answering model clear and traceable, effectively improving the transparency and credibility of the visual question answering system. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 1 As shown, Figure 1 A flowchart of a multimodal visual question answering method provided in an embodiment of the present invention includes the following steps: S110-S150.

[0015] S110. Obtain the input image and question, and input the image and question into the visual language encoding model for encoding to generate multimodal features.

[0016] Specifically, a multimodal visual question answering method is applied to a visual question answering system. The visual answering model in the visual question answering system includes a visual language encoding model, an answer reasoning module, a basic principle generation module, a large language model, and an object detector. The question is a natural language question. The visual language encoding model converts the original inputs of different modalities (images composed of pixels and questions composed of characters) into a unified, high-dimensional digital representation (i.e., multimodal features) that can be understood and processed by the machine. The core advantage of the visual language encoding model is that it has been trained on billions of image-text pairs of data, and can map images and texts to the same shared semantic space. In the output multimodal features, images and texts that are semantically similar are also located in similar positions in space. It should be noted that the visual question answering system can be a system in the financial technology field, such as a loan application material review system, or a system in the medical and health field, such as a medical image-assisted diagnosis system. It should also be noted that during the model training process, the model corresponding to the minimum text alignment loss value calculated by the text alignment loss function is taken as the visual question answering model. The text alignment loss function is shown in formula (1): (1); In formula (1), Here, M represents the text alignment loss value, and M is the number of samples included in a single training batch. This represents the feature vector of the core reasoning state of the "answer reasoning path" before reaching the final conclusion when processing the i-th sample; Let represent the feature vector of the core context state of the "fundamental principle generation path" driving the large language model to generate text when processing the i-th sample. cos represents the cosine similarity function, which measures whether the directions of two feature vectors are consistent in multidimensional space. The closer the directions of the two vectors are, the closer the function value is to 1, and the closer the text alignment loss value is to 0. This mechanism ensures that the final answer and explanation given by the model are "homogeneous," meaning they originate from the same internal, unified understanding, by continuously ensuring that the core state vectors of the two paths maintain consistency in direction during training. The "answer reasoning path" and the "fundamental principle generation path" will be explained later; for simplicity, they will not be elaborated upon here.

[0017] Step S110 includes the following steps: S111-S112: S111. Perform image preprocessing on the image to generate an image token sequence, and perform text preprocessing on the question to generate a text token sequence; S112. Input the image token sequence and the text token sequence into the visual language encoding model, so as to map the image token sequence and the text token sequence to the same high-dimensional semantic space through the cross-modal attention mechanism in the visual language encoding model, and generate the multimodal features.

[0018] Specifically, the input image undergoes image preprocessing, segmenting it into fixed-size image patches and converting them into image token sequences. Simultaneously, the input natural language question is segmented and embedded to generate a text token sequence. The image and text token sequences are then input into a pre-trained visual language encoding model. A cross-modal attention mechanism within the model enables bidirectional modal interaction: an image self-attention layer captures spatial relationships between visual elements, a text self-attention layer establishes semantic dependencies between words, and a cross-attention layer establishes visual-text semantic alignment. After multi-layer Transformer block multimodal fusion computation, the image and text token sequences are mapped to the same high-dimensional semantic space, ultimately outputting a unified multimodal feature containing closely related visual-text semantic information. It should be noted that for text, a token can be a word or a character; for images, a token can be a small image patch.

[0019] S120. Input the multimodal features into the answer reasoning module to generate the answer and the deep reasoning features corresponding to the answer.

[0020] Specifically, the answer reasoning module is an encoder-decoder module based on a collaborative attention mechanism. Its function is to deeply understand the complex relationship between the question and the image, perform logical reasoning, and ultimately predict the most likely answer. The encoder-decoder module based on the collaborative attention mechanism includes two attention mechanisms: a self-attention mechanism, used to understand the contextual relationships between different regions within the image and between different words within the question text; and a cross-attention mechanism, used to capture the correspondence between keywords in the question and specific regions in the image. It should be noted that the multimodal features include image features and text features.

[0021] Step S120 includes the following steps: S121-S123: S121. Input the multimodal features into the answer reasoning module to enhance the internal representation of the image features and the text features through a self-attention mechanism, and use a cross-attention mechanism to establish visual-language semantic alignment and generate a deeply fused joint representation. S122. Based on the joint representation, the classifier in the reasoning module outputs the answer probability distribution, and the final answer is determined according to the answer probability distribution; S123. Extract the hidden state most relevant to the answer as the deep reasoning feature.

[0022] Specifically, after the multimodal features are input into the encoder-decoder module based on the collaborative attention mechanism, the internal context of image features and text features is first modeled using a self-attention mechanism: the image self-attention layer captures the spatial relationships and visual dependencies between different image regions, and the text self-attention layer establishes the semantic associations and grammatical structures between question words; subsequently, a cross-attention mechanism is used to achieve visual-linguistic semantic alignment, where text features serve as query vectors and image features serve as key-value pairs, calculating the attention weights between modalities to generate a deeply fused joint representation. This joint representation fully integrates the interactive information of image semantics and linguistic semantics. Based on the deeply fused joint representation, the answer probability distribution is calculated using a classifier (usually a fully connected layer and a Softmax activation function) in the inference module, and the answer with the highest probability value is selected as the final answer. Simultaneously, the hidden state vector most significantly related to the final answer is extracted from the output layer of the cross-attention mechanism. This vector encodes the key visual and textual evidence on which the model relies for decision-making, and is used as a deep inference feature.

[0023] S130. Input the multimodal features into the basic principle generation module to generate guiding features, and input the guiding features into the large language model to generate text basic principles.

[0024] Specifically, the fundamental principle generation module is a Transformer-based projection module. This module transforms and adapts common multimodal features, making them suitable "instructions" or "hints" to drive large-scale language models in text creation. In other words, the Transformer-based projection module acts as an "adapter," fine-tuning and reorganizing features through multi-layered self-attention computation to make their format and content more consistent with the requirements of subsequent text fundamental principles. It should be noted that the generation paths of answer and deep reasoning features are considered the "answer reasoning path," while the guidance feature generation path is considered the "fundamental principle generation path." These two paths run in parallel. Furthermore, the guidance features are input as conditions into the large-scale language model. This large-scale language model is an auto-regressive model, meaning it generates text character by character (or word by word), referencing all previously generated text when generating the next character. This mechanism ensures that the generated text's fundamental principles are fluent, logically coherent, and logically sound.

[0025] Step S130 includes the following steps: S131-S134: S131. Input the multimodal features into the basic principle generation module to enhance the internal semantic association of the multimodal features using a self-attention mechanism, and reconstruct structured features that conform to the input specifications of the large language model. S132. Project the structured features onto the embedding space of the large language model to generate the guiding features containing visual semantic information; S133. The guiding features are input as conditions into the large language model to deeply fuse with the text query vector in the large language model to generate a contextual representation with enhanced visual semantics. S134. Based on the context representation, each word is generated sequentially using an autoregressive approach. The correlation between the current sequence and the guiding feature is calculated iteratively, and the basic principle of the text that conforms to visual semantic constraints is output.

[0026] Specifically, after the multimodal features are input into the basic principle generation module, the multimodal features are first processed deeply through the self-attention mechanism in the multi-layer Transformer structure to enhance the semantic association and dependency between image features and text features, and reconstruct structured features that conform to the input specifications of large language models. Then, the structured features are mapped to the embedding space of the large language model through a linear projection layer to generate guiding features containing rich visual semantic information. These guiding features can serve as conditional signals to guide the subsequent text generation process. The guiding features are input as key-value pair conditions into the decoder of the large language model and cross-attention is performed with the model's implicit text query vector (i.e., the hidden state of the generated text sequence) to achieve deep fusion of visual semantic information and text context, generating a visually semantically enhanced contextual representation. Based on this enhanced contextual representation, the probability distribution of the next word is predicted sequentially using an autoregressive generation method: long-distance text dependencies are captured through the self-attention mechanism, while the semantic association with the guiding features is maintained through cross-attention. Each word is iteratively generated until the end-of-sequence marker appears, and finally, a textual basic principle that is highly consistent with the visual content and logically coherent is output. The entire process ensures that the text generation conforms to the grammatical rules of a large language model and is strictly constrained by visual semantic information, thus forming an accurate and reliable multimodal interpretation.

[0027] S140. Extract the features of the text basic principle to obtain context features, and generate visual basic principles through the object detector based on the context features, the image, and the deep reasoning features.

[0028] Specifically, the generated text fundamentals are input into an encoder (such as a Transformer encoder or Bi-LSTM), which captures semantic dependencies between words through a self-attention mechanism, generating context-aware hidden states for each word. Finally, key hidden states are aggregated or selected as contextual features representing the semantics of the entire text. After obtaining the contextual features, a visual fundamental is generated using the object detector based on the contextual features, the image, and the deep inference features.

[0029] Step S140 includes the following steps: S141-S143: S141. The context features and the deep reasoning features are concatenated and linearly transformed to generate the multimodal conditional vector; S142. Input the image and the multimodal conditional vector as prior knowledge into the object detector to enhance the regional features in the image features corresponding to the image that are related to the text description and reasoning results; S143. Based on the enhanced image features, predict the target bounding box to filter out the visual entities most relevant to the context features and the depth inference, and output the visual basic principle labeled with the target bounding box.

[0030] Specifically, the context features and the deep reasoning features are concatenated to form a combined feature vector. Then, a linear transformation layer (such as a fully connected network) is used to adjust the dimensions and perform semantic fusion on the combined feature vector, generating a unified multimodal conditional vector. This vector simultaneously encodes the semantic information of the text explanation and the logical information of the answer reasoning. The original image and the multimodal conditional vector are input as prior knowledge into the object detector. Through the conditional attention mechanism in the object detector, the multimodal conditional vector is used as a query guidance signal, and its association weight with each spatial location in the image feature map is calculated. This enhances the representation of regional features closely related to the text description and reasoning results in the image features, suppressing interference from irrelevant background regions. Based on the attention-enhanced image features, candidate target bounding boxes and their category probability distributions are generated through the regression prediction head of the object detector. A non-maximum suppression algorithm is used to select the visual entities most relevant to the context features and deep reasoning features. Finally, the visual basic principle labeled with precise coordinate target bounding boxes is output, completing the visualization, localization, and presentation of key visual evidence.

[0031] S150, The visual fundamental principle, the textual fundamental principle, and the answer are taken as the final interpretable response.

[0032] Specifically, the visual fundamental principles (key visual evidence precisely labeled with bounding boxes), the textual fundamental principles (natural language explanations describing the reasoning process), and the answer (direct response to the question) are collectively used as the final interpretable answer output, forming a multimodal complementary evidence chain. This provides users with a complete decision-making basis that combines visual localization, semantic description, and final conclusion, significantly improving the model's transparency and credibility.

[0033] To facilitate understanding of the multimodal visual question answering method in this invention, an example is given below: Suppose a user inputs an image containing three zebras, two of which are facing left and one facing right, and asks: "Are all the zebras facing the same direction?". Step 1: Multimodal feature representation: The visual question answering model receives the image and the question, and encodes them into a feature representation in a unified semantic space through a pre-trained visual language encoding model. This feature implicitly contains semantic information such as "zebras", "number of three", and "direction". Step 2: Dual-path parallel reasoning and feature adaptation: (1) Answer reasoning path: Through a cross-modal attention mechanism, the "same direction" in the question is compared with the orientation features of the three zebras in the image. The system identifies inconsistencies, outputs the answer "No", and generates a deep feature representation containing the reasoning logic; (2) Basic principle generation path: adapts and reconstructs the unified multimodal features, converting them into guiding features suitable for processing by large language models; Step 3, Multimodal basic principle generation: (1) Text generation: Based on the guiding features, the large language model generates the text basic principle: "A zebra is facing the opposite direction", and outputs the corresponding contextual semantic features; (2) Visual localization: The object detector integrates the original image, deep reasoning features and contextual features to accurately locate visual entities consistent with the text description, that is, draws a bounding box at the position of the zebra facing right, ignoring other zebras facing the same direction. It not only outputs accurate answers, but also provides mutually verifiable visual and textual explanations, significantly enhancing the interpretability and credibility of the visual question answering system, and providing a technical foundation for reliable artificial intelligence applications in high-risk fields.

[0034] The multimodal visual question answering method in this invention first encodes images and questions into multimodal features using a visual language encoding model. Then, based on these multimodal features, an answer reasoning module generates answers and corresponding deep reasoning features, while a basic principle generation module and a large language model generate textual basic principles. Finally, based on contextual features, images, and deep reasoning features extracted from the textual basic principles, an object detector generates visual basic principles. The visual basic principles, textual basic principles, and answers are used as the final interpretable response, constructing a complete multimodal evidence chain. This makes the decision-making process of the visual question answering model clear and traceable, effectively improving the transparency and credibility of the visual question answering system.

[0035] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0036] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0037] In one embodiment, a multimodal visual question-answering device 200 is provided, which corresponds one-to-one with the multimodal visual question-answering method described in the above embodiments. The multimodal visual question-answering device includes an acquisition encoding unit 201, a first input generation unit 202, a second input generation unit 203, an extraction generation unit 204, and an input processing unit 205. Detailed descriptions of each functional module are as follows: The encoding unit 201 is used to acquire the input image and question, and input the image and question into the visual language encoding model for encoding to generate multimodal features; The first input generation unit 202 is used to input the multimodal features into the answer reasoning module to generate an answer and deep reasoning features corresponding to the answer; The second input generation unit 203 is used to input the multimodal features into the basic principle generation module to generate guiding features, and input the guiding features into the large language model to generate text basic principles; The extraction and generation unit 204 is used to extract the features of the text basic principle to obtain context features, and generate the visual basic principle through the object detector based on the context features, the image and the deep reasoning features. Unit 205 is used to take the visual fundamental principle, the textual fundamental principle, and the answer as the final interpretable response.

[0038] In one embodiment, the encoding unit 201 is specifically used for: The image is preprocessed to generate an image token sequence, and the question is preprocessed to generate a text token sequence. The image token sequence and the text token sequence are input into the visual language encoding model to map the image token sequence and the text token sequence to the same high-dimensional semantic space through the cross-modal attention mechanism in the visual language encoding model, thereby generating the multimodal features.

[0039] In one embodiment, the first input generation unit 202 is specifically used for: The multimodal features are input into the answer reasoning module to enhance the internal representation of the image features and the text features through a self-attention mechanism, and a visual-language semantic alignment is established using a cross-attention mechanism to generate a deeply fused joint representation. Based on the joint representation, the classifier in the reasoning module outputs the probability distribution of the answer, and the final answer is determined according to the probability distribution of the answer. Extract the hidden state most relevant to the answer as the deep reasoning feature.

[0040] In one embodiment, the second input generation unit 203 is specifically used for: The multimodal features are input into the basic principle generation module to enhance the internal semantic association of the multimodal features using a self-attention mechanism, and reconstruct structured features that conform to the input specifications of the large language model. The structured features are projected into the embedding space of the large language model to generate the guiding features containing visual semantic information; The guiding features are input as conditions into the large language model to be deeply fused with the text query vectors in the large language model to generate a contextual representation with enhanced visual semantics. Based on the context representation, each word is generated sequentially using an autoregressive approach. The correlation between the current sequence and the guiding features is calculated iteratively, and the basic principle of the text that conforms to visual semantic constraints is output.

[0041] In one embodiment, the extraction and generation unit 204 is specifically used for: The context features and the deep inference features are concatenated and linearly transformed to generate the multimodal conditional vector; The image and the multimodal conditional vector are input into the object detector as prior knowledge to enhance the regional features in the image features corresponding to the image that are related to the text description and reasoning results; Based on the enhanced image features, predict the target bounding box to filter out the visual entities most relevant to the context features and the depth inference, and output the visual principle labeled with the target bounding box.

[0042] The multimodal visual question answering device of this invention first encodes images and questions into multimodal features through a visual language encoding model. Then, based on the multimodal features, an answer reasoning module generates answers and corresponding deep reasoning features, and a basic principle generation module and a large language model generate textual basic principles. Finally, based on the contextual features, images, and deep reasoning features extracted from the textual basic principles, a visual basic principle is generated through an object detector. The visual basic principle, textual basic principle, and answer are used as the final interpretable response, constructing a complete multimodal evidence chain. This makes the decision-making process of the visual question answering model clear and traceable, effectively improving the transparency and credibility of the visual question answering system.

[0043] Specific limitations regarding multimodal visual question-answering devices can be found in the limitations of multimodal visual question-answering methods described above, and will not be repeated here. Each unit in the aforementioned multimodal visual question-answering device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0044] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 2 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal visual question-answering method server-side.

[0045] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a multimodal visual question-answering method.

[0046] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multimodal visual question answering method described above.

[0047] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal-based visual question-answering method described above.

[0048] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0049] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0050] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0051] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal visual question answering method, applied to a visual question answering system, characterized in that, The visual response model in the visual question answering system includes a visual language encoding model, an answer reasoning module, a basic principle generation module, a large-scale language model, and an object detector. The method includes: The input image and question are acquired, and the image and question are input into the visual language encoding model for encoding to generate multimodal features; The multimodal features are input into the answer reasoning module to generate an answer and the corresponding deep reasoning features; The multimodal features are input into the basic principle generation module to generate guiding features, and the guiding features are input into the large language model to generate text basic principles; The textual basic principle features are extracted to obtain contextual features, and the visual basic principle is generated by the object detector based on the contextual features, the image, and the deep reasoning features. The visual fundamentals, the textual fundamentals, and the answer are taken as the final interpretable response.

2. The multimodal visual question answering method as described in claim 1, characterized in that, The step of inputting the image and the question into the visual language encoding model for encoding to generate multimodal features includes: The image is preprocessed to generate an image token sequence, and the question is preprocessed to generate a text token sequence. The image token sequence and the text token sequence are input into the visual language encoding model to map the image token sequence and the text token sequence to the same high-dimensional semantic space through the cross-modal attention mechanism in the visual language encoding model, thereby generating the multimodal features.

3. The multimodal visual question answering method as described in claim 1, characterized in that, The multimodal features include image features and text features; The step of inputting the multimodal features into the answer reasoning module to generate the answer and the deep reasoning features corresponding to the answer includes: The multimodal features are input into the answer reasoning module to enhance the internal representation of the image features and the text features through a self-attention mechanism, and a visual-language semantic alignment is established using a cross-attention mechanism to generate a deeply fused joint representation. Based on the joint representation, the classifier in the reasoning module outputs the probability distribution of the answer, and the final answer is determined according to the probability distribution of the answer. Extract the hidden state most relevant to the answer as the deep reasoning feature.

4. The multimodal visual question answering method as described in claim 1, characterized in that, The step of inputting the multimodal features into the basic principle generation module to generate guiding features includes: The multimodal features are input into the basic principle generation module to enhance the internal semantic association of the multimodal features using a self-attention mechanism, and reconstruct structured features that conform to the input specifications of the large language model. The structured features are projected into the embedding space of the large language model to generate the guiding features containing visual semantic information.

5. The multimodal visual question answering method as described in claim 1, characterized in that, The step of inputting the guiding features into the large language model to generate text is as follows: The guiding features are input as conditions into the large language model to be deeply fused with the text query vectors in the large language model to generate a contextual representation with enhanced visual semantics. Based on the context representation, each word is generated sequentially using an autoregressive approach. The correlation between the current sequence and the guiding features is calculated iteratively, and the basic principle of the text that conforms to visual semantic constraints is output.

6. The multimodal visual question answering method as described in claim 1, characterized in that, The step of generating visual fundamentals using the object detector based on the context features, the image, and the deep inference features includes: The context features and the deep inference features are concatenated and linearly transformed to generate the multimodal conditional vector; The image and the multimodal conditional vector are input into the object detector as prior knowledge to enhance the regional features in the image features corresponding to the image that are related to the text description and reasoning results; Based on the enhanced image features, predict the target bounding box to filter out the visual entities most relevant to the context features and the depth inference, and output the visual principle labeled with the target bounding box.

7. The multimodal visual question answering method as described in any one of claims 1-6, characterized in that, During model training, the model that minimizes the text alignment loss value calculated by the text alignment loss function is taken as the visual question answering model.

8. A multimodal visual question-answering device, applied to a visual question-answering system, characterized in that, The visual answering system's visual answering model includes a visual language encoding model, an answer reasoning module, a basic principle generation module, a large-scale language model, and an object detector. The device includes: An encoding unit is used to acquire the input image and question, and input the image and question into the visual language encoding model for encoding to generate multimodal features; The first input generation unit is used to input the multimodal features into the answer reasoning module to generate an answer and deep reasoning features corresponding to the answer; The second input generation unit is used to input the multimodal features into the basic principle generation module to generate guiding features, and input the guiding features into the large language model to generate text basic principles; An extraction and generation unit is used to extract features of the textual basic principle to obtain contextual features, and generate a visual basic principle through the object detector based on the contextual features, the image, and the deep reasoning features. As a unit, it is used to take the visual fundamental principle, the textual fundamental principle, and the answer as the final interpretable response.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal visual question answering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal visual question answering method as described in any one of claims 1 to 7.

Images