Multimodal question-answering explanation method and related device

By performing logical reasoning and iterative optimization on visual content and questions, answers and explanatory texts that are highly relevant to the visual content and questions are generated. This solves the problem of low relevance of explanatory texts in existing technologies and improves the accuracy and transparency of question-answering systems.

CN118689997BActive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
CN202411155333.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-22
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

In existing text-based visual question answering solutions, the answer generation process and the explanation text generation process are independent of each other, resulting in a low correlation between the explanation text and the input image and question, which affects the accuracy of the explanation text.

Method used

By acquiring the target visual content and question, logical reasoning is performed to generate an initial answer and explanation text. Through an iterative optimization process, using the answer explanation generation model and the explanation evaluation model, the answer and explanation text are gradually adjusted until their semantic relevance to the visual content and question reaches a threshold, thus generating the final target answer and explanation text.

Benefits of technology

It improves the accuracy and transparency of multimodal question answering explanations, generates explanatory texts that are closer to the real text content, enhances the credibility and explainability of the question answering system, and reduces security risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118689997B_ABST
    Figure CN118689997B_ABST
Patent Text Reader

Abstract

The present application relates to the field of computer technology, and provides a multimodal question-answering interpretation method and related devices, the method comprising: performing logical reasoning based on target visual content and target questions, generating an initial answer and an initial interpretation text describing the logical reasoning process, and performing iterative optimization starting from the initial answer and the initial interpretation text to obtain a target answer and a target interpretation text. During each iteration, for the answer and the interpretation text of this iteration, a corresponding evaluation value representing the degree of semantic relevance is predicted. If the evaluation value exceeds the set evaluation value threshold, the answer and the interpretation text of this iteration are used as the target answer and the target interpretation text. Otherwise, based on the evaluation value and the interpretation text of this iteration, logical reasoning is performed again to generate the answer and the interpretation text of the next iteration, and then enter the next iteration. Information mining is fully carried out through iterative updates, thereby effectively improving the accuracy of questions and answers.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Multimodal question answering is a task that combines multiple types of data (such as text, images, videos, etc.) to answer questions. Taking visual question answering as an example, the visual question answering task requires giving corresponding answers to the input images and questions. As a subsequent subtask of visual question answering, the visual question answering explanation task needs to provide an intuitive and easy-to-understand explanation of the decision-making process while giving accurate answers. The explanation can be presented in the form of text, images, or graph structures.

[0003] In related technologies, text-based visual question answering interpretable solutions are usually implemented in the following way: first, the corresponding answer is generated based on the input image and question, and then the explanation text of the answer is generated based on the generated answer, combined with the input image and question. For example, for a given image, the question is "Do you need to bring an umbrella today?", the generated answer is "No", and the explanation text is "Because the blue sky and white clouds in the image indicate that it is not raining."

[0004] However, since the answer generation process and the explanation text generation process are independent of each other, there may be problems such as low correlation between the explanation text and the input image and question, and the explanation text contains information that does not exist in the image and question, which affects the accuracy of the explanation text. Summary of the invention

[0005] The embodiments of the present application provide a multimodal question and answer interpretation method and related devices to improve the accuracy of multimodal question and answer interpretation.

[0006] In a first aspect, an embodiment of the present application provides a multimodal question-answering interpretation method, comprising:

[0007] Acquire target visual content and target question, and perform logical reasoning based on visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process;

[0008] Based on the visual features and the target question, iterative optimization is performed starting from the initial answer and the initial explanation text to obtain a target answer and a target explanation text, and each iteration process includes:

[0009] Based on the semantic features of the answer and the explanation text of this iteration, a corresponding evaluation value is predicted, and the evaluation value represents: the semantic relevance between the answer and the explanation text generated in the same iteration process and the target visual content and the target question;

[0010] If the evaluation value exceeds the set evaluation value threshold, the answer and explanation text of this iteration will be used as the target answer and target explanation text; otherwise, based on the evaluation value and the explanation text of this iteration, logical reasoning will be performed again on the visual features and the target question to generate the answer and explanation text of the next iteration, and then enter the next iteration.

[0011] In a second aspect, an embodiment of the present application provides a multimodal question-answering interpretation device, comprising:

[0012] An initial processing unit, used to obtain target visual content and target question, and perform logical reasoning based on visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process;

[0013] An iterative optimization unit is used to perform iterative optimization starting from the initial answer and the initial explanation text based on the visual features and the target question to obtain a target answer and a target explanation text, wherein each iteration process includes:

[0014] Based on the semantic features of the answer and the explanation text of this iteration, a corresponding evaluation value is predicted, and the evaluation value represents: the semantic relevance between the answer and the explanation text generated in the same iteration process and the target visual content and the target question;

[0015] If the evaluation value exceeds the set evaluation value threshold, the answer and explanation text of this iteration will be used as the target answer and target explanation text; otherwise, based on the evaluation value and the explanation text of this iteration, logical reasoning will be performed again on the visual features and the target question to generate the answer and explanation text of the next iteration, and then enter the next iteration.

[0016] In a possible implementation, when performing logical reasoning again on the visual features and the target question based on the evaluation value and the interpretation text of the current iteration to generate the answer and interpretation text of the next iteration, the iterative optimization unit is specifically used to:

[0017] Based on the evaluation value and the explanation text of this iteration, the answer explanation generation model is used to perform logical reasoning on the visual features and the target question again to generate the answer and explanation text of the next iteration; wherein the answer explanation generation model is generated by fine-tuning the large language model using the answer explanation sample set.

[0018] In a possible implementation, the iterative optimization unit is further used to:

[0019] Obtaining each original sample, each original sample includes: sample content, sample questions, real answers and real explanation text;

[0020] Performing logical reasoning on the sample content and sample questions in each of the original samples respectively to generate corresponding reasoning results, and based on the generated reasoning results, combined with the real answers and real explanation texts in each of the original samples, obtaining answer explanation samples associated with each of the original samples;

[0021] Based on each answer explanation sample obtained, the answer explanation sample set is obtained.

[0022] In a possible implementation, when obtaining the answer explanation samples associated with each of the original samples based on the generated inference results and combining the real answers and real explanation texts in the original samples, the iterative optimization unit is specifically used to:

[0023] For each of the original samples, the following operations are performed respectively:

[0024] Based on the real explanation text in an original sample, the inference result corresponding to the original sample is evaluated to obtain a corresponding sample evaluation value, each sample evaluation value represents: the semantic relevance between the corresponding inference result and the sample question and the sample content;

[0025] Based on the sample content, sample question, true answer and true explanation text in the original sample, as well as the reasoning result and sample evaluation value corresponding to the original sample, a corresponding answer explanation sample is obtained.

[0026] In a possible implementation, when evaluating the inference result corresponding to an original sample based on the real interpretation text in the original sample and obtaining the corresponding sample evaluation value, the iterative optimization unit is specifically used to:

[0027] Based on the sample semantic features of the real interpretation text in the one original sample and the sample semantic features of the predicted interpretation text in the inference result corresponding to the one original sample, obtaining the text similarity between the real interpretation text and the predicted interpretation text;

[0028] Based on the obtained text similarity, the corresponding sample evaluation value is obtained.

[0029] In a possible implementation, when obtaining the answer explanation sample set based on each obtained answer explanation sample, the iterative optimization unit is specifically used to:

[0030] Based on the sample evaluation values ​​corresponding to the answer explanation samples, at least one answer explanation sample having a sample evaluation value exceeding a set sample evaluation value threshold is screened out from the answer explanation samples;

[0031] The at least one screened answer explanation sample is used as the answer explanation sample in the answer explanation sample set.

[0032] In a possible implementation, the iterative optimization unit is further used to:

[0033] The answer explanation sample set is used to iteratively train the answer explanation generation model to be trained to obtain a trained answer explanation generation model, wherein each round of iteration includes:

[0034] Based on sample images, sample questions, predicted explanation texts, and sample evaluation values ​​in a batch of several answer explanation samples, the answer explanation generation model to be trained is used to predict new answers and new explanation texts for the several answer explanation samples;

[0035] Based on the new answers and new explanation texts of the several answer explanation samples, combined with the real answers and real explanation texts of the several answer explanation samples, the model loss is obtained, and the model parameters are adjusted based on the model loss.

[0036] In a possible implementation, when predicting a corresponding evaluation value based on the semantic features of the answer and the explanation text of this iteration, the iterative optimization unit is specifically used to:

[0037] Based on the semantic features of the answer and the explanation text of this iteration, the corresponding evaluation value is predicted using the explanation evaluation model, wherein the explanation evaluation model is obtained by iterative training using the explanation evaluation sample set, and each round of iteration includes:

[0038] Based on sample images, sample questions, true answers, and predicted explanation texts in a batch of explanation evaluation samples, an explanation evaluation model is used to obtain new evaluation values ​​for each of the plurality of explanation evaluation samples;

[0039] Based on the new evaluation values ​​of each of the several explanation evaluation samples and in combination with the sample evaluation values ​​corresponding to each of the several explanation evaluation samples, a model loss is obtained, and model parameters are adjusted based on the model loss.

[0040] In a possible implementation, the iterative optimization unit is further used to:

[0041] Obtaining each original sample, each original sample includes: sample content, sample questions, real answers and real explanation text;

[0042] For each of the original samples, the following operations are performed respectively:

[0043] Performing logical reasoning on an original sample to generate a corresponding reasoning result, and evaluating the reasoning result corresponding to the original sample based on the real interpretation text in the original sample to obtain a corresponding sample evaluation value;

[0044] Based on the sample content, sample question, true answer, inference result and sample evaluation value in the one original sample, obtain a corresponding answer explanation sample;

[0045] Based on the obtained interpretation evaluation samples, the interpretation evaluation sample set is obtained.

[0046] In a possible implementation, the initial processing unit is further configured to:

[0047] Based on the semantic features of the target visual content, describe the target visual content to obtain corresponding content description text;

[0048] Based on each target object included in the target visual content, obtaining an object category and an object position corresponding to each target object;

[0049] Performing text scanning on the target visual content to obtain text information in the target visual content;

[0050] Based on the content description text, the object category and the object position corresponding to each of the target objects, and in combination with the text information, the visual features of the target visual content are obtained.

[0051] In a possible implementation, when acquiring the target visual content and the target question, and performing logical reasoning based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process, the initial processing unit is specifically used to:

[0052] When receiving a target image and a target question input by a target object for a visual question answering task, taking the target image as the target visual content;

[0053] Logical reasoning is performed based on the visual features of the target image and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of any of the above methods.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of any of the above-mentioned methods.

[0057] In an embodiment of the present application, logical reasoning is performed based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanatory text describing the logical reasoning process. Then, based on the visual features and the target question, iterative optimization is performed starting from the initial answer and the initial explanatory text to obtain the target answer and the target explanatory text.

[0058] In this way, through iterative updates, implicit textual knowledge in the target visual content and target questions can be effectively mined to assist in question decision-making and effectively improve the accuracy of question answering. At the same time, through iterative optimization, natural language-based explanations are generated while giving answers, thus presenting the decision process and basis in an intuitive and easy-to-understand way, so that the subject can more clearly understand the reasoning process and decision reasons, thereby improving the transparency and explainability of multimedia question answering, which in turn helps to increase the credibility of the question answering system and reduce potential security risks.

[0059] During each iteration, the corresponding evaluation value is predicted based on the semantic features of the answer and explanation text of this iteration. The evaluation value represents the semantic relevance between the answer and explanation text generated in the same iteration and the target visual content and the target question. If the evaluation value exceeds the set evaluation value threshold, the answer and explanation text of this iteration are used as the target answer and target explanation text. Otherwise, based on the evaluation value and the explanation text of this iteration, logical reasoning is performed on the visual features and the target question again to generate the answer and explanation text of the next iteration, and then enter the next iteration.

[0060] Each iteration process includes two interactive steps: answer explanation generation and explanation evaluation. The interaction and optimization through progressive iteration method helps to understand the visual content and questions, improve the quality of explanation, and generate high-quality explanation text. The evaluation is based on the semantic features of the answer and explanation text of this iteration, and then the evaluation value is used to guide the logical reasoning of visual features and target questions again. The generated explanation text is closer to the real text explanation content in terms of semantic and syntactic similarity, which can improve the accuracy of the explanation.

[0061] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0063] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0064] Figure 2 A flowchart of a multimodal question-answering interpretation method provided in an embodiment of the present application;

[0065] Figure 3 A logical schematic diagram of a visual feature extraction process provided in an embodiment of the present application;

[0066] Figure 4 A logical diagram of a multimodal question-answering interpretation process provided in an embodiment of the present application;

[0067] Figure 5 A logical diagram of a multiple iteration process provided in an embodiment of the present application;

[0068] Figure 6 A logical schematic diagram of a two-round iterative optimization process provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of a sample generation method provided in an embodiment of the present application;

[0070] Figure 8 This is a logical diagram of the first sample construction process provided in the embodiments of the present application;

[0071] Fig. 9 A logical diagram of the second sample construction process provided in an embodiment of the present application;

[0072] Fig.10 A logical schematic diagram of a training process of an answer explanation generation model provided in an embodiment of the present application;

[0073] Fig.11 A logical diagram of a training process of an explanation evaluation model provided in an embodiment of the present application;

[0074] Fig.12 This is a schematic diagram of an application of a multimodal question-answering interpretation process provided in an embodiment of the present application;

[0075] Fig.13 A schematic diagram of the structure of a multimodal question-answering interpretation device provided in an embodiment of the present application;

[0076] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0078] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0079] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0080] It is understandable that in the specific implementation of the present application, data related to target visual content, target problems, etc., when the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0081] Below, some nouns involved in the embodiments of the present application are explained.

[0082] Multimodal Question Answering (MQA) is an interdisciplinary research direction in the fields of natural language processing and computer vision, which involves processing and understanding information from different modalities (such as text, images, videos, etc.) and answering questions based on this information.

[0083] Visual Question Answering (VQA) refers to answering questions based on the content of an image, such as describing the objects, scenes, or activities in the image.

[0084] Visual Language Natural Language Explanation (VL-NLE) refers to the use of natural language explanation to explain the decision-making process of the model in tasks that combine vision and language. Take visual question answering as an example. In visual question answering tasks, in addition to the answer, the model also needs to output an explanation to explain why this answer is chosen. The explanation can be based on observation of the image content, or it can be an understanding of the problem and logical reasoning.

[0085] Large Language Models (LLMs) refer to pre-trained models with a large number of parameters. The models are trained with a large amount of text data and can perform well on a variety of natural language processing tasks.

[0086] Transformer model: It is a network structure model suitable for processing sequence data. It uses a full attention structure instead of the traditional recurrent neural network (RNN) or convolutional neural network (CNN). The core idea is to model the relationship between each element in the input sequence and other elements as an attention weight matrix, and calculate the representation of each element through the self-attention mechanism. This structure enables Transformer to have a strong ability to capture long-distance dependencies in the sequence, thus achieving good performance in Natural Language Processing (NLP) tasks. Transformer models are widely used in the field of NLP, including machine translation, text summarization, question-answering systems and other tasks.

[0087] Token: In text processing, it refers to the words or subwords in the text, that is, the basic elements used by the model when processing text.

[0088] Optical Character Recognition (OCR): refers to the technology of detecting and recognizing text content from images.

[0089] The following is a brief introduction to the design concept of the embodiment of the present application:

[0090] In the multimodal question-answering explanation task, it is necessary to give accurate answers to the target questions of the target visual content while providing an intuitive and understandable explanation of the decision-making process. In related technologies, VL-NLE usually adopts a post-explanation scheme, which means that the question-answering and explanation are implemented in two independent steps. Since the answer generation process and the explanation text generation process are independent of each other, there may be problems such as low correlation between the explanation text and the input image and question, and the explanation text contains information that does not exist in the image and question, which affects the accuracy of the explanation text.

[0091] In an embodiment of the present application, logical reasoning is performed based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanatory text describing the logical reasoning process. Then, based on the visual features and the target question, iterative optimization is performed starting from the initial answer and the initial explanatory text to obtain the target answer and the target explanatory text.

[0092] In this way, through iterative updates, implicit textual knowledge in the target visual content and target questions can be effectively mined to assist in question decision-making and effectively improve the accuracy of question answering. At the same time, through iterative optimization, natural language-based explanations are generated while giving answers, thus presenting the decision process and basis in an intuitive and easy-to-understand way, so that the subject can more clearly understand the reasoning process and decision reasons, thereby improving the transparency and explainability of multimedia question answering, which in turn helps to increase the credibility of the question answering system and reduce potential security risks.

[0093] During each iteration, the corresponding evaluation value is predicted based on the semantic features of the answer and explanation text of this iteration. The evaluation value represents the semantic relevance between the answer and explanation text generated in the same iteration and the target visual content and the target question. If the evaluation value exceeds the set evaluation value threshold, the answer and explanation text of this iteration are used as the target answer and target explanation text. Otherwise, based on the evaluation value and the explanation text of this iteration, logical reasoning is performed on the visual features and the target question again to generate the answer and explanation text of the next iteration, and then enter the next iteration.

[0094] Each iteration process includes two interactive steps: answer explanation generation and explanation evaluation. The interaction and optimization through progressive iteration method helps to understand the visual content and questions, improve the quality of explanation, and generate high-quality explanation text. The evaluation is based on the semantic features of the answer and explanation text of this iteration, and then the evaluation value is used to guide the logical reasoning of visual features and target questions again. The generated explanation text is closer to the real text explanation content in terms of semantic and syntactic similarity, which can improve the accuracy of the explanation.

[0095] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0096] See also Figure 1 As shown, it is a schematic diagram of a possible application scenario in an embodiment of the present application. The schematic diagram of the application scenario includes a client device 110 and a processing device 120.

[0097] In a possible implementation, after the target object inputs the target visual content and the target question on the client device 110, the client device 110 sends the input target visual content and the target question to the processing device 120. After receiving the target visual content and the target question, the processing device 120 generates a corresponding target answer and a target explanation text, and returns the generated target answer and the target explanation text to the client device 110. The target visual content includes but is not limited to an image or a video.

[0098] In a possible implementation, a client for multimodal question-answer interpretation may be installed in the client device 110, and the client may be in the form of, but not limited to, a small program, a web page, an application, etc. The client device 110 includes, but is not limited to, a mobile phone, a tablet computer, a notebook, an e-book reader, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc.

[0099] In one possible implementation, the processing device 120 is a background server corresponding to the client. The processing device 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0100] The client device 110 and the processing device 120 may communicate with each other via a wired network or a wireless network.

[0101] In a possible implementation, the client device 110 and the processing device 120 may correspond to the same device.

[0102] In a possible implementation, a graphical user interface (GUI) is provided on the client device 110. The target object can upload images, input text, upload videos, view answers, and explain texts through the GUI, but is not limited thereto.

[0103] See also Figure 1 As shown, Figure 1 A schematic diagram of a possible GUI is provided in . The target object uploads an image and asks a question in the GUI. The uploaded image is an outdoor scene picture, and the question is "Do I need to bring an umbrella when I go out today?" The client uses the image uploaded by the target object as the target visual content and the question asked as the target question, and sends the target visual content and the target question to the processing device. The server performs multimodal question-answering interpretation based on the target visual content and the target question, and generates a target answer and a target interpretation text "No, because the blue sky and white clouds in the picture indicate that it is not raining". The server returns the target answer and the target interpretation text to the client, so that the target object can view the answer through the client's GUI.

[0104] The following is a schematic illustration of the multimodal question-answering explanation process in combination with possible application scenarios:

[0105] Application scenario 1: interpretability scenario of visual-language tasks. Focus natural language interpretation on visual question-answering tasks, which include but are not limited to multimodal tasks such as image description, emotion recognition, and video question-answering. Generate high-quality explanation information through multimodal question-answering interpretation.

[0106] Taking image description as an example, in this application scenario, the target object can ask questions to the question-answering system through a graphical user interface. The target object's target question is to generate descriptive text for a given image. The question-answering system performs logical reasoning on the target image and the target question, generates an initial answer and an initial explanation text describing the logical reasoning process, and iteratively optimizes the initial answer and the initial explanation text to obtain the target answer and the target explanation text. For example, the target object is given a target image, and the question-answering system generates a descriptive text "a red car is driving" as the answer to the target image, and gives the corresponding explanation text. The explanation text can point out that when the question-answering system generates "red", it focuses on the color of the car, when it generates "car", it focuses on the vehicle itself, and when it generates "driving", it focuses on the position and dynamics of the vehicle.

[0107] Take emotion recognition as an example. In this application scenario, the target object can ask questions to the question-answering system through a graphical user interface. The target visual content can be audio, video, or image information. The target question is to identify the emotional state. The question-answering system performs logical reasoning on the target visual content and target question given by the target object, generates an initial answer and an initial explanation text describing the logical reasoning process, and iteratively optimizes the initial answer and initial explanation text to obtain the target answer and target explanation text. For example, the target object gives a video with facial expression changes, and the question-answering system generates the answer "anger" and gives the corresponding explanation text. The explanation text can point out the frowning brows and closed mouth in the video, which indicate the angry emotional state.

[0108] Taking video question answering as an example, in this application scenario, the target object can ask questions to the question answering system through a graphical user interface. The target visual content is a video. The question answering system performs logical reasoning on the target video and the target question, generates an initial answer and an initial explanation text describing the logical reasoning process, and iteratively optimizes the initial answer and the initial explanation text to obtain the target answer and the target explanation text. For example, the target object is given a video, and the target question is "How many kinds of fruits are there in the video?" The question answering system generates the answer "3 kinds" and gives the corresponding explanation text. The explanation text can point out that the video contains three types of fruits: bananas, apples, and strawberries.

[0109] In the visual dialogue task, the target object can have one or more rounds of dialogue with the question-answering system through a graphical user interface. In order to improve interpretability, the question-answering system can show how to understand each question and how to generate the answer. Suppose, the first question for a given image is "What is in this room?", and the answer is: "This room has a table, a chair, and a bookshelf." The explanation text can be "There are tables, chairs, and bookshelves in the image, so my answer is 'This room has tables, chairs, and bookshelves'". The second question for a given image is "What is the function of this room?", and the answer is "study room", and the explanation text can be "There are tables, chairs, and bookshelves in this room, so my answer is 'This room is a study room'".

[0110] It should be noted that in the embodiments of the present application, only the question-answering system is used as an example for explanation. In actual application, the multimodal question-answering interpretation method mentioned in the embodiments of the present application can also be applied to dialogue robots, agents, etc., but is not limited to this.

[0111] Application scenario 2: Knowledge-enhanced visual question answering scenario. Knowledge-enhanced visual question answering is a multimodal task. The question answering system must not only understand the content of the image, but also understand the natural language questions related to it and give accurate answers. This type of task requires the model to have a deep understanding of images and texts, as well as a certain degree of reasoning ability. In traditional VQA tasks, answers are usually based only on the visual features of the target visual content and the text information in the target question. However, many questions require additional background knowledge to be answered correctly. For example, a picture may show a person standing on a bridge. If the question is "Where might this person be going?", it is not enough to rely solely on the information in the image itself. Knowledge such as where the bridge connects or common activities in the area is also required.

[0112] The multimodal question-answering interpretation method mentioned in the embodiment of the present application can be implemented using a large language model. Specifically, the large language model includes: an answer explanation generation model and an explanation evaluation model, wherein the answer explanation generation model is used to perform logical reasoning, generate answers and explanation texts, and the explanation evaluation model is used to predict corresponding evaluation values. Since the large language model is pre-trained and learned on a large-scale text data set, it has accumulated a large amount of world knowledge and logical reasoning information, so it can better understand the questions and give more accurate answers. Furthermore, combined with prompt engineering, the large language model can be guided to generate high-quality factual knowledge related to visual content and questions. The high-quality factual knowledge generated by the large language model can be used in knowledge-enhanced visual question-answering tasks to improve the accuracy of knowledge-enhanced visual question-answering tasks.

[0113] Knowledge-enhanced visual question answering includes but is not limited to: VQA based on image description and external knowledge, VQA based on knowledge graphs, VQA using knowledge graphs to enhance representation, etc.

[0114] For example, in a VQA scenario based on image description and external knowledge, the target visual content may be an image depicting a person surfing at the beach, and the target question may be "In which season might this person be surfing?" The large language model can use environmental clues in the image (such as sunny weather, wearing shorts, etc.) and knowledge about the surfing season to generate the answer "summer" and the corresponding explanatory text.

[0115] For example, in a VQA scenario based on knowledge graphs, the target visual content is to describe a picture in which two people are standing in a park, one of whom is giving directions to the other. The target question is "What might be the relationship between these two people?" The large language model generates the answer "It might be the relationship between friends or a tour guide and a tourist" and the corresponding explanatory text based on the image content and common sense knowledge.

[0116] For example, in a VQA scenario using knowledge graph enhanced representation, the target visual content is an image of a scenic spot, and the target question is "Which city's famous landmark is this?" The large language model identifies the tourist attractions through the architectural features in the image, and combines geographical knowledge to generate the answer "Paris" and the corresponding explanatory text.

[0117] Application scenario three: downstream application scenario. The multimodal question-answering interpretation process mentioned in the embodiments of this application can be used as a derivative technical solution of visual question-answering and multimodal agent, and applied to business scenarios such as short video social platforms and content open platforms. It can also be applied to business scenarios such as multimodal content understanding and content review, and assist users in content understanding and decision-making by interpreting text.

[0118] See also Figure 2 , which is a flow chart of a multimodal question-answering interpretation method provided in an embodiment of the present application. The flow is applied to a processing device, and the specific flow is as follows:

[0119] S201, obtaining target visual content and target question, and performing logical reasoning based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process.

[0120] The target visual content may be an image or a video. The target question is a question input by the target object regarding the target visual content. The input form of the target question is not limited. It may be input by the target object in the form of text, voice, etc., or it may be text information contained in the target visual content, or it may be pre-set by the question-answering system. If the target question is information input in other non-text forms such as voice, the input information may be converted into text form. For example, the target visual content is an image describing an outdoor scene, and the target question is: "Should I bring an umbrella today?"

[0121] Taking the visual question answering scenario as an example, when receiving the target image and target question input by the target object for the visual question answering task, the target image is used as the target visual content, and then logical reasoning is performed based on the visual features of the target image and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process. The following only uses visual question answering as an example for explanation, that is, the target visual content is an image.

[0122] In an embodiment of the present application, the visual feature is an image feature used for logical reasoning. The visual feature includes, but is not limited to, one or more of the following information: a content description text of the target visual content, an object category of each target object in the target visual content, an object position of each target object, and one or more of the text information in the target visual content. Among them, the content description text of the target visual content can be obtained by using the semantic features of the target visual content to describe the target visual content, the object category and object position of each target object in the target visual content can be obtained by a target detection algorithm, and the text information in the target visual content can be extracted using OCR technology. Below, the visual feature includes the content description text of the target visual content, the object category of each target object in the target visual content, the object position of each target object, and the text information in the target visual content as an example for explanation.

[0123] In a possible implementation, the target visual content may be obtained by, but not limited to, the following methods:

[0124] Firstly, based on the semantic features of the target visual content, the target visual content is described to obtain the corresponding content description text;

[0125] Secondly, based on each target object contained in the target visual content, the object category and object position corresponding to each target object are obtained;

[0126] Thirdly, the target visual content is subjected to text scanning to obtain text information in the target visual content;

[0127] Finally, based on the content description text, the object category and object position corresponding to each target object, and text information, the visual features of the target visual content are obtained.

[0128] In visual question answering, the content description process can also be called image description generation, which is used to generate descriptive text (i.e., content description text) for the target visual content. The semantic features of the target visual content can be the basic attributes of the image (such as color, texture, shape, etc.), or one or more of the following information: object information (objects in the image, such as people, animals, vehicles, etc.), scene information (the overall scene of the image, such as indoors, outdoors, city streets, beaches, etc.), action information (what actions the characters in the image are taking, such as running, jumping, talking, etc.), but are not limited to these.

[0129] In one possible implementation, during the image description generation process, a convolutional neural network (CNN) or other deep learning models can be used to extract semantic features from the target visual content, and then, using the semantic features, a model such as a recurrent neural network (RNN), a long short-term memory network (LSTM) or a Transformer can be used to generate content description text. Exemplarily, the image description model BLIP-2 is used for image description generation. BLIP-2 is further developed based on BLIP (Bridging Language and Vision Pre-training). BLIP-2 learns how to map semantic features and text features into the same vector space, thereby achieving an effective association between the target visual content and the content description text.

[0130] For example, see Figure 3 As shown, feature extraction is performed on the target visual content, and the extracted visual features include the content description text "The picture depicts a tranquil seaside scene with blue sky and white clouds, boats on the water, street lights and signs on the concrete sidewalk, a person walking, a bicycle parked, and a tram line parallel to the cobblestone road."

[0131] In the embodiment of the present application, the target object may be an object appearing in the target visual content, and the object includes but is not limited to one or more of a person, an animal, a vehicle, etc. The object position is used to characterize the position of the target object in the target visual content.

[0132] In one possible implementation, the process of obtaining the object category and the object position can also be called the target detection process. The target detection process can be implemented by using but not limited to the target detection model DINO (DETR with Improved and Normalized Objectives), so as to obtain the object category of one or more target objects appearing in the target visual content, and obtain the corresponding object position. DETR is an end-to-end object detection method that introduces Transformer to handle the target detection task. Unlike anchor-based methods, DETR uses a set of fixed query points (queries) to predict the bounding box and object category of the target object, where the bounding box can also be called a detection box to indicate the object position of the target object. This method simplifies the detection process and can directly predict the object type and object position of each target object from the image.

[0133] For example, see Figure 3As shown, based on the target objects included in the target visual content, the object categories corresponding to each target object can be obtained, wherein the object categories of the target objects in the target visual content include: people, bicycles, ships, oceans, street lights, tram tracks, and the sky.

[0134] In a possible implementation, when scanning text, OCR technology can be used to extract the text content in the target visual content. Exemplarily, the target visual content is first preprocessed, and the preprocessing includes but is not limited to one or more of grayscale, binarization, noise removal, etc.; then, image features are extracted from the preprocessed target visual content, and the image features include but are not limited to one or more of edge, texture, shape, etc.; then, character segmentation is performed using the image features to segment the text in the target visual content into several characters or words; then, OCR technology is used to identify each character or word to obtain the text content in the target visual content.

[0135] For example, using OCR technology, we can determine Figure 3 The target visual content shown does not contain text content. For another example, assuming that the target visual content contains the text "No Passing", then the text information in the target visual content is obtained by using OCR technology to scan the text of the target visual content, and the text information is "No Passing".

[0136] In the above implementation, the content description text helps to understand the image content, the object category helps to determine the objects contained in the image, the object position helps to detect moving objects in the video, and helps to determine the relative position relationship between the target objects, while the text content can provide additional natural language information. Utilizing one or more of the above information helps to fully explore the semantic information of the target visual content, thereby improving the accuracy of the answer and the explanatory text.

[0137] In a possible implementation, the content description text, the object category and object position corresponding to each target object, and the text information are combined according to a set information combination method to obtain the visual features of the target visual content.

[0138] Assume that image I is used to represent the target visual content, question Q is used to represent the target question, and T CAP Indicates content description text, using T TAG Indicates the object category, using T LOC Indicates the position of the object, using T OCR Represents text information. The target object detection box is represented by a rectangular box, T LOC=[x1, y1, x2, y2], [x1, y1] represents the relative position of the upper left corner of the target object's target detection box, and [x2, y2] represents the relative position of the lower right corner of the target object's target detection box. IMAGE Represents visual features, using the initial information combination method to T CAP , T TAG , T LOC and T OCR Combine one or more of the above to obtain the visual feature T in text form IMAGE .

[0139] Exemplarily, the information combination is set as follows: IMAGE =Description: T CAP ; Object: T TAG , Position: T LOC ; Text information: T OCR For an image, the content description may also be referred to as the image description. The text information may also be referred to as the OCR text.

[0140] For example, the content description text is "The picture depicts a tranquil seaside scene with blue sky and white clouds, boats on the water, street lights and signs on the concrete sidewalk, a person walking, a bicycle parked, and a tram line parallel to the cobblestone road", and the target objects in the target visual content include people, bicycles, boats, oceans, street lights, tram tracks, and the sky, and the text information is "none". According to the set information combination method, the content description text, the object category and object position corresponding to each target object, and the text information are combined to obtain the visual features of the target visual content. The visual features are "Content description: The picture depicts a tranquil seaside scene with blue sky and white clouds, boats on the water, street lights and signs on the concrete sidewalk, a person walking, a bicycle parked, and a tram line parallel to the cobblestone road; object categories: people, bicycles, boats, oceans, street lights, tram tracks, sky; object positions: none; text information: none".

[0141] In one possible implementation, logical reasoning can be performed based on the visual features of the target visual content and the target question, and the answer explanation generation model can be used to perform logical reasoning to generate an initial answer and an initial explanation text. The answer explanation generation model is generated by fine-tuning the large language model using the answer explanation sample set. Accordingly, based on the evaluation value and the explanation text of this iteration, the answer explanation generation model is used to perform logical reasoning on the visual features and the target question again to generate the answer and explanation text of the next iteration. The training process of the answer explanation generation model is described below.

[0142] See also Figure 5As shown, A is used to represent the answer, E is used to represent the explanation, and T is used to represent the visual feature. Each iteration process includes two parts: answer explanation generation and explanation evaluation. In the first iteration, first, based on the visual feature T and the target question Q, the answer explanation generation model is used to generate the answer A and explanation text E of the first iteration. Then, based on the visual feature T, the target question Q, the answer A and explanation text E of the first iteration, the explanation evaluation model is used to predict the evaluation value of the first iteration. Assume that the evaluation value of the first iteration does not meet the termination condition, and enter the second iteration. In the second iteration, first, based on the visual feature T, the target question Q, the evaluation value of the first iteration, and the explanation text E of the first iteration, the answer explanation generation model is used to generate the answer A and explanation text E of the second iteration. Then, based on the visual feature T, the target question Q, the answer A and explanation text E of the second iteration, the explanation evaluation model is used to predict the evaluation value of the second iteration. Similarly, multiple iterations of optimization are performed until the termination condition is met, and the target answer and target explanation text are output.

[0143] Use the large language model for logical reasoning. Since the large model has emergence ability and logical reasoning ability, the emergence ability can enable the large language model to learn some knowledge that was not directly taught during its training process. The logical reasoning ability model can enable the large language model to make inferences based on existing knowledge and solve problems that require logical thinking and reasoning. After fine-tuning the large language model using the answer explanation sample set, the explainability and logical consistency of the question-answering system in multilingual question-answering explanation tasks are enhanced.

[0144] In one possible implementation, prompt engineering can be used to guide the answer explanation generation model to generate specific types of answers and explanation texts. Through prompt engineering, appropriate prompts can be designed to maximize the performance of the model, so that the model can understand the requirements of the task and generate high-quality answers and explanation texts.

[0145] Specifically, in the process of generating the initial answer and the initial explanation text, the initial prompt information is obtained by using the initial prompt template in combination with the visual features of the target visual content and the target question, and then based on the initial prompt information, the answer explanation generation model is used to perform logical reasoning to generate the initial answer and the initial explanation text.

[0146] Among them, the initial prompt information is used to guide the answer explanation generation model to generate answers and explanations in the initial answer and initial explanation generation stage.

[0147] Assume that the initial answer and initial explanation text [A, E] = φ LLM (φ Embedding (T IMAGE , Q)), where φ Embedding represents the hint embedding layer, φLLM Represents a large language model.

[0148] For example, the initial prompt template is: “As a question answering and explanation generation expert, you are required to generate appropriate answers and corresponding explanations for a given image and question. Image description: T CAP ; Object: T TAG ;OCR text: T OCR ; Question: Q ; Answer: [Answer], because [Explanation]".

[0149] See also Figure 6 As shown, in the first iteration, the initial prompt information is "As an expert in question answering and explanation generation, you are asked to generate appropriate answers and corresponding explanations for given images and questions. Image description: The picture shows a tranquil seaside scene with blue sky and white clouds, boats on the water, street lights and signs on the concrete sidewalk, a person walking, a bicycle parked, and a tram line parallel to the cobblestone road; objects: people, bicycles, boats, ocean, street lights, tram tracks, sky; OCR text: none; question: Should I bring an umbrella today? Answer: [ ], because [ ]", using the initial prompt information, the answer explanation generation model is adopted to generate an explanatory answer "No, because the sunny coast scene and clear blue sky indicate that it is not raining", where "no" is the initial answer and "the sunny coast scene and clear blue sky indicate that it is not raining" is the initial explanation text.

[0150] S202: Based on the visual features and the target question, iterative optimization is performed starting from the initial answer and the initial explanation text to obtain the target answer and the target explanation text.

[0151] See also Figure 4 In the embodiment of the present application, after feature extraction is performed on the target visual content, question and answer interpretation generation (i.e., logical reasoning) is performed based on the visual features and the target question to generate the answer and explanation text of this iteration. Then, for the answer and explanation text of this iteration, the corresponding evaluation value is predicted. When the termination condition is not met, the evaluation value of this iteration is used to re-generate question and answer interpretation for the visual features and the target question to generate the answer and explanation text of the next iteration, and then enter the next iteration optimization. When the termination condition is met, the target answer and target explanation text are output. Among them, the termination condition includes but is not limited to the evaluation value exceeding the set evaluation value threshold and reaching the set number of iterations, but is not limited to this. Since each iteration process is similar, the following only takes two iteration processes as an example to illustrate the iterative optimization process.

[0152] For details, see Figure 2 , each iteration process performs the following steps:

[0153] S2021. Based on the semantic features of the answer and explanation text of this iteration, the corresponding evaluation value is predicted, and the evaluation value represents: the degree of semantic relevance between the answer and explanation text generated in the same iteration process and the target visual content and the target question.

[0154] In the embodiment of the present application, the evaluation value can be expressed by a score or a level, and there is no limitation on this. In this article, only the score of 1 to 10 is used as an example for explanation. The higher the evaluation value, the higher the semantic correlation between the answer and the explanation text generated in the same iteration process and the target visual content and the target question. The lower the evaluation value, the lower the semantic correlation between the answer and the explanation text generated in the same iteration process and the target visual content and the target question, that is, 1 is the worst semantic correlation and 10 is the best semantic correlation.

[0155] In a possible implementation, an explanation evaluation model may be used to predict the evaluation value. The explanation evaluation model is generated by fine-tuning a large language model using an explanation evaluation sample set.

[0156] In one possible implementation, in the process of generating the explanation evaluation, the evaluation prompt template is used, combined with the answer of this iteration and the semantic features of the explanation text, to obtain the evaluation prompt information, and then based on the evaluation prompt information, the explanation evaluation model is used to perform the explanation evaluation to generate the evaluation value of this iteration.

[0157] Among them, the evaluation prompt information is used to guide the explanation evaluation model to score the explanation text.

[0158] Exemplarily, the explanation evaluation model includes a prompt embedding layer and a large language model layer. The prompt embedding layer in the explanation evaluation model is used to combine visual features, target questions, answers for this iteration, and semantic features of the explanation text using an evaluation prompt template to obtain evaluation prompt information. The evaluation prompt information is then used to predict the evaluation value of this iteration through the large language model layer in the explanation evaluation model.

[0159] Assume that for the initial answer and initial explanation text [A, E], the evaluation value S = φ LLM (φ Embedding (T IMAGE ,Q,A,E)), where,φ Embedding represents the hint embedding layer, φ LLM Represents a large language model.

[0160] For example, the evaluation prompt template is: "As a text evaluation expert, you are asked to generate an interpretive evaluation score from 1 to 10 based on the given image, question, and answer, where 1 is the worst and 10 is the best. Content description: T CAP ; Object: TTAG ; Text information: T OCR ; Question: Q; Answer: [Answer], because [Explanation text]; Score is [Evaluation value]".

[0161] Still reading Figure 6 As shown, in the first iteration of the explanation evaluation process, the evaluation prompt template is used to combine the visual features, the target question, the answer of this iteration and the semantic features of the explanation text to obtain the evaluation prompt information. The evaluation prompt information is "As a text evaluation expert, you are asked to generate an interpretive evaluation score from 1 to 10 based on the given image, question and answer, where 1 is the worst and 10 is the best. Image description: The picture shows a tranquil seaside scene with blue sky and white clouds, boats on the water, street lights and signs on the concrete sidewalk, a person walking, a bicycle parked, and tram tracks parallel to the cobblestone road; objects: people, bicycles, boats, oceans, street lights, tram tracks, sky; OCR text: none; question: Should I bring an umbrella today? Answer: No, because the sunny coast scene and clear blue sky indicate that it will not rain." Then, using the evaluation prompt information, the large language model layer in the explanation evaluation model is used to predict the evaluation value of this iteration, and the evaluation value of this iteration is 8.

[0162] S2022: Determine whether the evaluation value exceeds the set evaluation value threshold. If so, execute S2023; otherwise, execute S2024.

[0163] For example, assuming that the set evaluation value threshold is 9 and the evaluation value of this iteration is 8, then based on the evaluation value of this iteration and the explanatory text of this iteration, logical reasoning is performed again on the visual features and the target question to generate the answer and explanatory text for the next iteration, and then enter the next iteration, i.e. execute S2024.

[0164] S2023. The answer and explanation text of this iteration are used as the target answer and target explanation text.

[0165] S2024. Based on the evaluation value and the explanation text of this iteration, logical reasoning is performed again on the visual features and the target question to generate the answer and explanation text of the next iteration, and then enter the next iteration.

[0166] In one possible implementation, based on the evaluation value and the explanation text of this iteration, an answer explanation generation model is used to perform logical reasoning on the visual features and the target question again to generate the answer and explanation text of the next iteration; wherein the answer explanation generation model is generated by fine-tuning the large language model using the answer explanation sample set.

[0167] In one possible implementation, a prompt template is used to generate prompt information based on the explanation text and evaluation value of this iteration, and then the prompt information is used to perform logical reasoning again using the answer explanation generation model to obtain the evaluation value of this iteration.

[0168] Among them, the generated prompt information is used to guide the answer explanation generation model to generate answers and explanations in the answer and explanation generation stage that is not the first iteration.

[0169] Exemplarily, through the prompt embedding layer in the answer explanation generation model, a generation prompt template is used to combine the visual features, the target question, the explanation text of this iteration and the evaluation value to obtain the generation prompt information, and the generation prompt information is used to predict the evaluation value of this iteration through the large language model layer in the answer explanation generation model.

[0170] For example, the generation prompt template is "As an expert in question answering and explanation generation, you are asked to generate more accurate answers and corresponding explanations for a given image, question, previously generated explanation, and corresponding evaluation scores from 1 to 10 (1 is the worst and 10 is the best). Content description: T CAP ; Object: T TAG ; Text information: T OCR ; Question: Q ; Answer: [Answer from the previous iteration], because [Explanation text from the previous iteration]; Score is [Evaluation value from the previous iteration]; Answer: [Answer from this iteration], because [Explanation text from this iteration]"

[0171] Still reading Figure 6 As shown, through the prompt embedding layer in the answer explanation generation model, the generated prompt template is used to combine the visual features, the target question, the explanation text of this iteration and the evaluation value to obtain the generated prompt information. The generated prompt information is "As an expert in question answering and explanation generation, you are asked to generate more accurate answers and corresponding explanations for given images, questions, previously generated explanations and corresponding evaluation scores of 1 to 10 (1 is the worst and 10 is the best). Image description: The picture shows a tranquil seaside scene with blue sky and white clouds, boats on the water, street lights and signs on the concrete sidewalk, a person walking, a bicycle parked, and tram lines parallel to the cobblestone road; objects: people, bicycles, boats, oceans, street lights, tram lines, sky; OCR text: None; Question: Should I bring an umbrella today? Answer: No, because the sunny coast scene and clear blue sky indicate that it is not raining; Score: 8; Answer: [ ], because [ ]", using the generated prompt information, the large language model layer in the answer explanation generation model generates an explanatory answer of "No, because the clear blue sky indicates that it is not raining", where "No" is the answer for the next iteration and "The clear blue sky indicates that it is not raining" is the explanation text for the next iteration.

[0172] Next, in the second iteration of explanation evaluation, the evaluation prompt template is used to combine the visual features, the target question, the answer of this iteration and the semantic features of the explanation text to obtain the evaluation prompt information. The evaluation prompt information is "As a text evaluation expert, you are asked to generate an evaluation score from 1 to 10 for the interpretability based on the given image, question and answer, where 1 is the worst and 10 is the best. Image description: The picture shows a tranquil seaside scene with blue sky and white clouds, boats on the water, street lights and signs on the concrete sidewalk, a person walking, a bicycle parked, and a tram running parallel to the cobblestone road; objects: people, bicycles, boats, ocean, street lights, tram lines, sky; OCR text: none; question: Should I bring an umbrella today? Answer: No, because the clear blue sky indicates that it is not raining", and then, using the evaluation prompt information, the large language model layer in the explanation evaluation model is used to predict the evaluation value of this iteration, which is 9. At this time, the evaluation value of this iteration reaches the set evaluation value threshold of 9, so the answer and explanation text of the second iteration are used as the target answer and target explanation text.

[0173] It should be noted that, in the embodiment of the present application, the termination condition may also be that the answer explanation generation model and the explanation evaluation model interact and iterate for a maximum of K times, where the number of iterations K is a hyperparameter.

[0174] The following is an explanation of the model training process in the embodiment of the present application. The model training process involves two parts: the construction of a training sample set and the iterative training of the model using the training sample set. Since the training samples used by the answer explanation generation model and the explanation evaluation model may be different, the training sample set of the answer explanation generation model is called the answer explanation sample set, and the training sample set used by the explanation evaluation model is called the explanation evaluation sample set.

[0175] 1. Construction of the Answer Explanation Sample Set

[0176] In some embodiments, in order to generate more high-quality explanation information and evaluation scores, the present invention first manually constructs a small number of visual explanation and scoring samples, and guides the large model to generate more answers and explanations to the examples based on few-sample context learning.

[0177] For details, see Figure 7 As shown, the answer explanation sample set is constructed using the following steps:

[0178] S701. Obtain each original sample, each original sample including: sample content, sample question, real answer and real explanation text.

[0179] In the embodiment of the present application, the original sample may be data in an existing data set, but in order to improve the performance of the model, the original data may also be collected scene data. In the visual question-answering scenario, the scene data may be generated by historical feedback information triggered by the generated answer and explanatory text after the subject asks a question to a specified image. The historical feedback information is used to characterize the subject's satisfaction with the generated answer and explanatory text. For example, a user asks a question about an image depicting food, "Is there any fruit in the picture?" The answer and explanatory text generated by the question-answering system are "Yes, bananas are long and curved fruits with thick skin and soft and sweet flesh inside. They are yellow when ripe, although the skin may be green at first." The user likes the answer and explanatory text. When the accumulated number of likes exceeds the set number threshold, the image, question, answer and explanatory text are regarded as an original sample.

[0180] For ease of description, the following only takes the original sample i as an example. The original sample i can be any original sample among the original samples. In visual question answering, the sample content can also be called a sample image. The original sample i contains: Sample image I i Sample question Q i 、True answer A i , true explanation text E i .

[0181] Take the original sample 1 as an example. The original sample 1 contains: a sample image depicting food, a sample question "Is there any fruit in the picture?", the real answer "Yes" and the real explanation text "Banana is a long and curved fruit with a thick skin and soft and sweet flesh inside. It is yellow when ripe, although the skin may be green at first." For another example, Figure 1 For the images and questions in the video, the real answer can be "no" and the real explanation text can be "blue sky and white clouds" or "clear weather".

[0182] S702: Perform logical reasoning on the sample images and sample questions in each original sample to generate corresponding reasoning results.

[0183] Each inference result may include a predicted answer and a predicted explanation text.

[0184] In a possible implementation, at least one inference may be performed on the original sample i to obtain at least one inference result corresponding to the original sample i. In each inference process, the sample image I i Sample question Q i , using the pre-trained visual question answering model, generate an inference result corresponding to the original sample i, the inference result includes: predicted answer A' and predicted explanation text E'.

[0185] That is to say, for the original sample i, assuming that L inferences are performed, then L inference results corresponding to the original sample i can be obtained.

[0186] Take the original sample 1 as an example, see Figure 8 As shown, I, Q, A, and E are used to represent the sample image, sample question, true answer, and true explanation text in the original sample 1, respectively. The circle represents the information in the original sample, and the pentagon represents the information obtained based on the original sample. For the sample image and sample question in the original sample 1, three logical reasonings are performed to obtain three reasoning results, which are: {predicted answer "yes" (A1') and predicted explanation text "banana is a yellow fruit" (E1')}, {predicted answer "no" (A2') and predicted explanation text "banana is a yellow fruit, long and curved in shape" (E2')}, {predicted answer "yes" (A3') and predicted explanation text "banana is a yellow fruit" (E3')}.

[0187] S703. Based on the generated inference results, combined with the real answers and real explanation texts in the original samples, the answer explanation samples associated with the original samples are obtained.

[0188] In a possible implementation, taking an original sample i among the original samples as an example, the following operations are performed for the original sample i:

[0189] S703-a, based on the real interpretation text E in the original sample i i , evaluate the inference result corresponding to the original sample i and obtain the corresponding sample evaluation value.

[0190] Among them, each sample evaluation value represents: the semantic relevance between the corresponding reasoning result and the sample question and the sample image.

[0191] As a possible implementation method, the real interpretation text E i The sample semantic features of the original sample i and the sample semantic features of the predicted explanation text E' in the inference result corresponding to the original sample i are used to obtain the real explanation text E i The text similarity between the predicted interpretation text E' is calculated, and then, based on the obtained text similarity, the corresponding sample evaluation value is obtained.

[0192] Among them, the text similarity can be the real explanation text E i The cosine similarity between the sample semantic features of the predicted explanation text E' and the sample semantic features of the predicted explanation text E' is calculated, but not limited to this. The similarity score between the explanation texts is obtained by converting the generated explanation text and the original explanation text into high-dimensional vector embeddings with semantic information, and calculating the cosine similarity between the two.

[0193] Based on the obtained text similarity, a corresponding sample evaluation value can be obtained through normalization processing. Normalization processing can be performed by linear normalization, standard deviation normalization, logarithmic normalization, etc., but is not limited thereto. For example, the value of text similarity is converted into a sample evaluation value between 1 and 10 through normalization processing, where 10 indicates that the quality of the explained text is very high, and 1 indicates that the quality of the explained text is poor.

[0194] In one possible case, after performing L logical inferences on the original sample i, L inference results corresponding to the original sample i are generated. Assume that the L inference results are respectively inference result 1 {predicted answer A L1 ', predict the interpretation text E L1 '}, Reasoning result 2 {predicted answer A L2 ', predict the interpretation text E L2 '}, Reasoning result 3 {predicted answer A L3 ', predict the interpretation text E L3 '}, ..., reasoning result L{predicted answer A LL ', predict the interpretation text E LL '}.

[0195] Then, for each of the L inference results, we can explain the text E based on the real i The sample semantic features of the predicted explanation text E' in the inference result are used to obtain the real explanation text E i The text similarity between the predicted interpretation text E' is calculated, and then, based on the obtained text similarity, the corresponding sample evaluation value is obtained.

[0196] For example, see Figure 8 As shown, based on the sample semantic features of the true interpretation text (E) in the original sample 1 and the sample semantic features of E1', the text similarity 1 between E and E1' is obtained, and the text similarity 1 is 90%. Then, based on the value of the text similarity 1 is converted into a sample evaluation value 1 (S1) between 1 and 10 through normalization processing, the value of the sample evaluation value 1 is 9; based on the sample semantic features of E and the sample semantic features of E2', the text similarity 2 between E and E2' is obtained, and the text similarity 2 is 80%. Then, based on the value of the text similarity 2 is converted into a sample evaluation value 2 (S2) between 1 and 10 through normalization processing, the value of the sample evaluation value 2 is 8; based on the sample semantic features of E and the sample semantic features of E3', the text similarity 3 (S3) between E and E3' is obtained, and the text similarity 3 is 70%. Then, based on the value of the text similarity 3 is converted into a sample evaluation value 3 between 1 and 10 through normalization processing, the value of the sample evaluation value 3 is 7.

[0197] In the above implementation, by converting the real explanation text and the predicted explanation text into high-dimensional vector embeddings with semantic information respectively, and then using the similarity of the high-dimensional vector embeddings in the vector space as an indicator to measure the sample evaluation value, the accuracy of the sample evaluation value can be guaranteed.

[0198] S703-b, based on sample image I in original sample i i Sample question Q i 、True answer A i and the real interpretation text E i , as well as the inference result and sample evaluation value corresponding to the original sample i, to obtain the corresponding answer explanation sample.

[0199] In a possible implementation, after performing L logical inferences on the original sample i, L inference results corresponding to the original sample i are generated. For each inference result corresponding to the original sample i, the sample image I i Sample question Q i 、True answer A i , true explanation text E i , predict the explanation text E', sample evaluation value S, and construct an answer explanation sample.

[0200] For example, answer explanation sample 1 includes: a sample image depicting delicious food, a sample question "Is there any fruit in the picture?", the true answer "Yes", the true explanation text "A banana is a long, curved fruit with a thick skin and soft, sweet flesh inside. It is yellow when ripe, although the skin may start out green", the predicted explanation text "Bananas are yellow fruits", and a sample evaluation value of "9". Answer explanation sample 2 includes: a sample image depicting delicious food, a sample question "Is there any fruit in the picture?", the true answer "Yes", the true explanation text "A banana is a long, curved fruit with a thick skin and soft, sweet flesh inside. It is yellow when ripe, although the skin may start out green", the predicted explanation text "Bananas are yellow fruits with a long, curved shape, but they may start out green", and a sample evaluation value of "8". Answer explanation sample 3 includes: a sample image depicting delicious food, a sample question “Is there any fruit in the picture?”, the true answer “Yes”, the true explanation text “A banana is a long, curved fruit with a thick outer skin and soft, sweet flesh inside. It is yellow when ripe, although the skin may be green at first”, the predicted explanation text “A banana is a yellow fruit that is long and curved in shape”, and the sample evaluation value “7”.

[0201] In one possible implementation, in order to improve the sample quality, L logical inferences are performed on the original sample i to generate L inference results corresponding to the original sample i. Then, the inference results that meet the set sample screening conditions are screened out from the L inference results, and the corresponding answer explanation samples are constructed based on the screened inference results.

[0202] The sample screening conditions include but are not limited to at least one of the following conditions: the predicted answer is correct; the number of characters in the predicted explanation text exceeds the set character number threshold. Whether the predicted answer is correct can be obtained by comparing it with the real answer, and the set character number threshold can be 10.

[0203] See also Fig. 9 As shown, for the three inference results generated based on the original sample 1, since the predicted answers in the inference results {predicted answer "no" (A2') and predicted explanation text "banana is a yellow fruit, long and curved in shape" (E2')} are wrong, therefore, for the inference results {predicted answer "yes" (A1') and predicted explanation text "banana is a yellow fruit" (E1')}, answer explanation sample 1 is constructed, and for the inference results {predicted answer "yes" (A3') and predicted explanation text "banana is a yellow fruit" (E3')}, answer explanation sample 3 is constructed. Among them, answer explanation sample 1 contains: sample images depicting food, sample questions "Is there any fruit in the picture?", true answer "yes", true explanation text "banana is a long and curved fruit with thick skin and soft and sweet flesh inside. It is yellow when ripe, although the skin may be green at first", predicted explanation text "banana is a yellow fruit", and sample evaluation value "9". Answer explanation sample 3 includes: a sample image depicting delicious food, a sample question “Is there any fruit in the picture?”, the true answer “Yes”, the true explanation text “A banana is a long, curved fruit with a thick outer skin and soft, sweet flesh inside. It is yellow when ripe, although the skin may be green at first”, the predicted explanation text “A banana is a yellow fruit that is long and curved in shape”, and the sample evaluation value “7”.

[0204] In some implementations, the sample image I may also be i Perform image processing such as rotation, scaling, and flipping, and construct answer explanation samples based on the processed images.

[0205] In some implementations, one or more items in the real explanation text or the predicted explanation text may be subjected to text processing such as synonym replacement, sentence reorganization, etc. that does not change the text semantics, and an answer explanation sample may be constructed based on the processed text.

[0206] S704: Obtain an answer explanation sample set based on the obtained answer explanation samples.

[0207] In a possible implementation, each obtained answer explanation sample can be directly used as an answer explanation sample in an answer explanation sample set.

[0208] In a possible implementation, the obtained answer explanation samples may be screened, and the screened answer explanation samples may be used as answer explanation samples in the answer explanation sample set. As an example, based on the sample evaluation values ​​corresponding to the answer explanation samples, at least one answer explanation sample may be screened out from the answer explanation samples, and the sample evaluation value exceeds a set sample evaluation value threshold; the screened at least one answer explanation sample may be used as the answer explanation sample in the answer explanation sample set.

[0209] For example, assuming that the sample evaluation value threshold is 8, among answer explanation sample 1, answer explanation sample 2 and answer explanation sample 3, the answer explanation samples whose sample evaluation values ​​exceed 8 are answer explanation sample 1 and answer explanation sample 2. Therefore, answer explanation sample 1 and answer explanation sample 2 are taken as the answer explanation samples in the answer explanation sample set.

[0210] Through the above implementation method, the explanation texts are further screened, and only the explanations with scores higher than 6 are retained as training samples, ensuring the quality of the explanation texts in the training set during instruction fine-tuning, thereby improving the model training effect.

[0211] As another example, it is also possible to screen out at least one answer explanation sample whose predicted answer is correct from each answer explanation sample based on the real answer corresponding to each answer explanation sample; and use the screened out at least one answer explanation sample as the answer explanation sample in the answer explanation sample set. The predicted answer being correct may mean that the text similarity with the corresponding real answer exceeds a corresponding threshold, but is not limited thereto.

[0212] For example, among answer explanation sample 1, answer explanation sample 2 and answer explanation sample 3, the sample evaluation value predicts that the explanation samples whose correct answers are answer explanation sample 1 and answer explanation sample 3. Therefore, answer explanation sample 1 and answer explanation sample 3 are taken as the answer explanation samples in the answer explanation sample set.

[0213] As an example, based on the sample evaluation values ​​and true answers corresponding to each answer explanation sample, at least one answer explanation sample with a sample evaluation value exceeding a set sample evaluation value threshold and predicted to be the correct answer can be screened out from each answer explanation sample; the at least one screened out answer explanation sample is used as the answer explanation sample in the answer explanation sample set.

[0214] For example, the sample evaluation value threshold is 8, and among answer explanation sample 1, answer explanation sample 2, and answer explanation sample 3, the answer explanation samples whose sample evaluation values ​​exceed the set sample evaluation value threshold are answer explanation sample 1 and answer explanation sample 2, and among answer explanation sample 1, answer explanation sample 2, and answer explanation sample 3, the sample evaluation values ​​are predicted to be the answer explanation samples with correct answers are answer explanation sample 1 and answer explanation sample 3. Therefore, answer explanation sample 1 is taken as the answer explanation sample in the answer explanation sample set.

[0215] In the above construction method, a semi-supervised learning method is adopted to use a small amount of labeled data and a large amount of unlabeled data to learn the structure and pattern of the data, thereby improving the large model's ability to understand and generate explanations for question-answering tasks.

[0216] 2. Explanation of the Construction of Evaluation Sample Set

[0217] In some possible implementations, first, each original sample is obtained, each original sample contains: sample content, sample question, true answer and true explanation text. Then, for each original sample, the following operations are performed: logical reasoning is performed on an original sample to generate a corresponding reasoning result, and based on the true explanation text in an original sample, the reasoning result corresponding to an original sample is evaluated to obtain a corresponding sample evaluation value; and based on the sample content, sample question, true answer, reasoning result and sample evaluation value in an original sample, a corresponding answer explanation sample is obtained. Finally, based on each obtained explanation evaluation sample, an explanation evaluation sample set is obtained.

[0218] The generated inference result includes the predicted explanation text. Since the process of obtaining the sample evaluation value of the explanation evaluation sample is the same as the process of obtaining the sample evaluation value of the answer explanation sample in the above text, it will not be repeated here.

[0219] A possible implementation is to perform L logical inferences on the original sample i, generate L inference results corresponding to the original sample i, and for each inference result corresponding to the original sample i, convert the sample image I i Sample question Q i 、True answer A i , predict the explanation text E', sample evaluation value S, and construct an explanation evaluation sample. In order to guide the large model to score more comprehensively, it is also possible not to perform sample screening, but to directly use the explanation evaluation sample as the explanation evaluation sample in the explanation evaluation sample set.

[0220] It should be noted that, in the embodiment of the present application, the explanation evaluation sample set and the answer explanation sample set can be constructed using the same original sample. In this way, for each original sample, the sample image I i Sample question Qi 、True answer A i , true explanation text E, predicted explanation text E', sample evaluation value S, as the answer explanation sample, the sample image I i Sample question Q i 、True answer A i , predicted explanation text E', sample evaluation value S as explanation evaluation sample. Of course, different original samples can also be used to construct explanation evaluation sample set and answer explanation sample set respectively, and there is no limitation on this.

[0221] 3. Training process of answer explanation generation model

[0222] In an embodiment of the present application, during the iterative training process, the answer explanation samples are divided into designated batches, and training is performed based on the answer explanation samples of each sub-batch. Since the steps performed for training each batch in each iteration are similar, the training for one batch is taken as an example for explanation here.

[0223] In a possible implementation, the answer explanation sample set is used to iteratively train the answer explanation generation model to be trained to obtain the trained answer explanation generation model, wherein each round of iteration includes:

[0224] Based on sample images, sample questions, predicted explanation texts, and sample evaluation values ​​in a batch of answer explanation samples, the answer explanation generation model to be trained is used to predict new answers and new explanation texts for the answer explanation samples;

[0225] Based on the new answers and new explanation texts of several answer explanation samples, combined with the real answers and real explanation texts of several answer explanation samples, the model loss is obtained, and the model parameters are adjusted based on the model loss.

[0226] The model loss can be calculated using loss functions such as but not limited to the cross entropy loss function, the quadratic loss function, and the absolute loss function.

[0227] Take answer explanation sample 1 as an example, see Fig.10As shown, it is a logical schematic diagram of the answer explanation generation model training process provided by the embodiment of the present application. The circular I, Q, A, and E are used to represent the sample image, sample question, true answer, and true explanation text in the answer explanation sample 1, respectively. E1' represents the predicted explanation text, S1 represents the sample evaluation value, and the elliptical A and E represent the new answer and the new explanation text, respectively. Based on the sample image, sample question, predicted explanation text, and sample evaluation value, the answer explanation generation model to be trained is used to predict the new answer and new explanation text of the answer explanation sample 1. Then, based on the new answer and the new explanation text, combined with the true answer and true explanation text of the answer explanation sample 1, the model sub-loss corresponding to the answer explanation sample 1 is obtained, and then after obtaining the model sub-loss corresponding to a batch of several answer explanation samples, the model loss is obtained, and then the model parameter adjustment is performed based on the model loss.

[0228] In one possible implementation, the answer loss value can be obtained based on the new answers and true answers of the several answer explanation samples, and the explanation loss value can be obtained based on the new explanation text and true explanation text of the several answer explanation samples, and then the model loss can be obtained based on the answer loss value and the explanation loss value. The model loss can be the sum of the answer loss value and the explanation loss value, or different weights can be set for the answer loss value and the explanation loss value, and the weighted sum of the answer loss value and the explanation loss value can be obtained.

[0229] In one possible implementation, in order to reduce the training cost, the LORA fine-tuning technology can be used to fine-tune the LLM instructions. LORA is a large model training method based on low-rank matrix decomposition. It freezes the pre-trained parameters of the Transformer layer and adds two learnable low-rank matrices so that their product is used as a bypass matrix of the original parameter matrix to achieve efficient parameter fine-tuning learning.

[0230] Specifically, two smaller matrices (A) and (B) are added to the answer explanation generation model, and the original weight matrix (W) of the answer explanation generation model is decomposed into W+ΔW, where ΔW is a low-rank matrix consisting of the product of A and B, that is, ΔW=BA. During the fine-tuning process, only the newly added matrices are trained instead of the parameters of the entire model. In this way, only a small number of newly added parameters need to be trained, which greatly reduces training time and resource consumption. And because LoRA retains most of the weights of the pre-trained model, it can effectively fine-tune the multimodal question and answer explanation generation task while maintaining the performance of the original model.

[0231] 4. Explain the training process of the evaluation model

[0232] During the iterative training process, the explanation evaluation samples are divided into specified batches, and training is performed based on the explanation evaluation samples of each batch. Since the steps performed for training each batch in each iteration are similar, the training for one batch is taken as an example here.

[0233] In one possible implementation, the explanation evaluation sample set is used for iterative training, and each round of iteration includes:

[0234] Based on the sample images, sample questions, true answers, and predicted explanation texts in a batch of explanation evaluation samples, an explanation evaluation model is used to obtain new evaluation values ​​for each of the explanation evaluation samples.

[0235] Based on the new evaluation values ​​of several explanation evaluation samples and the sample evaluation values ​​corresponding to several explanation evaluation samples, the model loss is obtained, and the model parameters are adjusted based on the model loss.

[0236] The model loss can be calculated using loss functions such as, but not limited to, a cross entropy loss function, a square loss function, and an absolute value loss function.

[0237] The model loss can be calculated using loss functions such as but not limited to the cross entropy loss function, the quadratic loss function, and the absolute loss function.

[0238] Take the explanation of evaluation sample 1 as an example, see Fig.11 As shown, it is a logical schematic diagram of the explanation evaluation model training process provided by the embodiment of the present application. The circular I, Q, and A are used to represent the sample image, sample question, and true answer in the answer explanation sample 1, respectively, E1' represents the predicted explanation text, S1 represents the sample evaluation value, and the elliptical S represents the new evaluation value. Based on the sample image, sample question, true answer, and predicted explanation text in the explanation evaluation sample 1, the explanation evaluation model is used to obtain the new evaluation value of the explanation evaluation sample 1. Then, based on the new evaluation value and sample evaluation value of the explanation evaluation sample 1, the model sub-loss is obtained, and then after obtaining the model sub-loss corresponding to a batch of several explanation evaluation samples, the model loss is obtained, and then the model parameter adjustment is performed based on the model loss.

[0239] In one possible implementation, in order to reduce the training cost, the LORA fine-tuning technology can be used to fine-tune the LLM instructions. Similarly, two smaller matrices (A) and (B) are added to the explanation evaluation model, and the original weight matrix (W) of the explanation evaluation model is decomposed into W+ΔW, where ΔW is composed of the product of A and B. During the fine-tuning process, only the newly added matrices are trained, not the parameters of the entire model.

[0240] It should be noted that in the embodiments of the present application, the explanation evaluation model and the answer explanation generation model can be different models or the same model. If the explanation evaluation model and the answer explanation generation model are integrated into the same model, then it can be regarded as the model learning multiple tasks, that is, the model performs multi-task learning, wherein the multiple tasks include the answer explanation generation task and the explanation evaluation task.

[0241] Below, the present application is described in conjunction with a specific implementation method.

[0242] See also Fig.12 , the client device is a computer, the computing device is a server, and the multimodal question-answering interpretation method mentioned in the embodiment of the present application is applied to the content review scenario. In the content review scenario, the question-answering system needs to give an answer as to whether the target visual content contains abnormal content, and give an explanation text of the reasoning process for generating the answer.

[0243] Assume that an application for content review is installed in the computer, and the target object uploads the image to be detected in the application through the graphical user interface. The image to be detected can be a picture uploaded on the social media platform. After the computer receives the image to be detected, it sends the image to be detected to the server. The server uses the image to be detected as the target visual content and the preset question "whether it contains abnormal content" as the target question. It uses the multimodal question-answering interpretation method mentioned above to generate the corresponding target answer and target interpretation text, and sends the generated target answer and target interpretation text. Assume that the abnormal content includes apples, pineapples, and bananas. The target answer and target interpretation text can be "contains abnormal content because there are apples in the image."

[0244] In the embodiment of this application, in order to verify the correctness and effectiveness of the framework, an open source large language model LLaMA is used as a model to complete the answer explanation generation task and the explanation evaluation task, and LLaMA

[0245] Perform instruction fine-tuning training. Use different data sets to test the fine-tuned LLaMA and the reference model. The reference model is trained using fully supervised learning.

[0246] Table 1

[0247] (Model performance)

[0248]

[0249] Table 1 shows the text generation effect and question-answering accuracy of the multimodal question-answering interpretation method mentioned in the embodiment of the present application, and shows the text generation effect and question-answering accuracy of the reference model. It can be seen from the results that, compared with the reference model, the interpretation text generated by the multimodal question-answering interpretation method mentioned in the embodiment of the present application is closer to the real text interpretation content in terms of semantic and syntactic similarity, and can effectively mine the implicit text knowledge inside the large model through iterative updates, and assist the model in making problem decisions, effectively improving the accuracy of questions and answers. In business scenarios such as multimodal content understanding and content review, the requirements are cumbersome and the problems are difficult, and there are potential security risks in the output application of black box LLM. The present invention guides LLM to understand the requirements of multimodal tasks through instruction fine-tuning, and generates natural language-based explanations while improving request replies through iterative optimization processes, presenting the decision-making process and basis of the system in an intuitive and easy-to-understand way, so that users can more clearly understand the system behavior and decision-making reasons, thereby improving the transparency and interpretability of the system, which helps to increase users' trust in the system and reduce potential security risks.

[0250] Based on the same inventive concept, the present application embodiment provides a multi-modal question-answering interpretation device. Fig.13 As shown, it is a schematic diagram of the structure of the multimodal question-answering interpretation device 1300, which may include:

[0251] The initial processing unit 1301 is used to obtain the target visual content and the target question, and perform logical reasoning based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process;

[0252] The iterative optimization unit 1302 is used to perform iterative optimization starting from the initial answer and the initial explanation text based on the visual features and the target question to obtain a target answer and a target explanation text. Each iteration process includes:

[0253] Based on the semantic features of the answer and the explanation text of this iteration, a corresponding evaluation value is predicted, and the evaluation value represents: the semantic relevance between the answer and the explanation text generated in the same iteration process and the target visual content and the target question;

[0254] If the evaluation value exceeds the set evaluation value threshold, the answer and explanation text of this iteration will be used as the target answer and target explanation text; otherwise, based on the evaluation value and the explanation text of this iteration, logical reasoning will be performed again on the visual features and the target question to generate the answer and explanation text of the next iteration, and then enter the next iteration.

[0255] In a possible implementation, when performing logical reasoning again on the visual features and the target question based on the evaluation value and the interpretation text of the current iteration to generate the answer and interpretation text of the next iteration, the iterative optimization unit 1302 is specifically used to:

[0256] Based on the evaluation value and the explanation text of this iteration, the answer explanation generation model is used to perform logical reasoning on the visual features and the target question again to generate the answer and explanation text of the next iteration; wherein the answer explanation generation model is generated by fine-tuning the large language model using the answer explanation sample set.

[0257] In a possible implementation, the iterative optimization unit 1302 is further configured to:

[0258] Obtaining each original sample, each original sample includes: sample content, sample questions, real answers and real explanation text;

[0259] Performing logical reasoning on the sample content and sample questions in each of the original samples respectively to generate corresponding reasoning results, and based on the generated reasoning results, combined with the real answers and real explanation texts in each of the original samples, obtaining answer explanation samples associated with each of the original samples;

[0260] Based on each answer explanation sample obtained, the answer explanation sample set is obtained.

[0261] In a possible implementation, when obtaining the answer explanation samples associated with each of the original samples based on the generated inference results and combining the real answers and real explanation texts in the original samples, the iterative optimization unit 1302 is specifically used to:

[0262] For each of the original samples, the following operations are performed respectively:

[0263] Based on the real explanation text in an original sample, the inference result corresponding to the original sample is evaluated to obtain a corresponding sample evaluation value, each sample evaluation value represents: the semantic relevance between the corresponding inference result and the sample question and the sample content;

[0264] Based on the sample content, sample question, true answer and true explanation text in the original sample, as well as the reasoning result and sample evaluation value corresponding to the original sample, a corresponding answer explanation sample is obtained.

[0265] In a possible implementation, when evaluating the inference result corresponding to an original sample based on the real interpretation text in the original sample and obtaining the corresponding sample evaluation value, the iterative optimization unit 1302 is specifically used to:

[0266] Based on the sample semantic features of the real interpretation text in the one original sample and the sample semantic features of the predicted interpretation text in the inference result corresponding to the one original sample, obtaining the text similarity between the real interpretation text and the predicted interpretation text;

[0267] Based on the obtained text similarity, the corresponding sample evaluation value is obtained.

[0268] In a possible implementation, when obtaining the answer explanation sample set based on each obtained answer explanation sample, the iterative optimization unit 1302 is specifically used to:

[0269] Based on the sample evaluation values ​​corresponding to the answer explanation samples, at least one answer explanation sample having a sample evaluation value exceeding a set sample evaluation value threshold is screened out from the answer explanation samples;

[0270] The at least one screened answer explanation sample is used as the answer explanation sample in the answer explanation sample set.

[0271] In a possible implementation, the iterative optimization unit 1302 is further configured to:

[0272] The answer explanation sample set is used to iteratively train the answer explanation generation model to be trained to obtain a trained answer explanation generation model, wherein each round of iteration includes:

[0273] Based on sample images, sample questions, predicted explanation texts, and sample evaluation values ​​in a batch of several answer explanation samples, the answer explanation generation model to be trained is used to predict new answers and new explanation texts for the several answer explanation samples;

[0274] Based on the new answers and new explanation texts of the several answer explanation samples, combined with the real answers and real explanation texts of the several answer explanation samples, the model loss is obtained, and the model parameters are adjusted based on the model loss.

[0275] In a possible implementation, when the corresponding evaluation value is predicted based on the semantic features of the answer and the explanation text of this iteration, the iterative optimization unit 1302 is specifically used to:

[0276] Based on the semantic features of the answer and the explanation text of this iteration, the corresponding evaluation value is predicted using the explanation evaluation model, wherein the explanation evaluation model is obtained by iterative training using the explanation evaluation sample set, and each round of iteration includes:

[0277] Based on sample images, sample questions, true answers, and predicted explanation texts in a batch of explanation evaluation samples, an explanation evaluation model is used to obtain new evaluation values ​​for each of the plurality of explanation evaluation samples;

[0278] Based on the new evaluation values ​​of each of the several explanation evaluation samples and in combination with the sample evaluation values ​​corresponding to each of the several explanation evaluation samples, a model loss is obtained, and model parameters are adjusted based on the model loss.

[0279] In a possible implementation, the iterative optimization unit 1302 is further configured to:

[0280] Obtaining each original sample, each original sample includes: sample content, sample questions, real answers and real explanation text;

[0281] For each of the original samples, the following operations are performed respectively:

[0282] Performing logical reasoning on an original sample to generate a corresponding reasoning result, and evaluating the reasoning result corresponding to the original sample based on the real interpretation text in the original sample to obtain a corresponding sample evaluation value;

[0283] Based on the sample content, sample question, true answer, inference result and sample evaluation value in the one original sample, obtain a corresponding answer explanation sample;

[0284] Based on the obtained interpretation evaluation samples, the interpretation evaluation sample set is obtained.

[0285] In a possible implementation, the initial processing unit 1301 is further configured to:

[0286] Based on the semantic features of the target visual content, describe the target visual content to obtain corresponding content description text;

[0287] Based on each target object included in the target visual content, obtaining an object category and an object position corresponding to each target object;

[0288] Performing text scanning on the target visual content to obtain text information in the target visual content;

[0289] Based on the content description text, the object category and the object position corresponding to each of the target objects, and in combination with the text information, the visual features of the target visual content are obtained.

[0290] In a possible implementation, when acquiring the target visual content and the target question, and performing logical reasoning based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process, the initial processing unit 1301 is specifically used to:

[0291] When receiving a target image and a target question input by a target object for a visual question answering task, taking the target image as the target visual content;

[0292] Logical reasoning is performed based on the visual features of the target image and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process.

[0293] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0294] Regarding the device in the above embodiment, the specific manner in which each unit executes the request has been described in detail in the embodiment of the method, and will not be elaborated here.

[0295] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0296] Based on the same inventive concept, the embodiment of the present application also provides an electronic device. In one embodiment, the electronic device can be a server or a terminal device. Fig.14 As shown, it is a schematic diagram of the structure of a possible electronic device provided in an embodiment of the present application, Fig.14 In the embodiment, the electronic device 1400 includes: a processor 1410 and a memory 1420.

[0297] Among them, the memory 1420 stores a computer program that can be executed by the processor 1410, and the processor 1410 can perform the steps of the above-mentioned multimodal question and answer interpretation method by executing the instructions stored in the memory 1420.

[0298] The memory 1420 may be a volatile memory, such as a random-access memory (RAM); the memory 1420 may also be a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1420 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1420 may also be a combination of the above memories.

[0299] The processor 1410 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1410 is configured to implement the multimodal question-answering interpretation method when executing the computer program stored in the memory 1420 .

[0300] In some embodiments, the processor 1410 and the memory 1420 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0301] The specific connection medium between the processor 1410 and the memory 1420 is not limited in the embodiment of the present application. In the embodiment of the present application, the processor 1410 and the memory 1420 are connected via a bus as an example. Fig.14 The connections between the other components are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.14 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0302] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the multimodal question-answering interpretation method. In some possible implementations, various aspects of the multimodal question-answering interpretation method provided by the present application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to perform the steps in the multimodal question-answering interpretation method. For example, the electronic device can perform the following steps: Figure 2 Follow the steps shown in .

[0303] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (Compact Disk Read Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0304] The program product of the embodiment of the present application may adopt a CD-ROM and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a computer program, which may be used by or in combination with a command execution system, apparatus or device.

[0305] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a computer program for use by or in conjunction with a command execution system, apparatus, or device.

[0306] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0307] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multimodal question-answering explanation method, characterized in that: include: Acquire target visual content and target question, and perform logical reasoning based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process; wherein the visual features of the target visual content at least include a content description text of the target visual content, and the content description text is obtained through the semantic features of the target visual content; Based on the visual features and the target question, iterative optimization is performed starting from the initial answer and the initial explanation text to obtain a target answer and a target explanation text, and each iteration process includes: Based on the semantic features of the answer and the explanation text of this iteration, a corresponding evaluation value is predicted, and the evaluation value represents: the semantic relevance between the answer and the explanation text generated in the same iteration process and the target visual content and the target question; If the evaluation value exceeds the set evaluation value threshold, the answer and explanation text of this iteration will be used as the target answer and target explanation text; otherwise, based on the evaluation value and the explanation text of this iteration, logical reasoning will be performed again on the visual features and the target question to generate the answer and explanation text of the next iteration, and then enter the next iteration.

2. The method according to claim 1, characterized in that The step of performing logical reasoning again on the visual features and the target question based on the evaluation value and the explanation text of the current iteration to generate an answer and explanation text for the next iteration includes: Based on the evaluation value and the explanation text of this iteration, the answer explanation generation model is used to perform logical reasoning on the visual features and the target question again to generate the answer and explanation text of the next iteration; wherein the answer explanation generation model is generated by fine-tuning the large language model using the answer explanation sample set.

3. The method according to claim 2, characterized in that The answer explanation sample set is constructed in the following way: Obtaining each original sample, each original sample includes: sample content, sample questions, real answers and real explanation text; Performing logical reasoning on the sample content and sample questions in each of the original samples respectively to generate corresponding reasoning results, and based on the generated reasoning results, combined with the real answers and real explanation texts in each of the original samples, obtaining answer explanation samples associated with each of the original samples; Based on each answer explanation sample obtained, the answer explanation sample set is obtained.

4. The method according to claim 3, characterized in that The step of obtaining answer explanation samples associated with each of the original samples based on the generated inference results and combining the real answers and real explanation texts in the original samples comprises: For each of the original samples, the following operations are performed respectively: Based on the real explanation text in an original sample, the inference result corresponding to the original sample is evaluated to obtain a corresponding sample evaluation value, each sample evaluation value represents: the semantic relevance between the corresponding inference result and the sample question and the sample content; Based on the sample content, sample question, true answer and true explanation text in the original sample, as well as the reasoning result and sample evaluation value corresponding to the original sample, a corresponding answer explanation sample is obtained.

5. The method according to claim 4, characterized in that The step of evaluating the inference result corresponding to an original sample based on the real interpretation text in the original sample to obtain a corresponding sample evaluation value includes: Based on the sample semantic features of the real interpretation text in the one original sample and the sample semantic features of the predicted interpretation text in the inference result corresponding to the one original sample, obtaining the text similarity between the real interpretation text and the predicted interpretation text; Based on the obtained text similarity, the corresponding sample evaluation value is obtained.

6. The method according to claim 3, characterized in that The step of obtaining the answer explanation sample set based on each obtained answer explanation sample comprises: Based on the sample evaluation values ​​corresponding to the answer explanation samples, at least one answer explanation sample having a sample evaluation value exceeding a set sample evaluation value threshold is screened out from the answer explanation samples; The at least one screened answer explanation sample is used as the answer explanation sample in the answer explanation sample set.

7. The method according to any one of claims 2 to 6, characterized in that The answer explanation generation model is fine-tuned in the following way: The answer explanation sample set is used to iteratively train the answer explanation generation model to be trained to obtain a trained answer explanation generation model, wherein each round of iteration includes: Based on sample images, sample questions, predicted explanation texts, and sample evaluation values ​​in a batch of several answer explanation samples, the answer explanation generation model to be trained is used to predict new answers and new explanation texts for the several answer explanation samples; Based on the new answers and new explanation texts of the several answer explanation samples, combined with the real answers and real explanation texts of the several answer explanation samples, the model loss is obtained, and the model parameters are adjusted based on the model loss.

8. The method according to any one of claims 1 to 6, characterized in that The corresponding evaluation value is predicted based on the semantic features of the answer and the explanation text of this iteration, including: Based on the semantic features of the answer and the explanation text of this iteration, the corresponding evaluation value is predicted using the explanation evaluation model, wherein the explanation evaluation model is obtained by iterative training using the explanation evaluation sample set, and each round of iteration includes: Based on the sample images, sample questions, true answers, and predicted explanation texts in a batch of explanation evaluation samples, an explanation evaluation model is used to obtain new evaluation values ​​for each of the explanation evaluation samples. Based on the new evaluation values ​​of each of the several explanation evaluation samples and in combination with the sample evaluation values ​​corresponding to each of the several explanation evaluation samples, a model loss is obtained, and model parameters are adjusted based on the model loss.

9. The method according to claim 8, characterized in that The explanation evaluation sample set is constructed in the following way: Obtaining each original sample, each original sample includes: sample content, sample questions, real answers and real explanation text; For each of the original samples, the following operations are performed respectively: Performing logical reasoning on an original sample to generate a corresponding reasoning result, and evaluating the reasoning result corresponding to the original sample based on the real interpretation text in the original sample to obtain a corresponding sample evaluation value; Based on the sample content, sample question, true answer, inference result and sample evaluation value in the one original sample, obtain a corresponding answer explanation sample; Based on the obtained interpretation evaluation samples, the interpretation evaluation sample set is obtained.

10. The method according to any one of claims 1 to 6, characterized in that The visual features of the target visual content are obtained by: Based on the semantic features of the target visual content, describe the target visual content to obtain corresponding content description text; Based on each target object included in the target visual content, obtaining an object category and an object position corresponding to each target object; Performing text scanning on the target visual content to obtain text information in the target visual content; Based on the content description text, the object category and the object position corresponding to each of the target objects, and in combination with the text information, the visual features of the target visual content are obtained.

11. The method according to any one of claims 1 to 6, characterized in that The step of acquiring target visual content and target question, and performing logical reasoning based on visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process includes: When receiving a target image and a target question input by a target object for a visual question answering task, taking the target image as the target visual content; Logical reasoning is performed based on the visual features of the target image and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process.

12. A multimodal question-answering interpretation device, characterized in that: include: An initial processing unit, used to obtain target visual content and target question, and perform logical reasoning based on the visual features of the target visual content and the target question to generate an initial answer and an initial explanation text describing the logical reasoning process; wherein the visual features of the target visual content at least include a content description text of the target visual content, and the content description text is obtained through the semantic features of the target visual content; An iterative optimization unit is used to perform iterative optimization starting from the initial answer and the initial explanation text based on the visual features and the target question to obtain a target answer and a target explanation text, wherein each iteration process includes: Based on the semantic features of the answer and the explanation text of this iteration, a corresponding evaluation value is predicted, and the evaluation value represents: the semantic relevance between the answer and the explanation text generated in the same iteration process and the target visual content and the target question; If the evaluation value exceeds the set evaluation value threshold, the answer and explanation text of this iteration will be used as the target answer and target explanation text; otherwise, based on the evaluation value and the explanation text of this iteration, logical reasoning will be performed again on the visual features and the target question to generate the answer and explanation text of the next iteration, and then enter the next iteration.

13. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: It includes a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 11.

15. A computer program product, characterized in that It includes a computer program, which is stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of any method described in claims 1 to 11.

Citation Information

Patent Citations

  • Image visual question and answer model, method and system based on attention decision

    CN110134774A

  • Inference question and answer method based on knowledge fusion

    CN115730058A

  • Answer reasoning method and device, equipment and storage medium

    CN117216197A

  • Text marking method and device based on active learning, equipment and storage medium

    CN117828088A

  • Visual question and answer method and device, visual question and answer model training method and device, electronic equipment and storage medium

    CN117892140A

Cited By

  • A domain knowledge fusion and semantic question answering system based on text and multi-modal large model

    CN122946503A