Interpretable multimodal multi-hop reasoning question answering method with joint implied reasoning structure generation

Abstract

The present invention discloses an explainable multimodal multi-hop reasoning question-answering method for joint implication reasoning structure generation, comprising: decomposing a multi-step question and constructing a fact base based on the decomposition result; obtaining a premise corpus based on the fact base, generating an implication tree structure according to predefined symbols, the premise corpus and the original question; segmenting the implication tree structure to obtain multiple subtree sets, refining the implication tree structure according to the lengths of the multiple subtree sets to obtain a complete implication tree; constructing an expert hybrid model, obtaining overall features based on the complete implication tree and the expert hybrid model, optimizing the complete implication tree based on the overall features and an iterative feedback mechanism to obtain an updated fact set and answer.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and multimodal technology, and in particular to an explainable multimodal multi-hop reasoning question-answering method for joint implicit reasoning structure generation. Background Art

[0002] Multimodal multi-hop question answering (MMQA) is a complex task involving multiple input sources such as text, tables, and images. It requires reasoning across different modalities to generate accurate and complete answers. Currently, most multimodal multi-hop question answering methods adopt the method of converting multimodal information into text descriptions, that is, by converting images through image caption models, converting tables through natural language descriptions, and then using large-scale language models (LLMs) to generate answers. The significant advantage of this method is that it can utilize the powerful language understanding and generation capabilities of LLMs, as well as the interpretability of text descriptions. However, this method indiscriminately converts all multimodal information into text descriptions, which will inevitably generate a large amount of redundant information.

[0003] Multimodal multi-hop question answering is a task based on multimodal question answering, but it requires multi-hop reasoning to generate the final answer. VQA first proposed a method to answer questions from pure visual input. Later, WebQA and MultimodalQA required the integration of information such as free text, images, or semi-structured tables to answer multi-hop reasoning questions. In order to solve the challenge of finding answers from multiple information sources, MuRAG designed a multimodal Transformer architecture that accepts text and image feature inputs and built a million-scale dataset to pre-train the model. Using the image captioning model and table linearization method to uniformly convert multimodal information into text, a new multimodal question answering paradigm was proposed, but there was no restriction in the conversion process, resulting in a large amount of information redundancy, which affected the performance of the model. In addition, there are many studies on multimodal question answering using large models. The image captioning model is trained to generate image captions for gpt-3 to understand images and generate answers; the multimodal large model LLaVA is used to generate more accurate image captions, and then different context learning templates are constructed according to each modality, so that GPT-3 can give full play to its powerful performance in this task. Both methods require generating image captions for a large language model to understand the question, but there are no conditional restrictions in the image caption generation stage; or when generating image captions directly based on multi-hop questions, the questions contain information that a single image cannot answer, resulting in errors in the image caption generation stage. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an explainable multi-modal multi-hop reasoning question-answering method for joint implicit reasoning structure generation to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides an explainable multi-modal multi-hop reasoning question answering method for joint implicit reasoning structure generation, comprising:

[0006] Decompose multi-step problems and build a fact base based on the decomposition results;

[0007] Obtaining a premise corpus based on the fact base, and generating an implication tree structure according to predefined symbols, the premise corpus and the original question;

[0008] The implication tree structure is split to obtain a set of multiple subtrees, and the implication tree structure is refined according to the length of the set of multiple subtrees to obtain a complete implication tree;

[0009] An expert mixture model is constructed, and overall features are obtained based on the complete implication tree and the expert mixture model. The complete implication tree is optimized based on the overall features and an iterative feedback mechanism to obtain an updated fact set and answer.

[0010] Optionally, the process of building a fact base includes:

[0011] A multimodal evidence set is obtained based on multi-step question retrieval, the multi-step question is decomposed based on the multimodal evidence set to generate multiple sub-questions and corresponding evidence, the sub-questions and corresponding evidence are converted and stored based on the pattern of the corresponding evidence to obtain a fact base.

[0012] Optionally, the process of generating an implication tree structure includes:

[0013] A premise corpus is obtained based on the fact base, the premise corpus and a plurality of sub-questions are symbolically represented according to predefined symbols, and the symbolic representation is input into a natural language processing model to generate an implication tree structure.

[0014] Optionally, the process of refining the implication tree structure according to the length of the set of the multiple subtrees to obtain a complete implication tree includes:

[0015] If the length of the set is greater than 1, the subtree in the set is split into node sets, and the nodes in the node set are traversed. If the node has corresponding information in the fact base, it is replaced with the corresponding information. If there is no corresponding information, the content of the intermediate node is generated based on other nodes in the node set until the length of the set is equal to 1, and a complete implication tree is obtained.

[0016] Optionally, the process of obtaining multiple subtree sets includes:

[0017] Starting from the root node of the constructed implication tree structure, obtain the non-leaf nodes N rand its child nodes, and add them to the collection to obtain a subtree collection, until all nodes are traversed to obtain multiple subtree collections.

[0018] Optionally, the process of obtaining a fact index and a question answer based on the complete implication tree and the expert mixture model includes:

[0019] Extracting features of each fact based on the complete implication tree to obtain a fact feature set; converting the implication tree into a natural language description; obtaining a final feature based on the fact feature set and the natural language description; and inputting the final feature into the expert mixture model to obtain an overall feature.

[0020] Optionally, the expert mixture model includes two gating networks, two task-specific expert networks and one shared expert network;

[0021] The final features are input into the expert mixture model, and fact retrieval generation task experts and question answering task experts are selected from a given number of trainable experts through a gating network. The final features enter the task-specific expert network and shared expert network corresponding to the fact retrieval generation task experts and question answering task experts respectively, and the output is added to the final features to obtain the overall features.

[0022] Optionally, a cross-attention operation is performed on the fact retrieval generation decoder, the fact feature set and the final feature, and a cross-attention operation is performed on the question answer decoder and the natural language description until the cross entropy loss meets the requirements, thereby obtaining the fact index and the final answer.

[0023] Optionally, the process of optimizing the complete implication tree includes:

[0024] The fact index is replaced with the corresponding fact in the fact library, the replaced fact set is connected with the final answer, and input into the natural language processing model to optimize the complete implication tree.

[0025] Compared with the prior art, the present invention has the following advantages and technical effects:

[0026] The present invention decomposes a multi-step problem and constructs a fact base based on the decomposition result; obtains a premise corpus based on the fact base, and generates an implication tree structure according to predefined symbols, the premise corpus and the original problem; divides the implication tree structure to obtain multiple subtree sets, refines the implication tree structure according to the lengths of the multiple subtree sets, and obtains a complete implication tree; constructs an expert hybrid model, obtains overall features based on the complete implication tree and the expert hybrid model, optimizes the complete implication tree based on the overall features and an iterative feedback mechanism, and obtains an updated fact set and answer.

[0027] The present invention adopts joint learning of implication reasoning structure generation and answer generation tasks; adopts iterative feedback mechanism to guide the large model to generate more reasonable implication trees, and vice versa, more reasonable implication trees provide more reasonable facts, helping the joint task of fact evidence extraction and answer generation. Using the problem decomposition strategy and with the help of the large model, multimodal information is converted into text modal information, and an explainable reasoning process from evidence to answer is constructed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Embodiment 1

[0033] like Figure 1 As shown, this embodiment provides an explainable multi-modal multi-hop reasoning question answering method for joint implicit reasoning structure generation, including:

[0034] (I) Implication Tree Initialization Phase

[0035] (1) Construction of fact database

[0036] In the fact base construction module, it is necessary to decompose the multi-step problem into multiple sub-problems according to different evidences, and process the multiple sub-problems obtained by decomposition in different ways according to the patterns of the corresponding evidences.

[0037] Decomposing multi-hop questions: It is necessary to retrieve multi-modal evidence required to answer multi-step questions. However, since this embodiment generates image descriptions by decomposing questions, it uses global image descriptions and image attribute features (referring to using Blip to extract the Caption of the entire image and using fast r-cnn to extract the text corresponding to the entity in the image), sorts the retrieved library, and selects evidence with a score of more than 0.5, and finally retrieves the required multi-modal evidence set E:

[0038] E=BERT retri (Text evidence )=[E1,E2.....,E n ], (1)

[0039] Where "n" represents the number of pieces of evidence in the evidence set. evidence ' refers to all the evidence obtained after converting all "images, texts, tables" into text, which is called "Text evidence ”. BERT retri () refers to searching the pre-extracted evidence text features through Bert.

[0040] After obtaining all the retrieved multimodal evidence E = [E1, E2....., E n ], then prompt GPT-3.5 to decompose the original question based on the multimodal evidence E and generate n sub-questions q s , each sub-problem has its corresponding evidence. Assume q s The kth question in has L tags. It is represented as The decoding process can be described as:

[0041]

[0042] where p q is the instruction hint. The instruction hint p of the Large Language Model (LLM) q An overview is as follows:

[0043]

[0044] After obtaining all decomposed sub-problems based on each evidence, we process them in different ways according to the patterns of the corresponding evidence and convert them into facts stored in the fact base.

[0045] Image facts: For the image mode, GPT-3.5 is further used to decompose each sub-problem into atomic questions q i Assume that The rth atomic problem q obtained by decomposition i YesL i Mark, represented by The decoding process can be described as:

[0046]

[0047] Then, the decomposed atomic problem q i and the corresponding image E im is input into the VQA model to get the answer. The VQA model adopts LLaVA-1.5, and the model uses Clip as a visual feature extractor:

[0048] Feature img = CLIP(E k ) (4)

[0049] Assume that the answer to the rth atomic question is a r YesL a Marked, marked The decoding process can be described as:

[0050]

[0051] GPT-3.5 is then used to refine the obtained question-answer pairs consisting of atomic questions and corresponding answers, and refine them into facts stored in the fact base.

[0052] Tabular facts: Subproblems q that break down the original problem and obtain tabular evidence t After b, a simple natural language template is used to convert the table into sentences that sound natural to humans. The table can be converted into sentences by arranging the cells in a linear manner, such as: "The season of the first row is..., the date is..., and the place is...", and then this information is fed into GPT-3.5 to obtain (sub-question, answer) pairs, and these facts are stored in the fact base. Suppose the sub-question q of the t-th table is t b t YesL tb Mark. The decoding process can be described as:

[0053]

[0054] (2) Implication Tree Generation

[0055] The input of the implication tree generation task includes a premise corpus C and a hypothesis h. The premise corpus C comes from the facts in the fact base, and the hypothesis h is the original question. The goal is to generate an implication tree T that explains the hypothesis h by using the premise subset in C as a building block. The implication tree is represented as a tuple T = (h, L, E, S), where the leaf node l i ∈L is retrieved from the corpus, that is, Internal tree node e i ∈E is an intermediate conclusion, a new sentence that does not exist in the corpus C, and the intermediate conclusion is generated by a large language model (LLM), and s i ∈S is a list of implied steps that explain hypothesis h, which is always the root node and final conclusion of the tree.

[0056] The main function of the implication tree structure generation module is to select leaf nodes and generate the implication tree structure. In this stage, the implication tree structure generation task is performed using predefined symbols, where the facts in the fact base are used as a set of possible leaf nodes and the original question is used as the conclusion. The symbol "&" is used to represent "and" and "->" is used to represent "implication".

[0057] GPT-3.5 is used to generate an implication tree structure, where facts are used as leaf nodes, and multiple leaf nodes are combined to form intermediate nodes. In the implication tree structure generation module, only facts and original questions are given, and the generated intermediate nodes do not contain any specific information. In the initialization stage of the implication tree, the leaf nodes in the implication tree are continuously filled by predicting the structure of the implication tree, and the corresponding intermediate nodes are continuously improved by combining leaf nodes.

[0058] After the structure of the implication tree is predicted based on the question and the implication tree is improved, since the root node does not contain the answer at this time, the root node is replaced with an "answer" placeholder in this embodiment and is input into the second stage to improve the implication tree.

[0059] Use the symbol “&” to represent “and” and “->” to represent “implies”. Suppose the jth original question s j and fact base FB = [fact1,fact2.....,fact m ] is input into GPT-3.5 and generates the implication tree structure t s , the structure contains L t Mark. The decoding process can be described as:

[0060]

[0061] where p t is an indicator.

[0062] (3) Implication Tree Refinement

[0063] The implication tree refinement module mainly completes the refinement of the intermediate nodes in the generated implication tree structure.

[0064] First, the implication tree structure is divided into multiple subtree sets T. If the length of the subtree set is greater than 1, then the following operations are performed on each subtree: the subtree Ti is divided into a node set N, and then the following operations are performed on each node Nj: if the node Nj is in the fact base, it is replaced with the corresponding information in the fact base; otherwise, GPT-3.5 is used to infer other nodes and generate the content of the intermediate nodes. After completing the processing of all subtrees, they are connected to form a complete implication tree. If the length of the subtree set is equal to 1, then the complete implication tree is the subtree set itself.

[0065] In this embodiment, “splitting the implication tree structure into multiple subtree sets T” means starting from the root node of the constructed implication tree structure, continuously obtaining subtree sets T consisting of non-leaf nodes N r and its child nodes N c (Deptth(N c )=Deptth(N r +1)) and continuously add them to the set to form a subtree set T. This embodiment splits all subtree sets in the implication tree structure. If the number of subtrees in the subtree set is greater than 1, each subtree is input into the large model one by one in order, and the root (intermediate node) of each subtree is derived based on the leaf node (fact). Finally, they are merged into a complete implication tree Tree initial .

[0066] 2. Iterative Expert Hybrid Optimization Phase

[0067] Since there may be leaf node selection errors and structural errors in the initialized implication tree, this embodiment uses a mixture of experts (MoE) model to jointly learn the fact retrieval generation task (retrieval leaf nodes) and the question answering task through an iterative feedback mechanism, and correct the implication tree structure.

[0068] (1) Joint Learning of Fact Retrieval Generation and Question Answering

[0069] First, use the T5 encoder (shared encoder) to pre-extract features for each fact in the fact base. Then convert the implication tree into a natural language description and the original question q are input into the shared encoder of T5 to obtain the final feature F et :

[0070]

[0071] Once we have the final features, we feed them into the mixture of experts model.

[0072] The expert mixture model mainly consists of three parts: two gating networks, two task-specific expert networks and a shared expert network. The two task-specific expert networks focus on the fact retrieval generation task and the question answering task respectively, while the shared expert network can be used by both tasks. The gating network is responsible for selecting appropriate experts. Specifically, gating network A selects from the fact retrieval generation task and the shared expert, while gating network B selects from the question answering task and the shared expert.

[0073] Top-2 selection is performed according to formula (9). Top-2 selection means that for a set of expert networks, the set contains n expert sub-networks, and the two expert sub-networks with the highest probability are selected to process the features. When g(·) is a sparse vector, only some experts will be activated and updated during training. In this embodiment, the gating layer is set to Top-K selection:

[0074] g(F et )=TopK(softmax(f(F et ))) (9)

[0075] Among them, f(F et ) Routing linear transformation R D →R E .

[0076] The Token-choice Routing mechanism is used to select different tasks for different tokens to ensure training stability. Given Ex trainable experts and input representation The output of the MoE model can be expressed as:

[0077]

[0078] Among them, e i (F et ) is the nonlinear transformation R of the (i)th expert D →R E At the same time, g(F et ) i is a trainable router g(F et ), which is a nonlinear mapping R D →R E . Usually, e(F et ) and g(F et ) are all parameterized by neural networks. In this embodiment, each expert is a feed-forward network (FFN) layer, rather than a complete Transformer model.

[0079] After selecting experts for fact retrieval generation task and question answering task through gating network, MoE(Fet ) and F et Add together the final MoE result:

[0080] MoE f =add(MoE(F et ),F et ) (11)

[0081] Among them, MoE f Refers to the output of the MoE model for the fact retrieval generation task and the question answering task respectively.

[0082] Then MoE f Input to the decoder of the respective task.

[0083] In this embodiment, the two decoders are processed differently. The fact retrieval generation decoder performs cross-attention operations with all fact features and the shared encoder output, while the question answering decoder only performs cross-attention operations with the implication tree description of the shared encoder output. The implication tree is optimized through the fact retrieval generation model and the iterative feedback mechanism until the question answering model can answer questions based only on the implication tree. Specifically, only the EM (Exact Match) score in QA is used as an indicator until the EM score of the validation set no longer improves.

[0084] Decoder for fact retrieval generation frg Factual feature F f act and the shared encoder output F et To conduct cross-attention:

[0085] w t ,g t =CrossAttention(CrossAttention(Decoder(q),MoE f ),F fact ) (12)

[0086] where w t represents the cross attention weight at time step t.

[0087] Then the decoder frg We start by retrieving at time step |Q|, ie, the length of question q, and then apply the cross entropy loss to it:

[0088]

[0089] where α t ,i represents the cross attention score of the i-th fact at time step t, α t,t+represents the score of the target source (the fact index sequence extracted from the implication tree generated by LLM) at time step t, and M is the number of retrieval steps. Question answering task Decoder qa Decoder and overall feature MoE f : Cross attention:

[0090] g q =softmax(crossattention(Decoder(q),MoE f )) (14)

[0091] Decoder qa We start at time step |Q|, ie we retrieve the length of question q, and then we apply the cross entropy loss to it:

[0092]

[0093] The overall loss function is as follows:

[0094] L=L frg +L qa (16)

[0095] (2) Iterative feedback mechanism

[0096] After completing the fact retrieval generation and question answering tasks, first replace the fact index obtained in the fact retrieval generation task with the corresponding facts in the fact base. Then, concatenate these facts with the final answer and input them into GPT-3.5 as additional information to correct the implication tree and conduct a second round of training. Add a prompt p to the original prompt i fm, uses the prompt information to generate an implication tree structure: "Given the following possibly related facts and possibly correct answers, please generate an implication tree of n words. Fact:Answer:A Question:Q".

[0097] This example uses an iterative feedback mechanism. The input of the iterative feedback mechanism is a fact set f, which comes from the fact retrieval generation task, and an answer a, which comes from the question answering task. At the beginning of the algorithm, the number of iterations is set to n, and f is initialized. i and a i The algorithm then iterates n times. In each iteration, first f i and a i Fill in the prompt to get the complete prompt p c i Then p c i Compared with the p shown in the figure t Connect them. Continue to process the first stage and get a new implication tree Ti Then through T i The second stage steps are carried out to obtain f from the fact retrieval generation task and the question answering task respectively. i new and a new i Finally, update f i and a i f new i and a new i .

[0098] "Hints" provide relatively correct evidence and answers (mainly relatively correct evidence) to the large model. The reason why the output of the small model can correct the implication tree generated by the large model through the iterative feedback mechanism is mainly because the large model often selects too much irrelevant evidence when dealing with complex reasoning problems, and the small model can remove a large amount of irrelevant evidence for the large model because it can retrieve evidence in a supervised manner, so that the large model can generate a more accurate implication tree, thereby helping reasoning and generating answers.

[0099] This example conducts experiments on two representative multimodal question answering (MMQA) datasets: WebQA and MultimodalQA.

[0100] WebQA is a multimodal and multi-hop question answering dataset that contains question pairs that require one or two images and text snippets to answer. Each question has a set of distractors that the model must consider along with the correct clues to provide an answer. WebQA uses BARTScore to measure answer fluency and keyword accuracy, denoted as QA-FL and QA-Acc, respectively, as shown in Table 2. These two scores are multiplied together to get the QA score. Clue retrieval can be easily evaluated by the F1 score.

[0101] MultimodalQA involves answering multi-hop complex questions by combining text, table, and image information. Each question also contains visual and text distractors. Performance is measured by word-level F1 score and ExactMatch (EM) of the predicted answer.

[0102] WebQA and MultimodalQA. The information sources of WebQA include text and image modalities, while MultimodalQA focuses on text, images, and tables. For WebQA and MultimodalQA, a list of candidate clues is provided, and the model needs to find the most relevant clues to evaluate the accuracy of clue retrieval. The backbone of retrieval is BERT. LLaVA-1.5 is used as the VQA model, and T5 is used as the FRG and QA models. Experiments are conducted using the Transformers library and the pre-trained parameters of HuggingFace, and a 24G GPU card is used for experiments. In addition, AdamW is used as the optimization algorithm, the learning rate is 1e-4, and the batch sizes for retrieval and QA are 32 and 12, respectively.

[0103] Table 1

[0104]

[0105] The results on WebQA are shown in Tables 1-3. Table 1 shows all methods using large models on the WebQA dataset. Promptcap trains an image caption model to generate image captions so that GPT-3 can obtain more information about the image. However, the image captions generated by promptcap are too coarse-grained. This embodiment decomposes the questions through GPT-3.5 and filters the questions to generate image captions, which allows the image caption model to focus more on specific areas.

[0106] Table 2

[0107]

[0108] The method of the present invention achieves the highest results in QA-fl, QA-acc, and QA scores.

[0109] Table 3

[0110]

[0111] The present invention is significantly superior to other existing methods in terms of qa accuracy.

[0112] Table 2 shows all the results on the WebQA official test set. MuRAG designed a multimodal Transformer, SKURG designed an entity fusion method, and solar unified multimodal into text. This example lists the results of VitaminC, CMU ITL, HIT TMG, and SDU on the EvalAI WebQA public leaderboard in Table 3.

[0113] Table 4

[0114]

[0115] In addition, as shown in Table 4, this embodiment surpasses the current best results (sota) on the MultimodalQA task of unimodal and multimodal datasets, which shows that the present invention has excellent reasoning ability.

[0116] Both WebQA and MultimodalQA datasets require measuring the accuracy of the final answer. This embodiment achieves higher scores in the accuracy of the final answer than other methods because in the first step of question decomposition, GPT-3.5 is able to fully decompose the question and match the final multimodal clues. This leads to more accurate generation of image descriptions based on the decomposed question, providing a reasoning step for the generation of the final answer. Therefore, good results are achieved in terms of accuracy, as shown in Table 5.

[0117] Table 5

[0118]

[0119] In this embodiment, the fact retrieval generation module is called FRG, and the iterative feedback mechanism is called IFM.

[0120] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An explainable multi-modal multi-hop reasoning question answering method for joint entailment reasoning structure generation, characterized by: The following steps are involved: Decompose multi-step problems and build a fact base based on the decomposition results; Obtaining a premise corpus based on the fact base, and generating an implication tree structure according to predefined symbols, the premise corpus and the original question; The implication tree structure is split to obtain a set of multiple subtrees, and the implication tree structure is refined according to the length of the set of multiple subtrees to obtain a complete implication tree; The process of refining the implication tree structure according to the length of the set of the multiple subtrees to obtain a complete implication tree includes: If the length of the set is greater than 1, the subtree in the set is split into node sets, and the nodes in the node set are traversed. If the node has corresponding information in the fact base, the corresponding information is used to replace it. If there is no corresponding information, the content of the intermediate node is generated based on other nodes in the node set until the length of the set is equal to 1, and a complete implication tree is obtained; An expert mixture model is constructed, and overall features are obtained based on the complete implication tree and the expert mixture model. The complete implication tree is optimized based on the overall features and an iterative feedback mechanism to obtain an updated fact set and answer.

2. The explainable multi-modal multi-hop reasoning question answering method for joint implied reasoning structure generation according to claim 1 is characterized in that: The process of building a fact base includes: A multimodal evidence set is obtained based on multi-step question retrieval, the multi-step question is decomposed based on the multimodal evidence set to generate multiple sub-questions and corresponding evidence, the sub-questions and corresponding evidence are converted and stored based on the pattern of the corresponding evidence to obtain a fact base.

3. The explainable multi-modal multi-hop reasoning question answering method for joint implied reasoning structure generation according to claim 2 is characterized in that: The process of generating the implication tree structure includes: A premise corpus is obtained based on the fact base, the premise corpus and a plurality of sub-questions are symbolically represented according to predefined symbols, and the symbolic representation is input into a natural language processing model to generate an implication tree structure.

4. The explainable multi-modal multi-hop reasoning question answering method for joint implied reasoning structure generation according to claim 1 is characterized in that: The process of obtaining multiple subtree sets includes: Starting from the root node of the constructed implication tree structure, obtain the non-leaf nodes N r and its child nodes, and add them to the collection to obtain a subtree collection, until all nodes are traversed to obtain multiple subtree collections.

5. The explainable multi-modal multi-hop reasoning question answering method for joint implied reasoning structure generation according to claim 1, characterized in that: The process of obtaining fact index and question answer based on the complete implication tree and the expert mixture model includes: The implication tree is converted into a natural language description; a final feature is obtained based on the natural language description; and the final feature is input into the expert mixture model to obtain an overall feature.

6. The explainable multi-modal multi-hop reasoning question answering method for joint implied reasoning structure generation according to claim 5 is characterized in that: The expert mixture model includes two gating networks, two task-specific expert networks and a shared expert network; The final features are input into the expert mixture model, and fact retrieval generation task experts and question answering task experts are selected from a given number of trainable experts through a gating network. The final features enter the task-specific expert network and shared expert network corresponding to the fact retrieval generation task experts and question answering task experts respectively, and the output is added to the final features to obtain the overall features.

7. The explainable multi-modal multi-hop reasoning question answering method for joint implied reasoning structure generation according to claim 5, characterized in that: Extracting features of each fact based on the complete implication tree to obtain a fact feature set; A cross-attention operation is performed on the fact retrieval generation decoder, the fact feature set and the final feature, and a cross-attention operation is performed on the question answer decoder and the natural language description until the cross entropy loss meets the requirements, thereby obtaining a fact index and a final answer.

8. The explainable multi-modal multi-hop reasoning question answering method for joint implied reasoning structure generation according to claim 7 is characterized in that: The process of optimizing the complete implication tree includes: The fact index is replaced with the corresponding fact in the fact library, the replaced fact set is connected with the final answer, and input into the natural language processing model to optimize the complete implication tree.

Images