A knowledge-enhanced traceable multi-modal reasoning method, device and medium

Abstract

The present application relates to the technical field of multi-modal reasoning, and particularly relates to a knowledge-enhanced traceable multi-modal reasoning method and system, the method constructs a structured candidate hypothesis for the current problem, performs explicit confidence evaluation on the candidate visual evidence and the candidate semantic entity, and further utilizes high-confidence evidence to constrain the final reasoning process, thereby ensuring the accuracy of the answer while significantly enhancing the explainability, verifiability and traceability of the reasoning process. The method effectively reduces the influence of text analysis ambiguity and cascading error propagation, and is suitable for various multi-modal reasoning scenes of image and text joint input, such as visual question answering, scientific question answering, disaster event understanding and the like.

Description

Technical Field

[0001] This invention relates to the field of multimodal reasoning technology, and in particular to a knowledge-enhanced traceable multimodal reasoning method, device, and medium. Background Technology

[0002] With the development of multimodal large-scale models and visual language models, completing complex reasoning tasks under joint image and text input conditions has become a key issue in scenarios such as visual question answering, scientific question answering, disaster event understanding, and decision support. In these scenarios, the system not only needs to identify objects, attributes, spatial relationships, and scene semantics in images, but also needs to combine question text, candidate answers, and external common sense information to complete joint judgments. Among these, the correspondence between evidence and conclusions is an important foundation for the reliable operation of multimodal reasoning systems. Knowing the visual regions, entity relationships, and structured semantic evidence relied upon in the reasoning process can directly explain why the model outputs a certain conclusion, and can also serve as intermediate evidence to check, track, and correct the final decision-making process. Therefore, how to obtain localizable, verifiable, and traceable evidence has become an important research topic in the field of multimodal reasoning.

[0003] To address the interpretability and reliability issues of traditional multimodal deep learning methods, researchers have begun to incorporate structured knowledge, chained reasoning, visual localization, and cross-modal verification mechanisms into multimodal reasoning models. This allows the models to explicitly utilize intermediate information for decision support before or during answer generation. Consequently, attention-based multimodal reasoning methods, knowledge-enhanced multimodal reasoning methods, and methods based on multi-stage chained reasoning have gradually emerged in existing technologies. Simultaneously, techniques such as open-vocabulary detectors, image segmentation models, knowledge graphs, and image-text alignment models have also been used for candidate region generation, candidate knowledge organization, and intermediate semantic modeling, enhancing the ability of multimodal reasoning systems to comprehensively utilize image content and text semantics to a certain extent.

[0004] Preliminary research and existing studies indicate that while the aforementioned methods have achieved good results in some multimodal reasoning tasks, the following problems still exist: 1. Attention-based multimodal reasoning methods typically highlight image regions through heatmaps or attention distributions, but their output is essentially a continuous response rather than addressable discrete evidence units, making it difficult to clearly indicate which entity, location, or attribute relationship truly supports the final conclusion; 2. Knowledge-enhanced multimodal reasoning methods typically retrieve triples from external knowledge bases or knowledge graphs to assist decision-making, but existing methods mostly limit knowledge retrieval to the global semantic matching level, lacking specificity for the current context. 3. The joint constraints of the image and the current problem can easily introduce noisy knowledge that is irrelevant to the current task; 4. Although the multi-stage chain reasoning method can provide intermediate steps in text form, there is usually a lack of explicit cross-modal consistency verification between different stages. Errors generated in the previous stage can easily be passed on to subsequent stages, and the final reasoning chain is often just a textual description, which is difficult to form a stable binding with specific visual evidence; 5. Visual localization and semantic verification in existing solutions are usually processed separately. Even if the detector or segmenter can locate the candidate region, the existing technology often lacks a mechanism to determine whether the semantic content of the region truly supports the current reasoning conclusion.

[0005] Therefore, it is necessary to provide a complete method that can construct structured candidate hypotheses around the current problem, explicitly assess the confidence of candidate visual evidence and candidate semantic entities, and further utilize high-confidence evidence to constrain the final reasoning process. How to construct an evidence-driven, verifiable, and traceable multimodal reasoning technology solution? Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a knowledge-enhanced traceable multimodal reasoning method, device, and medium.

[0007] In a first aspect, the present invention provides a knowledge-enhanced traceable multimodal reasoning method, which inputs an image and a question into a multimodal reasoning model and outputs the final answer and corresponding evidence. The multimodal reasoning model includes: S1. Hypothesis-driven knowledge discovery stage: For the input image and question, a pre-trained visual language model is used to generate structured scene description information, and a set of candidate concepts is extracted from the structured scene description information; then, combined with the input image and question, a graph-text joint relevance screening mechanism is used to retain candidate concepts that are simultaneously related to the image content and question semantics, and entities, attributes and candidate relationships are organized based on the screened candidate concepts to establish an initial hypothesis graph for the current task. S2, Dual-channel evidence verification stage: Dual-channel evidence verification is performed on the initial hypothesis graph, multi-granularity visual representation and multi-level semantic representation are extracted, and verification scores between entities and candidate visual regions are generated through cross-modal consistency calculation and adaptive fusion mechanism; based on the verification scores, the region-level scores of candidate visual regions are aggregated to obtain entity-level verification scores, and the entity-level verification scores and their corresponding candidate visual units are written into the initial hypothesis graph to form an enhanced evidence graph with explicit verification scores; S3, Evidence-Constrained Reasoning Stage: Based on the enhanced evidence graph, high-confidence evidence units relevant to the question are selected, and the question context, the selected high-confidence evidence, the supplementary graph structure background information, and the reasoning constraint instructions are organized into hierarchical prompts. These are then input into the visual language model for evidence-constrained reasoning and answer generation, and the final answer and corresponding evidence are output.

[0008] Optionally, in S1, the expression for the image-text joint relevance screening mechanism is as follows: ; ; in, This represents the input image. This indicates the input question. Indicates the first One candidate concept, and These represent the visual encoder and the text encoder, respectively. A balance coefficient representing the relationship between image relevance and question relevance. Indicates the filtering threshold. This represents the selected candidate concepts.

[0009] Optionally, in S1, the expression for the initial hypothesis graph is as follows: ; in, Represents the initial hypothesis graph. Represents a set of entity nodes. Represents a set of relational triples. Represents a set of entity attributes. Represents the association strength matrix between entities. This indicates lightweight search enhancement generation.

[0010] Optionally, in S2, the dual-channel evidence verification includes a perception channel and a knowledge channel, wherein: In the perceptual channel, an open vocabulary detector is used to generate entities in the input image. The corresponding candidate box set, and retaining the top ones by confidence score. There are 1 candidate box, and its expression is: ; in, Indicates the candidate boxes to be retained. Represents the set of candidate boxes. Indicates the candidate box. Indicates the confidence level. This indicates that the ranking based on prior confidence level should be retained. One candidate box; Then, a pixel-level mask is generated for each candidate box using a segmentation model, resulting in a set of candidate visual units: ; in, Represents the set of candidate visual units. Indicates candidate visual units. Represents the segmentation model. Candidate boxes The corresponding pixel-level mask; For each candidate visual unit, a multi-granularity visual representation is extracted. The multi-granularity visual representation includes overall semantic features, coarse-grained spatial structure features, and fine-grained task cue features. The overall semantic features are used to represent the global visual meaning of the candidate region, the coarse-grained spatial structure features are used to represent the distribution relationship of different spatial parts within the candidate region, and the fine-grained task cue features are used to emphasize the local details most relevant to the current reasoning task. Within the knowledge pathway, a three-level progressive semantic enhancement representation is constructed around each candidate entity; By fusing entity text representations with graph context embedding, a first-level entity representation is obtained. : ; in, Text embedding representing entity names, This represents a graph context embedding formed by entity names, attributes, and high-weight neighborhood descriptions. Indicates the first weighting coefficient; The entity representations are weighted and aggregated using graph structure neighborhood information to obtain the second-level entity representation. : ; in, Representation and entity The neighboring nodes with the highest association strength, This represents the normalized neighborhood weights. This represents the second weighting coefficient; By introducing a problem-condition gating mechanism, the semantics of the current problem are injected into the entity representation, resulting in a third-level entity representation. : ; ; ; in, The embedding vector representing the question text. This represents the intermediate representation obtained after performing linear transformation, layer normalization, and nonlinear activation on the second-level entity representation. This represents the gated vector generated based on the question text. It is a non-linear activation function. For layer normalization function, For the Sigmoid function; , , The weight matrix is ​​learned during model training. It is the bias vector; This indicates that two vectors are concatenated. This indicates that elements at corresponding positions are multiplied; After obtaining the three-granularity visual representation of the perceptual channel and the three-level semantic representation of the knowledge channel, the final evidence verification score is obtained through cross-modal consistency calculation and adaptive fusion mechanism. For the first Visual representation at each granularity and entity representation Their similarity The definition is as follows: ; A fusion context vector is constructed by combining multi-granularity visual representations and three-level progressive semantic enhancement representations. A lightweight multilayer perceptron is then used to generate aggregation weights of different granularities, ultimately yielding a validation score between the entity and candidate visual regions. : ; in, This represents a multilayer perceptron. Adaptive weights representing similarity at different granularities. This represents the enhanced weights for maximum granularity similarity.

[0011] Optionally, in S2, the process of generating the enhanced evidence map includes: The regional-level validation scores are aggregated into entity-level validation scores using the following expression: ; in, Indicates the entity-level verification score; Set of candidate visual units The corresponding regional and entity-level scores are written into the initial hypothesis graph to form an enhanced evidence graph. .

[0012] Optionally, in S3, high-confidence evidence units are selected from the enhanced evidence map for entities with candidate visual regions. The candidate visual unit with the highest verification score was selected as its representative evidence. : ; For triples The evidence supports the score. Defined as entity-level validation score at two endpoints and Average value: ; Only evidence supporting a score not lower than the threshold is retained. And the sorting is in front The evidence units, along with the problem context, graph structure background semantics, and reasoning constraint instructions, constitute hierarchical prompt information. The hierarchical prompt information Including the problem context Selected evidence Image background semantics and structured instructions ; When confirming the answer, a two-stage reasoning strategy is adopted. In the first stage, a reasoning chain is generated under the constraint of high-confidence evidence. In the second stage, based on the reasoning chain, the conditional probabilities of the candidate answer labels are compared to obtain the final answer. Its expression is: ; in, This represents the set of single-labels corresponding to candidate answers. Indicates the first Each candidate answer corresponds to a single tag. Indicates the first 10 candidate answers This represents the index of the candidate answer that maximizes the conditional probability; This represents a visual language model with parameter θ. Indicates input Reasoning chain and evidence constraints Under the conditions, the model Output candidate answer tags The conditional probability.

[0013] Optionally, model training is also included before using the multimodal inference model for inference, as follows: Set the model to the training state and initialize the relevant parameters of the visual language model, text encoder, knowledge organization module, detector, segmenter and dual-channel verification module; Input images and questions from the training samples, and generate structured scene descriptions, filter candidate concepts, and construct an initial hypothesis graph in the hypothesis-driven knowledge discovery stage; The dual-channel evidence verification stage is used to perform candidate region localization, multi-granularity visual representation extraction, three-level semantic enhancement, cross-modal consistency calculation, and entity-level score aggregation on candidate entities to obtain an enhanced evidence map; Training labels are constructed using images and entity-level weakly supervised signals, and the total loss is calculated jointly based on ranking loss and regression loss. Update the parameters of the dual-channel validation module based on the total loss, and repeat the above process until the model converges or reaches the preset number of training rounds.

[0014] Optionally, during model training, a weak supervision signal can be defined using the global similarity between images and entity text. : ; in, Represents an image. Represents entity text; A dual-threshold strategy is used to divide the samples into positive and negative samples, and the validation module is jointly optimized using ranking loss and regression loss. The total loss is... Defined as: ; in, and These are the adjustable weights corresponding to the ranking loss and regression loss, respectively; Among them, the ranking loss for: ; Regression loss for: ; in, This represents the set of sample pairs consisting of positive and negative samples. Indicates the number of sample pairs; Indicates a positive sample. Indicates a negative sample; This represents the entity corresponding to the positive sample. This represents the entity corresponding to the negative sample; This represents the interval parameter in the ranking loss; The evidence verification score represents the score of the positive sample entity. The evidence verification score represents the score of the negative sample entity; This represents the function that takes the maximum value. This represents the set of entity-level training samples in the current training batch. This indicates the number of samples in the training batch; Indicates the input image. Indicates candidate entities, Indicates candidate entities The corresponding set of candidate visual regions; Indicates based on the image With candidate entities The weakly supervised target score is obtained from the image-text similarity between the two. During training, the visual encoder, text encoder, and visual language model body are frozen, and only the projection layer, gating parameters, fusion parameters, and similarity aggregation parameters in the dual-channel validation module are updated.

[0015] In addition, the present invention also provides a computer device, including a memory and a processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the traceable multimodal reasoning method described above.

[0016] In addition, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the traceable multimodal reasoning method described above.

[0017] The application of the technical solution of the present invention has at least the following beneficial effects: This invention provides a knowledge-enhanced, traceable multimodal reasoning method. By dividing the multimodal reasoning process into three stages—task-related knowledge discovery, fine-grained evidence verification, and evidence-constrained answer decision—a clear correspondence is established between candidate hypotheses, verification evidence, and the final answer. This significantly improves the interpretability, verifiability, and traceability of the reasoning process. Compared to existing methods that directly generate answers in a single forward process, this invention can explicitly specify which entities, relationships, and visual regions the final conclusion depends on.

[0018] The method of this invention constructs an initial hypothesis graph oriented towards the current problem in the first stage, enabling the formation of structured candidate hypotheses before answer generation and providing actionable entity anchors for subsequent visual evidence localization and semantic consistency verification. Compared to existing technologies that rely solely on global semantic matching, the method of this invention reduces the entry of irrelevant concepts and noisy nodes into the subsequent reasoning process, thereby improving the targeting and effectiveness of subsequent evidence screening.

[0019] This invention unifies candidate visual region localization, entity semantic enhancement, and cross-modal consistency assessment within a single technical framework through a dual-channel evidence verification mechanism. This mechanism extracts complementary information from multi-granularity visual region representations and multi-level entity semantic representations, respectively, and generates an explicit verification score through adaptive fusion to determine whether candidate visual regions support the current reasoning conclusion. Compared to verification methods that rely solely on coarse-grained global image-text similarity, this invention establishes a finer-grained consistency judgment relationship between region-level visual evidence, entity semantic information, and the question context, thereby improving the accuracy and robustness of evidence selection and reducing the interference of irrelevant or weakly related visual regions on subsequent reasoning processes, demonstrating significant technical advantages.

[0020] This invention constructs an enhanced evidence graph and organizes hierarchical prompts based on high-confidence evidence, ensuring that the final reasoning process is always constrained by explicit evidence. This effectively reduces problems such as unfounded generation, interference from noisy knowledge, and the cumulative propagation of errors across multiple stages, thereby improving the reliability and stability of the answer decision. Furthermore, through a two-stage answer confirmation mechanism, a reasoning chain supported by evidence is first generated, and then the conditional probabilities of candidate answer identifiers are compared to confirm the final answer, which reduces analytical ambiguity caused by freely generated text. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the steps of the traceable multimodal reasoning method in a preferred embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1: like Figure 1 As shown, this embodiment provides a knowledge-enhanced traceable multimodal reasoning method that inputs an image and a question into a multimodal reasoning model and outputs the final answer and corresponding evidence.

[0025] In this embodiment, the input image and question include an experimental image and a corresponding question text. The question requires determining the correct conclusion from multiple candidate answers based on the experimental apparatus, component attributes, and their connections in the image. This type of example simultaneously involves target recognition in the image, spatial relationship understanding, attribute judgment, and question semantic matching, and can well reflect the entire process of evidence discovery, evidence verification, and answer decision in multimodal reasoning.

[0026] Based on the above task scenario, this embodiment constructs an evidence-driven multimodal reasoning model. The model generally includes three stages: a hypothesis-driven knowledge discovery stage, a dual-channel evidence verification stage, and an evidence-constrained reasoning stage. The first stage constructs an initial hypothesis graph relevant to the task based on the input image and input question. The second stage locates candidate visual regions in the image around candidate entities in the initial hypothesis graph and calculates the consistency score between candidate evidence and candidate entities by combining graph structure semantic information, thereby obtaining an enhanced evidence graph. The third stage filters high-confidence evidence based on the enhanced evidence graph and uses the high-confidence evidence to organize hierarchical prompts, guiding the frozen visual language model to output the final answer.

[0027] The multimodal reasoning model includes (steps S1 to S3): S1. Hypothesis-driven knowledge discovery stage: For the input image and question, a pre-trained visual language model is used to generate structured scene description information, and a set of candidate concepts is extracted from the structured scene description information; then, combined with the input image and question, a graph-text joint relevance screening mechanism is used to retain candidate concepts that are simultaneously related to the image content and question semantics, and entities, attributes and candidate relationships are organized based on the screened candidate concepts to establish an initial hypothesis graph for the current task.

[0028] In this embodiment, the expression for the image-text joint relevance filtering mechanism is as follows: ; ; in, This represents the input image. This indicates the input question. Indicates the first One candidate concept, and These represent the visual encoder and the text encoder, respectively. A balance coefficient representing the relationship between image relevance and question relevance. Indicates the filtering threshold. This represents the selected candidate concepts.

[0029] Specifically, the expression for the initial hypothesis graph is as follows: ; in, Represents the initial hypothesis graph. Represents a set of entity nodes. Represents a set of relational triples. Represents a set of entity attributes. Represents the association strength matrix between entities. This indicates lightweight search enhancement generation.

[0030] S2, Dual-channel evidence verification stage: Dual-channel evidence verification is performed on the initial hypothesis graph, multi-granularity visual representation and multi-level semantic representation are extracted, and verification scores between entities and candidate visual regions are generated through cross-modal consistency calculation and adaptive fusion mechanism; based on the verification scores, the region-level scores of candidate visual regions are aggregated to obtain entity-level verification scores, and the entity-level verification scores and their corresponding candidate visual units are written into the initial hypothesis graph to form an enhanced evidence graph with explicit verification scores.

[0031] In this embodiment, the dual-channel evidence verification includes a perception channel and a knowledge channel. The perception channel is used to locate candidate visual regions corresponding to candidate entities in the input image and extract multi-granularity visual representations such as global visual features, regional visual features, and local detail visual features. The knowledge channel is used to construct progressively enhanced semantic representations around candidate entities. These progressive enhancements include at least semantic enhancement of the entity itself, semantic enhancement of the neighborhood structure, and semantic enhancement under problem conditions, wherein: In the perceptual channel, an open vocabulary detector is used to generate entities in the input image. The corresponding candidate box set, and retaining the top ones by confidence score. There are 1 candidate box, and its expression is: ; in, Indicates the candidate boxes to be retained. Represents the set of candidate boxes. Indicates the candidate box. Indicates the confidence level. This indicates that the ranking based on prior confidence level should be retained. One candidate box; Then, a pixel-level mask is generated for each candidate box using a segmentation model, resulting in a set of candidate visual units: ; in, Represents the set of candidate visual units. Indicates candidate visual units. Represents the segmentation model. Candidate boxes The corresponding pixel-level mask; For each candidate visual unit, a multi-granularity visual representation is extracted. The multi-granularity visual representation includes overall semantic features, coarse-grained spatial structure features, and fine-grained task cue features. The overall semantic features are used to represent the global visual meaning of the candidate region, the coarse-grained spatial structure features are used to represent the distribution relationship of different spatial parts within the candidate region, and the fine-grained task cue features are used to emphasize the local details most relevant to the current reasoning task.

[0032] Within the knowledge pathway, a three-level progressive semantic enhancement representation is constructed around each candidate entity; By fusing entity text representations with graph context embedding, a first-level entity representation is obtained. : ; in, Text embedding representing entity names, This represents a graph context embedding formed by entity names, attributes, and high-weight neighborhood descriptions. Indicates the first weighting coefficient; The entity representations are weighted and aggregated using graph structure neighborhood information to obtain the second-level entity representation. : ; in, Representation and entity The neighboring nodes with the highest association strength, This represents the normalized neighborhood weights. This represents the second weighting coefficient; By introducing a problem-condition gating mechanism, the semantics of the current problem are injected into the entity representation, resulting in a third-level entity representation. : ; ; ; in, The embedding vector representing the question text. This represents the intermediate representation obtained after performing linear transformation, layer normalization, and nonlinear activation on the second-level entity representation. This represents the gated vector generated based on the question text. It is a non-linear activation function. For layer normalization function, For the Sigmoid function; , , The weight matrix is ​​learned during model training. It is the bias vector; This indicates that two vectors are concatenated. This indicates that elements at corresponding positions are multiplied; After obtaining the three-granularity visual representation of the perceptual channel and the three-level semantic representation of the knowledge channel, the final evidence verification score is obtained through cross-modal consistency computation and adaptive fusion mechanism.

[0033] For the first Visual representation at each granularity and entity representation Their similarity The definition is as follows: ; A fusion context vector is constructed by combining multi-granularity visual representations and three-level progressive semantic enhancement representations. A lightweight multilayer perceptron is then used to generate aggregation weights of different granularities, ultimately yielding a validation score between the entity and candidate visual regions. : ; in, This represents a multilayer perceptron. Adaptive weights representing similarity at different granularities. This represents the enhanced weights for maximum granularity similarity.

[0034] Furthermore, the process of generating the enhanced evidence map includes: The regional-level validation scores are aggregated into entity-level validation scores using the following expression: ; in, Indicates the entity-level verification score; Set of candidate visual units The corresponding regional and entity-level scores are written into the initial hypothesis graph to form an enhanced evidence graph. .

[0035] S3, Evidence-Constrained Reasoning Stage: Based on the enhanced evidence graph, high-confidence evidence units relevant to the question are selected, and the question context, the selected high-confidence evidence, the supplementary graph structure background information, and the reasoning constraint instructions are organized into hierarchical prompts. These are then input into the visual language model for evidence-constrained reasoning and answer generation, and the final answer and corresponding evidence are output.

[0036] In this embodiment, high-confidence evidence units are screened from the enhanced evidence image, specifically for entities with candidate visual regions. The candidate visual unit with the highest verification score was selected as its representative evidence. : ; For triples The evidence supports the score. Defined as entity-level validation score at two endpoints and Average value: ; Only evidence supporting a score not lower than the threshold is retained. And the sorting is in front The evidence units, along with the problem context, graph structure background semantics, and reasoning constraint instructions, constitute hierarchical prompt information. The hierarchical prompt information Including the problem context Selected evidence Image background semantics and structured instructions ; When confirming the answer, a two-stage reasoning strategy is adopted. In the first stage, a reasoning chain is generated under the constraint of high-confidence evidence. In the second stage, based on the reasoning chain, the conditional probabilities of the candidate answer labels are compared to obtain the final answer. Its expression is: ; in, This represents the set of single-labels corresponding to candidate answers. Indicates the first Each candidate answer corresponds to a single tag. Indicates the first 10 candidate answers This represents the index of the candidate answer that maximizes the conditional probability; This represents a visual language model with parameter θ. Indicates input Reasoning chain and evidence constraints Under the conditions, the model Output candidate answer tags The conditional probability.

[0037] Optionally, model training is also included before using the multimodal inference model for inference, as follows: Set the model to the training state and initialize the relevant parameters of the visual language model, text encoder, knowledge organization module, detector, segmenter and dual-channel verification module; Input images and questions from the training samples, and generate structured scene descriptions, filter candidate concepts, and construct an initial hypothesis graph in the hypothesis-driven knowledge discovery stage; The dual-channel evidence verification stage is used to perform candidate region localization, multi-granularity visual representation extraction, three-level semantic enhancement, cross-modal consistency calculation, and entity-level score aggregation on candidate entities to obtain an enhanced evidence map; Training labels are constructed using images and entity-level weakly supervised signals, and the total loss is calculated jointly based on ranking loss and regression loss. Update the parameters of the dual-channel validation module based on the total loss, and repeat the above process until the model converges or reaches the preset number of training rounds.

[0038] Optionally, during model training, a weak supervision signal can be defined using the global similarity between images and entity text. : ; in, Represents an image. Represents entity text; A dual-threshold strategy is used to divide the samples into positive and negative samples, and the validation module is jointly optimized using ranking loss and regression loss. The total loss is... Defined as: ; in, and These are the adjustable weights corresponding to the ranking loss and regression loss, respectively; Among them, the ranking loss for: ; Regression loss for: ; in, This represents the set of sample pairs consisting of positive and negative samples. Indicates the number of sample pairs; Indicates a positive sample. Indicates a negative sample; This represents the entity corresponding to the positive sample. This represents the entity corresponding to the negative sample; This represents the interval parameter in the ranking loss; The evidence verification score represents the score of the positive sample entity. The evidence verification score represents the score of the negative sample entity; This represents the function that takes the maximum value. This represents the set of entity-level training samples in the current training batch. This indicates the number of samples in the training batch; Indicates the input image. Indicates candidate entities, Indicates candidate entities The corresponding set of candidate visual regions; Indicates based on the image With candidate entities The weakly supervised target score is obtained from the image-text similarity between the two. During training, the visual encoder, text encoder, and visual language model body are frozen, and only the projection layer, gating parameters, fusion parameters, and similarity aggregation parameters in the dual-channel validation module are updated.

[0039] To verify the technical effectiveness of the method proposed in this application, a simulation experiment was conducted using the publicly available multimodal science question-answering dataset ScienceQA as the test dataset. The samples in this dataset include question text, candidate answers, and image information corresponding to some questions, which can be used to evaluate the model's multimodal reasoning ability under conditions involving visual information, textual semantic information, and structured evidence.

[0040] In this experiment, several existing methods were selected as comparison objects, including the basic multimodal visual language model Qwen2.5-VL-7B, the DDCoT method based on multimodal thinking chains, the VaLiK Image-only method based on visual knowledge graphs, the VaLiK Text-only method based on text knowledge graphs, and the VaLiK Text-Image-7B method based on text-image knowledge graphs. The average accuracy rate was used as the test metric; a higher value indicates more accurate inference results. The experimental results are shown in Table 1.

[0041] Table 1 Experimental Results

[0042] As shown in Table 1, compared with the basic multimodal visual language model Qwen2.5-VL-7B, the average accuracy of the method in this embodiment is improved from 73.80% to 84.03%, an improvement of 10.23 percentage points; compared with the DDDOT method, the method in this embodiment is improved by 3.88 percentage points; compared with the VaLiK Image-only method, the method in this embodiment is improved by 5.15 percentage points; compared with the VaLiK Text-only method, the method in this application is improved by 1.05 percentage points; and compared with the VaLiKText-Image-7B method with similar parameter scale, the method in this application is still improved by 0.87 percentage points.

[0043] The above results indicate that while existing basic visual language models can directly process image and text inputs, they lack explicit organization and verification of intermediate evidence, making them susceptible to irrelevant visual regions or implicit reasoning biases. Existing knowledge graph augmentation methods, while capable of introducing structured knowledge, are insufficient in modeling the supporting relationships between candidate visual regions, entity semantics, and question context. This embodiment's method constructs a local hypothesis graph, generates an evidence graph with verification scores, and uses high-confidence structured evidence for subsequent reasoning. This allows the model to filter and constrain the correspondences between candidate entities, visual regions, and question semantics before answer generation, thereby improving reasoning accuracy in multimodal question answering tasks.

[0044] Therefore, the method of this embodiment has at least the following beneficial effects: it can reduce the interference of irrelevant visual areas and noisy knowledge on the reasoning results; it can improve the consistency judgment ability between candidate evidence and the current question; it can make the final answer generation subject to explicit evidence, thereby reducing the risk of unfounded reasoning and the spread of erroneous knowledge, and improving the accuracy and stability of multimodal reasoning results.

[0045] The above simulation experiments are only used to illustrate the technical effects that this application can achieve under one implementation method, and do not constitute a limitation on the applicable dataset, basic model, task type or evaluation index of this application.

[0046] In addition, this embodiment also provides a computer device, including a memory and a processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the traceable multimodal reasoning method described above.

[0047] It should be noted that computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0048] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0049] In addition, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the traceable multimodal reasoning method described above.

[0050] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described traceable multimodal reasoning method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the traceable multimodal reasoning method provided in the above embodiments, and will not be repeated here.

[0051] The above description is only a preferred embodiment of the present invention and does not limit the scope of the present invention. All equivalent structural transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of the present invention.

Claims

1. A knowledge-enhanced traceable multimodal reasoning method, characterized in that, An image and a question are input into a multimodal reasoning model, which outputs the final answer and corresponding evidence. The multimodal reasoning model includes: S1. Hypothesis-driven knowledge discovery stage: For the input image and question, a pre-trained visual language model is used to generate structured scene description information, and a set of candidate concepts is extracted from the structured scene description information; then, combined with the input image and question, a graph-text joint relevance screening mechanism is used to retain candidate concepts that are simultaneously related to the image content and question semantics, and entities, attributes and candidate relationships are organized based on the screened candidate concepts to establish an initial hypothesis graph for the current task. S2, Dual-channel evidence verification stage: Dual-channel evidence verification is performed on the initial hypothesis graph, multi-granularity visual representation and multi-level semantic representation are extracted, and verification scores between entities and candidate visual regions are generated through cross-modal consistency calculation and adaptive fusion mechanism; based on the verification scores, the region-level scores of candidate visual regions are aggregated to obtain entity-level verification scores, and the entity-level verification scores and their corresponding candidate visual units are written into the initial hypothesis graph to form an enhanced evidence graph with explicit verification scores; The dual-channel evidence verification includes a perception channel and a knowledge channel; The multi-granularity visual representation includes overall semantic features, coarse-grained spatial structure features, and fine-grained task cue features. The overall semantic features are used to represent the global visual meaning of the candidate region, the coarse-grained spatial structure features are used to represent the distribution relationship of different spatial parts within the candidate region, and the fine-grained task cue features are used to emphasize the local details most relevant to the current reasoning task. Within the knowledge pathway, a multi-level semantic representation is constructed around each candidate entity. This multi-level semantic representation is a three-level progressive semantic enhancement representation, and the construction process includes: The entity text representation is fused with the graph context embedding to obtain the first-level entity representation; The entity representations are weighted and aggregated using graph structure neighborhood information to obtain the second-level entity representations; A problem condition gating mechanism is introduced to inject the current problem semantics into the entity representation, resulting in a third-level entity representation; S3, Evidence-Constrained Reasoning Stage: Based on the enhanced evidence graph, high-confidence evidence units relevant to the question are selected, and the question context, the selected high-confidence evidence, the supplementary graph structure background information, and the reasoning constraint instructions are organized into hierarchical prompts. These are then input into the visual language model for evidence-constrained reasoning and answer generation, and the final answer and corresponding evidence are output.

2. The traceable multimodal reasoning method according to claim 1, characterized in that, In S1, the expression for the image-text joint relevance screening mechanism is as follows: ； ； in, This represents the input image. This indicates the input question. Indicates the first One candidate concept, and These represent the visual encoder and the text encoder, respectively. A balance coefficient representing the relationship between image relevance and question relevance. Indicates the filtering threshold. This represents the selected candidate concepts.

3. The traceable multimodal reasoning method according to claim 2, characterized in that, In S1, the expression for the initial hypothesis graph is as follows: ； in, Represents the initial hypothesis graph. Represents a set of entity nodes. Represents a set of relational triples. Represents a set of entity attributes. Represents the association strength matrix between entities. This indicates lightweight search enhancement generation.

4. The traceable multimodal reasoning method according to claim 2, characterized in that, In S2, the dual-channel evidence verification includes a perception channel and a knowledge channel, wherein: In the perceptual channel, an open vocabulary detector is used to generate entities in the input image. e The corresponding candidate box set, and retaining the top ones by confidence score. N There are 1 candidate box, and its expression is: ； in, Indicates the candidate boxes to be retained. Represents the set of candidate boxes. Indicates the candidate box. Indicates the confidence level. This indicates that the ranking based on prior confidence level should be retained. N One candidate box; Then, a pixel-level mask is generated for each candidate box using a segmentation model, resulting in a set of candidate visual units: ； in, Represents the set of candidate visual units. Indicates candidate visual units. Represents the segmentation model. Candidate boxes The corresponding pixel-level mask; For each candidate visual unit, a multi-granularity visual representation is extracted. The multi-granularity visual representation includes overall semantic features, coarse-grained spatial structure features, and fine-grained task cue features. The overall semantic features are used to represent the global visual meaning of the candidate region, the coarse-grained spatial structure features are used to represent the distribution relationship of different spatial parts within the candidate region, and the fine-grained task cue features are used to emphasize the local details most relevant to the current reasoning task. Within the knowledge pathway, a three-level progressive semantic enhancement representation is constructed around each candidate entity; By fusing entity text representations with graph context embedding, a first-level entity representation is obtained. : ； in, Text embedding representing entity names, This represents a graph context embedding formed by entity names, attributes, and high-weight neighborhood descriptions. Indicates the first weighting coefficient; The entity representations are weighted and aggregated using graph structure neighborhood information to obtain the second-level entity representation. : ； in, Representation and entity e The neighboring nodes with the highest association strength, This represents the normalized neighborhood weights. This represents the second weighting coefficient; By introducing a problem-condition gating mechanism, the semantics of the current problem are injected into the entity representation, resulting in a third-level entity representation. : ； ； ； in, The embedding vector representing the question text. This represents the intermediate representation obtained after performing linear transformation, layer normalization, and nonlinear activation on the second-level entity representation. This represents the gated vector generated based on the question text. It is a non-linear activation function. For layer normalization function, for function; , , The weight matrix is ​​learned during model training. It is the bias vector; This indicates that two vectors are concatenated. This indicates that elements at corresponding positions are multiplied; After obtaining the three-granularity visual representation of the perceptual channel and the three-level semantic representation of the knowledge channel, the final evidence verification score is obtained through cross-modal consistency calculation and adaptive fusion mechanism. For the first i Visual representation at each granularity and entity representation Their similarity The definition is as follows: ； A fusion context vector is constructed by combining multi-granularity visual representations and three-level progressive semantic enhancement representations. A lightweight multilayer perceptron is then used to generate aggregation weights of different granularities, ultimately yielding a validation score between the entity and candidate visual regions. : ； in, This represents a multilayer perceptron. Adaptive weights representing similarity at different granularities. This represents the enhanced weights for maximum granularity similarity.

5. The traceable multimodal reasoning method according to claim 4, characterized in that, In S2, the process of generating the enhanced evidence map includes: The regional-level validation scores are aggregated into entity-level validation scores using the following expression: ； in, Indicates the entity-level verification score; Set of candidate visual units The corresponding regional and entity-level scores are written into the initial hypothesis graph to form an enhanced evidence graph. .

6. The traceable multimodal reasoning method according to claim 5, characterized in that, In S3, high-confidence evidence units are selected from the enhanced evidence map for entities with candidate visual regions. e The candidate visual unit with the highest verification score was selected as its representative evidence. : ； For triples The evidence supports the score. Defined as entity-level validation score at two endpoints and Average value: ； Only evidence supporting a score not lower than the threshold is retained. And the sorting is in front The evidence units, along with the problem context, graph structure background semantics, and reasoning constraint instructions, constitute hierarchical prompt information. P The hierarchical prompt information P Including the problem context Selected evidence Image background semantics and structured instructions ; When confirming the answer, a two-stage reasoning strategy is adopted. In the first stage, a reasoning chain is generated under the constraint of high-confidence evidence. In the second stage, based on the reasoning chain, the conditional probabilities of the candidate answer labels are compared to obtain the final answer. Its expression is: ； in, This represents the set of single-labels corresponding to candidate answers. Indicates the first k Each candidate answer corresponds to a single tag. This represents the k-th candidate answer. This represents the index of the candidate answer that maximizes the conditional probability; This represents a visual language model with parameter θ. Indicates input Reasoning chain and evidence constraints Under the conditions, the model Output candidate answer tags The conditional probability.

7. The traceable multimodal reasoning method according to claim 6, characterized in that, Before using a multimodal inference model for inference, model training is also included, as follows: Set the model to the training state and initialize the relevant parameters of the visual language model, text encoder, knowledge organization module, detector, segmenter and dual-channel verification module; Input images and questions from the training samples, and generate structured scene descriptions, filter candidate concepts, and construct an initial hypothesis graph in the hypothesis-driven knowledge discovery stage; The dual-channel evidence verification stage is used to perform candidate region localization, multi-granularity visual representation extraction, three-level semantic enhancement, cross-modal consistency calculation, and entity-level score aggregation on candidate entities to obtain an enhanced evidence map; Training labels are constructed using images and entity-level weakly supervised signals, and the total loss is calculated jointly based on ranking loss and regression loss. Update the parameters of the dual-channel validation module based on the total loss, and repeat the above process until the model converges or reaches the preset number of training rounds.

8. The traceable multimodal reasoning method according to claim 7, characterized in that, In model training, a weak supervision signal is defined using the global similarity between images and entity text. : ； in, Represents an image. Represents entity text; A dual-threshold strategy is used to divide the samples into positive and negative samples, and the validation module is jointly optimized using ranking loss and regression loss. The total loss is... Defined as: ； in, and These are the adjustable weights corresponding to the ranking loss and regression loss, respectively; Among them, the ranking loss for: ； Regression loss for: ； in, This represents the set of sample pairs consisting of positive and negative samples. Indicates the number of sample pairs; Indicates a positive sample. Indicates a negative sample; This represents the entity corresponding to the positive sample. This represents the entity corresponding to the negative sample; This represents the interval parameter in the ranking loss; The evidence verification score represents the score of the positive sample entity. The evidence verification score represents the score of the negative sample entity; This represents the function that takes the maximum value. This represents the set of entity-level training samples in the current training batch. This indicates the number of samples in the training batch; Indicates the input image. Indicates candidate entities, Indicates candidate entities The corresponding set of candidate visual regions; This represents the weakly supervised target score obtained based on the image-text similarity between image I and candidate entity e; During training, the visual encoder, text encoder, and visual language model body are frozen, and only the projection layer, gating parameters, fusion parameters, and similarity aggregation parameters in the dual-channel validation module are updated.

9. A computer device, characterized in that, Including memory and processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the traceable multimodal reasoning method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the traceable multimodal reasoning method as described in any one of claims 1 to 8.

Images