Multi-stage inference optimization method and device based on cross-modal consistency detection
By employing cross-modal consistency detection and conflict-driven re-inference mechanisms, the problems of inter-modal contradiction detection and weight adjustment in multimodal reasoning are solved, achieving highly accurate and reliable multimodal information fusion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies cannot automatically detect inconsistencies between modalities in multimodal input scenarios, leading to misleading inference results. Furthermore, fixed-weight fusion cannot adapt to changes in data quality, reducing the accuracy and reliability of multimodal inference.
A cross-modal consistency detection method is adopted, which generates a comprehensive conflict score through semantic matching, numerical attribute comparison and logical relationship verification. When the score exceeds the threshold, a conflict-driven re-inference mechanism is triggered to dynamically adjust the modal weights and perform targeted re-inference to correct errors.
It improves the accuracy and reliability of multimodal reasoning, can automatically detect contradictions between modalities and correct errors, achieves adaptive fusion, and enhances the adaptability to complex scenarios.
Smart Images
Figure CN121684058B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a multi-stage reasoning optimization method and apparatus based on cross-modal consistency detection. Background Technology
[0002] In the field of artificial intelligence, large multimodal models are widely used in complex reasoning tasks, such as visual question answering, image description generation, and multimodal information retrieval. These tasks involve data input from multiple modalities, including text, images, and audio. These applications require models to efficiently fuse information from different modalities to achieve accurate reasoning and decision-making.
[0003] In related technologies, multimodal information fusion mainly employs feature-level fusion, decision-level fusion, and multi-round inference verification. Feature-level fusion directly concatenates or weights feature vectors from different modalities to form a unified multimodal feature representation, which is then input into the inference module for processing. Decision-level fusion performs independent inference for each modality, obtaining its own decision results, and then makes a final decision through methods such as voting and weighted summation. Multi-round inference verification performs preliminary inference on the multimodal input, checks the logical consistency of the inference results through a verification mechanism, and if the verification fails, a second round of inference is performed.
[0004] However, in multimodal input scenarios, different modalities may produce contradictory inference results. Existing technologies cannot automatically detect these contradictions, leading to misleading inference results and ultimately incorrect answers. Once the model misinterprets information from a particular modality in the early stages of inference, the error propagates throughout the entire inference chain. The lack of automatic checking and correction mechanisms reduces the accuracy of inference. Secondly, existing methods typically use fixed weights for information fusion, failing to dynamically adjust the weights of each modality based on the quality of the input data, thus dragging down the overall reliability of inference. Summary of the Invention
[0005] This application provides a multi-stage inference optimization method and apparatus based on cross-modal consistency detection. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.
[0006] In a first aspect, embodiments of this application provide a multi-stage inference optimization method based on cross-modal consistency detection, the method comprising:
[0007] In response to the inference node carrying multimodal data, independent modal inference is performed on each modal data to obtain the initial inference result of each modal data;
[0008] Cross-modal consistency detection is performed based on the initial inference results of each modality data to obtain a comprehensive conflict score. Cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification. Semantic matching is used to determine whether the semantic content of different initial inference results expresses the same or similar meaning. Numerical attribute comparison is used to determine whether there is numerical consistency in the numerical information in different initial inference results. Logical relationship verification is used to verify whether the logical relationship and causal relationship in different initial inference results are the same.
[0009] If the overall conflict score is greater than the preset threshold, the inference node is marked as a high-risk inference node, and the conflict-driven re-inference mechanism based on the preset multi-strategy is triggered to perform re-inference and obtain the final inference results of each modality data.
[0010] The final inference results of each modality are fused together to obtain the final answer corresponding to the inference node.
[0011] Optionally, cross-modal consistency detection is performed based on the initial inference results of each modality to obtain a comprehensive conflict score, including:
[0012] Semantic matching is performed on the initial inference results of each modality to determine whether the semantic content of different initial inference results expresses the same or similar meaning, and a semantic consistency score is obtained.
[0013] The initial inference results of each modality data are compared with numerical attributes to determine whether there is numerical consistency in the numerical information in different initial inference results, and thus obtain the attribute matching consistency score.
[0014] Logical relationship verification is performed on the initial inference results of each modality data to verify whether the logical and causal relationships in different initial inference results are the same, and a logical relationship consistency score is obtained.
[0015] The semantic consistency score, attribute matching consistency score, and logical relationship consistency score are weighted and summed to obtain the comprehensive conflict score.
[0016] Optionally, semantic matching is performed on the initial inference results of each modality, including:
[0017] A pre-defined semantic similarity model is used to perform a semantic-level comparative analysis of the initial inference results of each modality data in order to calculate the semantic distance between the initial inference results of each modality data.
[0018] Semantic distance is used as a semantic consistency score.
[0019] Optionally, the initial inference results for each modality can be compared in terms of numerical attributes, including:
[0020] Extract key numerical and attribute information from the initial inference results of each modality data;
[0021] Establish an attribute correspondence matrix for each modality of data by using numerical and attribute information;
[0022] Compare the consistency of structured attributes in the attribute correspondence matrix of each modality data to generate an attribute matching score.
[0023] Optionally, the initial inference results for each modality data are subjected to logical relationship verification, including:
[0024] Extract the logical and causal relationships between the modal data from the initial inference results of each modality;
[0025] The logical and causal relationships of each modality data are formally represented to obtain the logical relationship diagram of each modality data;
[0026] Examine the isomorphism and consistency of the logical relationship diagrams of each modality to verify the cross-modal consistency of the reasoning logic and generate a logical relationship consistency score.
[0027] Optionally, the preset multiple strategies include an enhanced prompt-based reasoning strategy, a region-based and local reasoning strategy, a constrained reasoning strategy, and a reasoning path backtracking strategy; among them,
[0028] The enhanced prompt word re-inference strategy guides the model to re-infer by constructing enhanced prompt word templates containing conflict descriptions; the region pruning and local inference strategy locates and prunes the relevant regions when conflicts are located in specific regions or attributes of the input data, and re-infers the pruned local information; the constraint inference strategy extracts logical constraints from the initial inference results and domain knowledge of each modality, formally represents them as a constraint set, and performs inference under the constraints of the constraint set; the inference path backtracking strategy automatically identifies the first divergence point in the inference process by comparing the inference chains of different modalities and restarts the inference process from that divergence point.
[0029] Optionally, trigger a conflict-driven re-inference mechanism based on a preset multi-strategy to perform re-inference and obtain the final inference results for each modality of data, including:
[0030] Identify specific conflicting information from the results of semantic matching, numerical attribute comparison, and logical relationship verification;
[0031] By incorporating specific conflict information as contextual information into the prompts of the reasoning nodes, a reinforced prompt template containing conflict descriptions is obtained.
[0032] By strengthening the prompt word template, the inference for each modality of data is re-performed to obtain the final inference result for each modality; or,
[0033] From each modal data, locate and extract the local information corresponding to the specific conflict information; re-infer the local information to obtain the final inference result of each modal data.
[0034] Optionally, trigger a conflict-driven re-inference mechanism based on a preset multi-strategy to perform re-inference and obtain the final inference results for each modality of data, including:
[0035] Acquire domain knowledge;
[0036] Extract logical constraints from the initial inference results and domain knowledge of each modality;
[0037] Represent logical constraints formally as a set of constraints;
[0038] By using constraint sets, the inference for each modality is re-performed to obtain the final inference result for each modality; or,
[0039] The inference path is backtracked from the initial inference results of each modality to obtain the inference chain of each modality; the first divergence point in the inference process is identified from the inference chain of each modality; the inference process is restarted from the divergence point to obtain the final inference result of each modality.
[0040] Optionally, the final inference results of each modality are fused to obtain the final answer corresponding to the inference node, including:
[0041] Obtain the final confidence score of the final inference results for each modality of data;
[0042] The final confidence scores of the final inference results of each modality data are weighted and summed to obtain the final fusion result corresponding to the inference node;
[0043] Calculate the semantic consistency score, attribute matching consistency score, and logical relationship consistency score for each modality of data under its respective re-inference path, and select the re-inference path with the highest consistency score as the final inference explanation chain;
[0044] If the consistency score of each modality data under its respective re-inference path is greater than the consistency score of each modality data under its respective initial inference path, the final fusion result and the final inference explanation chain corresponding to the inference node will be taken as the final answer corresponding to the inference node.
[0045] Secondly, embodiments of this application provide a multi-stage inference optimization device based on cross-modal consistency detection, the device comprising:
[0046] The independent modal reasoning module is used to respond to inference nodes carrying multimodal data, perform independent modal reasoning on each modal data, and obtain the initial reasoning results for each modal data;
[0047] The cross-modal consistency detection module is used to perform cross-modal consistency detection based on the initial inference results of each modality data to obtain a comprehensive conflict score. Cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification. Semantic matching is used to determine whether the semantic content of different initial inference results expresses the same or similar meaning. Numerical attribute comparison is used to determine whether there is numerical consistency in the numerical information in different initial inference results. Logical relationship verification is used to verify whether the logical relationships and causal relationships in different initial inference results are the same.
[0048] The marking and reasoning module is used to mark reasoning nodes as high-risk reasoning nodes when the overall conflict score is greater than a preset threshold, and to trigger a conflict-driven re-reasoning mechanism based on preset multi-strategy to perform re-reasoning and obtain the final reasoning results of each modality data.
[0049] The reasoning result fusion module is used to fuse the final reasoning results of each modality to obtain the final answer corresponding to the reasoning node.
[0050] The technical solutions provided in this application embodiment may include the following beneficial effects:
[0051] In this embodiment, on the one hand, cross-modal consistency detection is set during the reasoning process. This cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification to obtain a comprehensive conflict score. If the comprehensive conflict score is greater than a preset threshold, the system will mark the reasoning node as high-risk and trigger a conflict-driven re-reasoning mechanism. This mechanism can automatically detect contradictions between modalities and perform targeted re-reasoning, thereby correcting errors and preventing the propagation of errors in the reasoning chain, obtaining an accurate final answer, and thus improving the accuracy of multimodal reasoning. On the other hand, when the comprehensive conflict score is greater than the preset threshold, the system triggers the conflict-driven re-reasoning mechanism. In this way, the system can dynamically adjust the weight of each modality according to the actual situation, so that the contribution of high-quality modalities is enhanced and the misleading effect of low-quality modalities is weakened, realizing adaptive fusion of multimodal information, improving the adaptability to complex scenarios and the reliability of overall reasoning.
[0052] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0054] Figure 1 This is a flowchart illustrating a multi-stage inference optimization method based on cross-modal consistency detection provided in an embodiment of this application;
[0055] Figure 2 This is a schematic block diagram of a cross-modal consistency detection process provided in this application;
[0056] Figure 3 This is a schematic block diagram of a multi-stage inference optimization process based on cross-modal consistency detection provided in this application;
[0057] Figure 4 This is a schematic diagram of the structure of a multi-stage inference optimization device based on cross-modal consistency detection provided in an embodiment of this application;
[0058] Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation
[0059] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them.
[0060] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0061] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0062] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.
[0063] This application provides a multi-stage reasoning optimization method and apparatus based on cross-modal consistency detection to address the problems existing in the aforementioned related technologies. In the embodiments of this application, on the one hand, cross-modal consistency detection is set during the reasoning process. This cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification to obtain a comprehensive conflict score. If the comprehensive conflict score is greater than a preset threshold, the system will mark the reasoning node as high-risk and trigger a conflict-driven re-reasoning mechanism. This mechanism can automatically detect contradictions between modalities and perform targeted re-reasoning, thereby correcting errors and preventing the propagation of errors in the reasoning chain, obtaining an accurate final answer, and thus improving the accuracy of multimodal reasoning. On the other hand, when the comprehensive conflict score is greater than a preset threshold, the system triggers the conflict-driven re-reasoning mechanism. Through this method, the system can dynamically adjust the weights of each modality according to the actual situation, thereby increasing the contribution of high-quality modalities and weakening the misleading effect of low-quality modalities. This achieves adaptive fusion of multimodal information, improves the adaptability to complex scenarios, and enhances the reliability of overall reasoning. The following describes the process in detail using exemplary embodiments.
[0064] The following will be combined with the appendix Figure 1 -Appendix Figure 3 This application provides a detailed description of the multi-stage inference optimization method based on cross-modal consistency detection provided in its embodiments. This method can be implemented using a computer program and can run on a multi-stage inference optimization device based on the von Neumann architecture and cross-modal consistency detection. This computer program can be integrated into applications or run as a standalone utility application.
[0065] Please see Figure 1 This document provides a flowchart illustrating a multi-stage inference optimization method based on cross-modal consistency detection, as described in an embodiment of this application. Figure 1 As shown, the method in this application embodiment may include the following steps:
[0066] S101, in response to the inference node carrying multimodal data, performs independent modal inference on each modal data to obtain the initial inference result of each modal data;
[0067] Multimodal data comprises two or more different types of information or data, such as text, images, and audio, where different types of data provide complementary information about the same topic or question. An inference node is a step or stage in the inference process. Independent modal reasoning involves analyzing data for each modality separately, allowing the system to focus more on the information and features specific to each modality. The initial inference result is a preliminary conclusion or answer obtained after independent modal reasoning.
[0068] In some embodiments of this application, the system receives input data containing multiple modalities, obtaining multimodal data, such as an image (visual modality), a text describing the image (text modality), and related audio information (audio modality). For the text modality, a text inference module is used to understand the text content and perform inference. For the visual modality, a visual encoder is used to analyze the image content and extract image features, and then inference is performed. For the audio modality, speech recognition and audio analysis techniques are used to process the audio data and perform inference. The inference process for each modality generates an initial inference result and also outputs a confidence score for the initial inference result.
[0069] S102, perform cross-modal consistency detection based on the initial inference results of each modality data to obtain a comprehensive conflict score; cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification. Semantic matching is used to determine whether the semantic content of different initial inference results expresses the same or similar meanings. Numerical attribute comparison is used to determine whether the numerical information in different initial inference results is consistent. Logical relationship verification is used to verify whether the logical relationships and causal relationships in different initial inference results are the same.
[0070] In some embodiments of this application, the specific process of performing cross-modal consistency detection based on the initial inference results of each modal data to obtain a comprehensive conflict score includes: performing semantic matching on the initial inference results of each modal data to determine whether the semantic content of different initial inference results expresses the same or similar meanings, thereby obtaining a semantic consistency score; comparing the numerical attributes of the initial inference results of each modal data to determine whether the numerical information in different initial inference results is numerically consistent, thereby obtaining an attribute matching degree consistency score; verifying the logical relationships of the initial inference results of each modal data to verify whether the logical relationships and causal relationships in different initial inference results are the same, thereby obtaining a logical relationship consistency score; and weighted summing of the semantic consistency score, attribute matching degree consistency score, and logical relationship consistency score to obtain a comprehensive conflict score.
[0071] Specifically, the process of semantic matching of the initial inference results of each modality data includes: using a preset semantic similarity model to perform a semantic-level comparative analysis of the initial inference results of each modality data to calculate the semantic distance between the initial inference results of each modality data; and using the semantic distance as a semantic consistency score.
[0072] It should be noted that the process of determining whether inference results express the same or similar meanings by calculating the semantic distance between them can be formalized as calculating the similarity of the semantic vector representations of the inference results.
[0073] Specifically, the process of comparing the numerical attributes of the initial inference results of each modality data includes: extracting key numerical and attribute information from the initial inference results of each modality data; establishing an attribute correspondence matrix for each modality data using the numerical and attribute information; comparing the consistency of the structured attributes in the attribute correspondence matrix of each modality data to generate an attribute matching score.
[0074] It should be noted that numerical inconsistencies can be identified by comparing the consistency of these structured attributes one by one.
[0075] Specifically, the process of verifying the logical relationships of the initial inference results of each modality data includes: extracting the logical relationships and causal relationships of each modality data from the initial inference results of each modality data; formally representing the logical relationships and causal relationships of each modality data to obtain the logical relationship diagram of each modality data; and verifying the isomorphism and consistency of the logical relationship diagram of each modality data to verify the cross-modal consistency of the inference logic and generate a logical relationship consistency score.
[0076] It should be noted that the cross-modal consistency of reasoning logic is verified by examining the isomorphism and consistency of the logical relationship diagrams between different modalities.
[0077] In this embodiment, the comprehensive conflict score is used to quantify the degree of contradiction between the results of different modal reasoning.
[0078] For example Figure 2 As shown, Figure 2 This is a schematic flowchart of a cross-modal consistency detection process provided in this application. It obtains independent inference results from different modalities (text and visual). The system's inference result from the text modality is "3 people," with a confidence level of 0.88. The system's inference result from the visual modality is "2 people," with a confidence level of 0.95. Comparing the semantic results of the text and visual modalities, the text semantic is "3 people," while the visual semantic is "2 people," resulting in a mismatch. Comparing the numerical results of the text and visual modalities, the text numerical value is 3, while the visual numerical value is 2, resulting in an inconsistency. Comparing the logical relationship between the text and visual modalities, both the text logic and the visual logic are "people," resulting in a match. The comprehensive conflict score is calculated by weighted fusion of semantic consistency score, attribute matching consistency score, and logical relationship consistency score. The calculation expression is: Comprehensive Conflict Score = w1 × (1 The formula is: (semantic consistency score) + w2 × (1 - attribute matching consistency score) + w3 × (1 - logical relationship consistency score), where w1, w2 and w3 are weight coefficients.
[0079] It should be noted that different consistency scores can be quantified based on specific matching results. For example, the matching or non-matching mentioned above can be assigned values by simply setting thresholds. A higher threshold can be assigned to a matching case, and a lower threshold can be assigned to a non-matching case. The specific quantification can be based on the actual application scenario, which will not be elaborated here.
[0080] S103, when the comprehensive conflict score is greater than the preset threshold, mark the inference node as a high-risk inference node and trigger the conflict-driven re-inference mechanism based on the preset multi-strategy to perform re-inference and obtain the final inference result of each modality data.
[0081] The pre-defined multi-strategy includes an enhanced prompt word re-inference strategy, a region pruning and local inference strategy, a constraint inference strategy, and an inference path backtracking strategy. The enhanced prompt word re-inference strategy guides the model to re-infer by constructing an enhanced prompt word template containing conflict descriptions. The region pruning and local inference strategy locates and prunes the relevant region when a conflict is located in a specific region or attribute of the input data, and then re-infers the pruned local information. The constraint inference strategy extracts logical constraints from the initial inference results and domain knowledge of each modality, formally represents them as a constraint set, and performs inference under the constraints of this set. The inference path backtracking strategy automatically identifies the first divergence point in the inference process by comparing the inference chains of different modalities and restarts the inference process from that divergence point.
[0082] In one possible implementation, the specific process of triggering a conflict-driven re-reasoning mechanism based on a preset multi-strategy to obtain the final reasoning result of each modality data includes: identifying specific conflict information from the judgment results of semantic matching, numerical attribute comparison, and logical relationship verification; integrating the specific conflict information as contextual information into the prompt words of the reasoning node to obtain an enhanced prompt word template containing conflict description; and re-reasoning each modality data through the enhanced prompt word template to obtain the final reasoning result of each modality data.
[0083] It should be noted that this conflict-aware cue design allows the reasoning module to re-reason based on a full understanding of the contradictions between the multiple modalities, thereby improving the prudence and accuracy of the reasoning.
[0084] In another possible implementation, the specific process of triggering a conflict-driven re-inference mechanism based on a preset multi-strategy to obtain the final inference result of each modality data includes: locating and extracting the local information corresponding to the specific conflict information from each modality data; and re-performing local re-inference on the local information to obtain the final inference result of each modality data.
[0085] It should be noted that the local re-reasoning method can eliminate the interference of irrelevant information and improve the relevance and efficiency of reasoning.
[0086] In another possible implementation, the specific process of triggering a conflict-driven re-inference mechanism based on a preset multi-strategy to obtain the final inference results of each modality data includes: acquiring domain knowledge; extracting logical constraints from the initial inference results and domain knowledge of each modality data; formally representing the logical constraints as a constraint set; and re-inferring each modality data through the constraint set to obtain the final inference results of each modality data.
[0087] It should be noted that the logical consistency and rationality of the reasoning results are ensured through constraint propagation and constraint satisfaction mechanisms.
[0088] In another possible implementation, the specific process of triggering a conflict-driven re-inference mechanism based on a preset multi-strategy to obtain the final inference result of each modality data includes: backtracking the inference path of the initial inference result of each modality data to obtain the inference chain of each modality data; identifying the first divergence point in the inference process from the inference chain of each modality data; restarting the inference process from the divergence point to obtain the final inference result of each modality data.
[0089] It should be noted that the backtracking mechanism can accurately pinpoint the root cause of reasoning errors and prevent the accumulation and propagation of errors in subsequent reasoning steps.
[0090] It should be noted that during the execution of the above-mentioned multiple re-inference strategies, the system fully records the results of each re-inference, the change in confidence, and the improvement of the consistency score, providing multiple candidate inference schemes and corresponding quality assessment indicators for the subsequent result fusion stage.
[0091] S104, fuse the final inference results of each modality data to obtain the final answer corresponding to the inference node.
[0092] In some embodiments of this application, the specific process of fusing the final inference results of each modality data to obtain the final answer corresponding to the inference node includes: obtaining the final confidence score of the final inference results of each modality data; performing a weighted summation based on the final confidence scores of the final inference results of each modality data to obtain the final fusion result corresponding to the inference node; calculating the semantic consistency score, attribute matching consistency score, and logical relationship consistency score of each modality data under its respective re-inference path, and selecting the re-inference path with the highest consistency score as the final inference explanation chain; if the consistency score of each modality data under its respective re-inference path is greater than the consistency score of each modality data under its respective initial inference path, using the final fusion result and the final inference explanation chain corresponding to the inference node as the final answer corresponding to the inference node.
[0093] In one possible implementation, after implementing various re-inference strategies, the re-inference results from all modalities are collected and prepared for fusion. The system performs a weighted summation based on the confidence score of each inference result, giving higher weights to results with higher confidence scores. The consistency score of each inference path is calculated, and the inference path with the highest consistency is selected as the final inference explanation chain. The system compares the consistency scores before and after re-inference to evaluate the effect of re-inference. If re-inference significantly improves the consistency of the multimodal results, the re-inference result is adopted; otherwise, the original fusion result is used. Finally, the system outputs a structured result including the final answer, which may include other relevant information from the inference process (such as confidence scores, modal weights, etc.).
[0094] For example Figure 3 As shown, Figure 3This application provides a schematic flowchart of a multi-stage reasoning optimization process based on cross-modal consistency detection. The user inputs a text question, such as "How many people?" and a description like "Three pedestrians are chatting next to a white walled building...". The system receives a relevant image. The language model processes the text input, understands the question, and performs preliminary reasoning. The visual encoder processes the image input, extracts visual features, and performs preliminary reasoning. Cross-modal consistency detection is performed on the results, including semantic matching, attribute alignment, and logical verification. Semantic matching: Compares whether the semantic content of the text and visual modal is consistent. Attribute alignment: Compares whether the numerical attributes of the text and visual modal are consistent. Logical verification: Compares whether the logical relationships of the text and visual modal are consistent. Conflict score calculation: Calculates a comprehensive conflict score based on the above matching results. Checks whether the conflict score exceeds a preset threshold. If it does, a conflict-driven re-reasoning mechanism is triggered. If not, weighted fusion or path selection is performed. In result fusion, weighted fusion is a weighted sum based on the confidence score of each modality. Path selection selects the reasoning path with the highest consistency as the final answer. The system outputs the final answer, including the answer content and a confidence score. The reasoning chain provides a detailed reasoning process, including details and decision-making basis for each step. For example, the textual result indicates there are 3 people, with a confidence score of 0.88. The visual result indicates there are 2 people, with a confidence score of 0.95. The system performs cross-modal consistency detection and finds a conflict (conflict score of 0.75). Because the conflict score exceeds the threshold, the system triggers a re-reasoning mechanism, applying all strategies to arrive at the final answer of 2 people, with a confidence score of 0.94. Finally, the system outputs "Final Answer: 2 people (Confidence: 0.94)" and provides the detailed reasoning chain.
[0095] "Inference Chain": [
[0096] Step 1: Problem Understanding - Calculating the Number of People
[0097] Step 2: Independent Reasoning - Visual: 2 people, Text: 3 people
[0098] Step 3: Collision Detection - Score: 0.75
[0099] Step 4: Apply optimization - All strategies → Result = 2
[0100] Step 5: Final Answer - 2 people (Confidence level: 0.94)
[0101] ].
[0102] In this embodiment, on the one hand, cross-modal consistency detection is set during the reasoning process. This cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification to obtain a comprehensive conflict score. If the comprehensive conflict score is greater than a preset threshold, the system will mark the reasoning node as high-risk and trigger a conflict-driven re-reasoning mechanism. This mechanism can automatically detect contradictions between modalities and perform targeted re-reasoning, thereby correcting errors and preventing the propagation of errors in the reasoning chain, obtaining an accurate final answer, and thus improving the accuracy of multimodal reasoning. On the other hand, when the comprehensive conflict score is greater than the preset threshold, the system triggers the conflict-driven re-reasoning mechanism. In this way, the system can dynamically adjust the weight of each modality according to the actual situation, so that the contribution of high-quality modalities is enhanced and the misleading effect of low-quality modalities is weakened, realizing adaptive fusion of multimodal information, improving the adaptability to complex scenarios and the reliability of overall reasoning.
[0103] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0104] Please see Figure 4 This illustration shows a schematic diagram of a multi-stage inference optimization device based on cross-modal consistency detection provided in an exemplary embodiment of this application. This multi-stage inference optimization device based on cross-modal consistency detection can be implemented as all or part of a device through software, hardware, or a combination of both. The device 1 includes an independent modal inference module 10, a cross-modal consistency detection module 20, a labeled inference module 30, and an inference result fusion module 40.
[0105] The independent modal reasoning module 10 is used to respond to the reasoning node carrying multimodal data, perform independent modal reasoning on each modal data, and obtain the initial reasoning result of each modal data;
[0106] The cross-modal consistency detection module 20 is used to perform cross-modal consistency detection based on the initial inference results of each modality data to obtain a comprehensive conflict score. The cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification. Semantic matching is used to determine whether the semantic content of different initial inference results expresses the same or similar meaning. Numerical attribute comparison is used to determine whether there is numerical consistency in the numerical information in different initial inference results. Logical relationship verification is used to verify whether the logical relationship and causal relationship in different initial inference results are the same.
[0107] The marking inference module 30 is used to mark the inference node as a high-risk inference node when the comprehensive conflict score is greater than a preset threshold, and to trigger the conflict-driven re-inference mechanism based on preset multi-strategy to perform re-inference and obtain the final inference result of each modality data.
[0108] The reasoning result fusion module 40 is used to fuse the final reasoning results of each modality data to obtain the final answer corresponding to the reasoning node.
[0109] It should be noted that the multi-stage inference optimization device based on cross-modal consistency detection provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the multi-stage inference optimization method based on cross-modal consistency detection. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multi-stage inference optimization device based on cross-modal consistency detection provided in the above embodiments and the multi-stage inference optimization method embodiments based on cross-modal consistency detection belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.
[0110] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0111] In this embodiment, on the one hand, cross-modal consistency detection is set during the reasoning process. This cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification to obtain a comprehensive conflict score. If the comprehensive conflict score is greater than a preset threshold, the system will mark the reasoning node as high-risk and trigger a conflict-driven re-reasoning mechanism. This mechanism can automatically detect contradictions between modalities and perform targeted re-reasoning, thereby correcting errors and preventing the propagation of errors in the reasoning chain, obtaining an accurate final answer, and thus improving the accuracy of multimodal reasoning. On the other hand, when the comprehensive conflict score is greater than the preset threshold, the system triggers the conflict-driven re-reasoning mechanism. In this way, the system can dynamically adjust the weight of each modality according to the actual situation, so that the contribution of high-quality modalities is enhanced and the misleading effect of low-quality modalities is weakened, realizing adaptive fusion of multimodal information, improving the adaptability to complex scenarios and the reliability of overall reasoning.
[0112] This application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the multi-stage inference optimization method based on cross-modal consistency detection provided in the above-described method embodiments.
[0113] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the multi-stage inference optimization method based on cross-modal consistency detection in the above-described method embodiments.
[0114] Please see Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Figure 5As shown, device 1000 may include: at least one processor 1001, at least one network interface 1004, user interface 1003, memory 1005, and at least one communication bus 1002.
[0115] The communication bus 1002 is used to realize the connection and communication between these components.
[0116] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.
[0117] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0118] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the device 1000 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1001 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles operating devices, user interfaces, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 1001 and may be implemented as a separate chip.
[0119] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing the operating device, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 5 As shown, the memory 1005, which serves as a computer storage medium, may include an operating device, a network communication module, a user interface module, and a multi-stage inference optimization application based on cross-modal consistency detection.
[0120] exist Figure 5 In the device 1000 shown, the user interface 1003 is mainly used to provide an interface for user input and to obtain user input data; while the processor 1001 can be used to call the multi-stage inference optimization application based on cross-modal consistency detection stored in the memory 1005, and specifically perform the following operations:
[0121] In response to the inference node carrying multimodal data, independent modal inference is performed on each modal data to obtain the initial inference result of each modal data;
[0122] Cross-modal consistency detection is performed based on the initial inference results of each modality data to obtain a comprehensive conflict score. Cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification. Semantic matching is used to determine whether the semantic content of different initial inference results expresses the same or similar meaning. Numerical attribute comparison is used to determine whether there is numerical consistency in the numerical information in different initial inference results. Logical relationship verification is used to verify whether the logical relationship and causal relationship in different initial inference results are the same.
[0123] If the overall conflict score is greater than the preset threshold, the inference node is marked as a high-risk inference node, and the conflict-driven re-inference mechanism based on the preset multi-strategy is triggered to perform re-inference and obtain the final inference results of each modality data.
[0124] The final inference results of each modality are fused together to obtain the final answer corresponding to the inference node.
[0125] In one embodiment, when the processor 1001 performs cross-modal consistency detection based on the initial inference results of each modality's data to obtain a comprehensive conflict score, it specifically performs the following operations:
[0126] Semantic matching is performed on the initial inference results of each modality to determine whether the semantic content of different initial inference results expresses the same or similar meaning, and a semantic consistency score is obtained.
[0127] The initial inference results of each modality data are compared with numerical attributes to determine whether there is numerical consistency in the numerical information in different initial inference results, and thus obtain the attribute matching consistency score.
[0128] Logical relationship verification is performed on the initial inference results of each modality data to verify whether the logical and causal relationships in different initial inference results are the same, and a logical relationship consistency score is obtained.
[0129] The semantic consistency score, attribute matching consistency score, and logical relationship consistency score are weighted and summed to obtain the comprehensive conflict score.
[0130] In one embodiment, when the processor 1001 performs semantic matching on the initial inference results of each modality's data, it specifically performs the following operations:
[0131] A pre-defined semantic similarity model is used to perform a semantic-level comparative analysis of the initial inference results of each modality data in order to calculate the semantic distance between the initial inference results of each modality data.
[0132] Semantic distance is used as a semantic consistency score.
[0133] In one embodiment, when the processor 1001 performs a numerical attribute comparison on the initial inference results of each modality's data, it specifically performs the following operations:
[0134] Extract key numerical and attribute information from the initial inference results of each modality data;
[0135] Establish an attribute correspondence matrix for each modality of data by using numerical and attribute information;
[0136] Compare the consistency of structured attributes in the attribute correspondence matrix of each modality data to generate an attribute matching score.
[0137] In one embodiment, when the processor 1001 performs logical relationship verification on the initial inference results of each modality's data, it specifically performs the following operations:
[0138] Extract the logical and causal relationships between the modal data from the initial inference results of each modality;
[0139] The logical and causal relationships of each modality data are formally represented to obtain the logical relationship diagram of each modality data;
[0140] Examine the isomorphism and consistency of the logical relationship diagrams of each modality to verify the cross-modal consistency of the reasoning logic and generate a logical relationship consistency score.
[0141] In one embodiment, when the processor 1001 executes a conflict-driven re-inference mechanism based on a preset multi-strategy to perform re-inference and obtain the final inference results of each modality of data, it specifically performs the following operations:
[0142] Identify specific conflicting information from the results of semantic matching, numerical attribute comparison, and logical relationship verification;
[0143] By incorporating specific conflict information as contextual information into the prompts of the reasoning nodes, a reinforced prompt template containing conflict descriptions is obtained.
[0144] By strengthening the prompt word template, the inference for each modality of data is re-performed to obtain the final inference result for each modality; or,
[0145] From each modal data, locate and extract the local information corresponding to the specific conflict information; re-infer the local information to obtain the final inference result of each modal data.
[0146] In one embodiment, when the processor 1001 executes a conflict-driven re-inference mechanism based on a preset multi-strategy to perform re-inference and obtain the final inference results of each modality of data, it specifically performs the following operations:
[0147] Acquire domain knowledge;
[0148] Extract logical constraints from the initial inference results and domain knowledge of each modality;
[0149] Represent logical constraints formally as a set of constraints;
[0150] By using constraint sets, the inference for each modality is re-performed to obtain the final inference result for each modality; or,
[0151] The inference path is backtracked from the initial inference results of each modality to obtain the inference chain of each modality; the first divergence point in the inference process is identified from the inference chain of each modality; the inference process is restarted from the divergence point to obtain the final inference result of each modality.
[0152] In one embodiment, when the processor 1001 fuses the final inference results of each modality's data to obtain the final answer corresponding to the inference node, it specifically performs the following operations:
[0153] Obtain the final confidence score of the final inference results for each modality of data;
[0154] The final confidence scores of the final inference results of each modality data are weighted and summed to obtain the final fusion result corresponding to the inference node;
[0155] Calculate the semantic consistency score, attribute matching consistency score, and logical relationship consistency score for each modality of data under its respective re-inference path, and select the re-inference path with the highest consistency score as the final inference explanation chain;
[0156] If the consistency score of each modality data under its respective re-inference path is greater than the consistency score of each modality data under its respective initial inference path, the final fusion result and the final inference explanation chain corresponding to the inference node will be taken as the final answer corresponding to the inference node.
[0157] In this embodiment, on the one hand, cross-modal consistency detection is set during the reasoning process. This cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification to obtain a comprehensive conflict score. If the comprehensive conflict score is greater than a preset threshold, the system will mark the reasoning node as high-risk and trigger a conflict-driven re-reasoning mechanism. This mechanism can automatically detect contradictions between modalities and perform targeted re-reasoning, thereby correcting errors and preventing the propagation of errors in the reasoning chain, obtaining an accurate final answer, and thus improving the accuracy of multimodal reasoning. On the other hand, when the comprehensive conflict score is greater than the preset threshold, the system triggers the conflict-driven re-reasoning mechanism. In this way, the system can dynamically adjust the weight of each modality according to the actual situation, so that the contribution of high-quality modalities is enhanced and the misleading effect of low-quality modalities is weakened, realizing adaptive fusion of multimodal information, improving the adaptability to complex scenarios and the reliability of overall reasoning.
[0158] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for multi-stage inference optimization based on cross-modal consistency detection can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program for multi-stage inference optimization based on cross-modal consistency detection can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.
[0159] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.
Claims
1. A multi-stage inference optimization method based on cross-modal consistency detection, characterized in that, The method includes: In response to the inference node carrying multimodal data, independent modal inference is performed on each modal data to obtain the initial inference result of each modal data; Cross-modal consistency detection is performed based on the initial inference results of each modality data to obtain a comprehensive conflict score. The cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification. The semantic matching is used to determine whether the semantic content of different initial inference results expresses the same or similar meaning. The numerical attribute comparison is used to determine whether the numerical information in different initial inference results is consistent. The logical relationship verification is used to verify whether the logical relationship and causal relationship in different initial inference results are the same. If the overall conflict score exceeds a preset threshold, the inference node is marked as a high-risk inference node, and a conflict-driven re-inference mechanism based on preset multi-strategies is triggered to perform re-inference, obtaining the final inference results for each modality of data. The preset multi-strategies include an enhanced prompt re-inference strategy, a region pruning and local inference strategy, a constraint inference strategy, and an inference path backtracking strategy. The enhanced prompt re-inference strategy guides the model to re-infer by constructing enhanced prompt templates containing conflict descriptions. The region pruning and local inference strategy locates and prunes the relevant region when a conflict is located in a specific region or attribute of the input data, and then re-infers the pruned local information. The constraint inference strategy extracts logical constraints from the initial inference results and domain knowledge of each modality of data, formally represents them as a constraint set, and performs inference under the constraints of the constraint set. The inference path backtracking strategy automatically identifies the first divergence point in the inference process by comparing the inference chains of different modalities and restarts the inference process from that divergence point. Triggering a conflict-driven re-reasoning mechanism based on preset multi-strategies to perform re-reasoning and obtain the final reasoning result of each modality data includes: identifying specific conflict information from the judgment results of semantic matching, numerical attribute comparison, and logical relationship verification; integrating the specific conflict information as context information into the prompt words of the reasoning node to obtain an enhanced prompt word template containing conflict description; re-reasoning each modality data using the enhanced prompt word template to obtain the final reasoning result of each modality data; or, locating and extracting local information corresponding to the specific conflict information from each modality data; and re-performing local re-reasoning on the local information to obtain the final reasoning result of each modality data. The final inference results of each modality are fused to obtain the final answer corresponding to the inference node, including: weighted summation based on the confidence score of each final inference result, with higher confidence results receiving greater weight; calculation of the consistency score of each inference path, and selection of the inference path with the highest consistency as the final inference explanation chain; comparison of the consistency scores before and after re-inference, if re-inference improves the consistency of the multimodal results, the result after re-inference is adopted; otherwise, the original fusion result is adopted; and outputting a structured result including the final answer.
2. The method according to claim 1, characterized in that, The step of performing cross-modal consistency detection based on the initial inference results of each modal data to obtain a comprehensive conflict score includes: Semantic matching is performed on the initial inference results of each modality data to determine whether the semantic content of different initial inference results expresses the same or similar meaning, and a semantic consistency score is obtained. The initial inference results of each modality data are compared with numerical attributes to determine whether there is numerical consistency in the numerical information in different initial inference results, and to obtain an attribute matching consistency score. Logical relationship verification is performed on the initial inference results of each modality data to verify whether the logical and causal relationships in different initial inference results are the same, and a logical relationship consistency score is obtained. The semantic consistency score, the attribute matching consistency score, and the logical relationship consistency score are weighted and summed to obtain the comprehensive conflict score.
3. The method according to claim 2, characterized in that, Semantic matching is performed on the initial inference results of each modality data, including: A pre-defined semantic similarity model is used to perform a semantic-level comparative analysis on the initial inference results of each modality data in order to calculate the semantic distance between the initial inference results of each modality data. The semantic distance is used as the semantic consistency score.
4. The method according to claim 2, characterized in that, The numerical attribute comparison of the initial inference results for each modality data includes: Key numerical and attribute information is extracted from the initial inference results of each modality data; Using the numerical and attribute information, an attribute correspondence matrix for each modal data is established; The consistency of structured attributes in the attribute correspondence matrix of each modality data is compared to generate an attribute matching score.
5. The method according to claim 2, characterized in that, The logical relationship verification of the initial inference results of each modality data includes: From the initial inference results of each modality data, extract the logical and causal relationships of each modality data; The logical and causal relationships of each modal data are formally represented to obtain the logical relationship diagram of each modal data; The isomorphism and consistency of the logical relationship diagrams of the various modal data are examined to verify the cross-modal consistency of the reasoning logic and generate a logical relationship consistency score.
6. The method according to claim 1, characterized in that, Triggering a conflict-driven re-inference mechanism based on preset multi-strategy to perform re-inference, obtaining the final inference results for each modality of data, including: Acquire domain knowledge; Logical constraints are extracted from the initial inference results of each modality data and the domain knowledge. The logical constraints are formally represented as a set of constraints. Using the constraint set, the modal data is re-inferred to obtain the final inference result for each modal data; or, The initial inference results of each modality data are backtracked to obtain the inference chain of each modality data; the first divergence point in the inference process is identified from the inference chain of each modality data; the inference process is restarted from the divergence point to obtain the final inference result of each modality data.
7. The method according to claim 1, characterized in that, The step of fusing the final inference results of each modality data to obtain the final answer corresponding to the inference node includes: Obtain the final confidence score of the final inference result for each modality of data; The final fusion result corresponding to the inference node is obtained by weighted summation of the final confidence scores of the final inference results of each modality data. Calculate the semantic consistency score, attribute matching consistency score, and logical relationship consistency score of each modality data under its respective re-inference path, and select the re-inference path with the highest consistency score as the final inference explanation chain; If the consistency score of each modality data under its respective re-inference path is greater than the consistency score of each modality data under its respective initial inference path, the final fusion result corresponding to the inference node and the final inference explanation chain are taken as the final answer corresponding to the inference node.
8. A multi-stage inference optimization device based on cross-modal consistency detection implemented using the method described in any one of claims 1-7, characterized in that, The device includes: An independent modal reasoning module is used to respond to a reasoning node carrying multimodal data, perform independent modal reasoning on each modal data, and obtain the initial reasoning result of each modal data; The cross-modal consistency detection module is used to perform cross-modal consistency detection based on the initial inference results of each modality data to obtain a comprehensive conflict score. The cross-modal consistency detection includes semantic matching, numerical attribute comparison, and logical relationship verification. The semantic matching is used to determine whether the semantic content of different initial inference results expresses the same or similar meaning. The numerical attribute comparison is used to determine whether the numerical information in different initial inference results is numerically consistent. The logical relationship verification is used to verify whether the logical relationships and causal relationships in different initial inference results are the same. The marking inference module is used to mark the inference node as a high-risk inference node when the comprehensive conflict score is greater than a preset threshold, and trigger a conflict-driven re-inference mechanism based on a preset multi-strategy to perform re-inference and obtain the final inference result of each modality data. The reasoning result fusion module is used to fuse the final reasoning results of each modality data to obtain the final answer corresponding to the reasoning node.